Close: https://github.com/apache/airflow/issues/16533

We have a mechanism that retrieves missing inventories from the remote cache if they are not present locally. This helps to prevent the problem of eggs and chickens. For this to work, we need to determine if this package has versioning or not in the docs/exts/docs_build/fetch_inventories.py file.

<!-- Thank you for contributing! Please make sure that your code changes are covered with tests. And in case of new features or big changes remember to adjust the documentation.

Feel free to ping committers for the review!

In case of existing issue, reference it using one of the following:

closes: #ISSUE related: #ISSUE

How to write a good git commit message: http://chris.beams.io/posts/git-commit/ -->

^ Add meaningful description above

Read the Pull Request Guidelines for more information. In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed. In case of a new dependency, check compliance with the ASF 3rd Party License Policy. In case of backwards incompatible changes please leave a note in UPDATING.md.

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst) Here are some useful points:

• Pay attention to the quality of your code (flake8, pylint and type annotations). Our pre-commits will help you with that.
• In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
• Consider using Breeze environment for testing locally, it’s a heavy docker but it ships with a working Airflow and a lot of integrations.
• Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
• Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
• Be sure to read the Airflow Coding style. Apache Airflow is a community-driven project and together we are making it better 🚀. In case of doubts contact the developers at: Mailing List: [email protected] Slack: https://s.apache.org/airflow-slack

The current docstring for the airflow.models.dag.DAG class is missing the render_template_as_native_obj parameter and therefore missing in the public documentation as well. This PR updates the docstring accordingly.

^ Add meaningful description above

Read the Pull Request Guidelines for more information. In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed. In case of a new dependency, check compliance with the ASF 3rd Party License Policy. In case of backwards incompatible changes please leave a note in UPDATING.md.

Apache Airflow version: 2.0.1

Environment:

• Cloud provider or hardware configuration: AWS
• OS (e.g. from /etc/os-release): Ubuntu 20.04

What happened:

I trigger my dag with the API from a lambda function with a trigger on a file upload. I get the file path from the lambda context i.e. : ingestion.archive.dev/yolo/PMS_2_DXBTD_RTBD_2021032800000020210328000000SD_20210329052822.XML

I put this variable in the API call to get it back as "{{ dag_run.conf['file_path'] }}"

At some point, I need to extract information from this string by splitting it by / so inside the DAG to use the S3CopyObjectOperator.

So here the first approach I had

from datetime import datetime

from airflow import DAG
from airflow.providers.amazon.aws.operators.s3_copy_object import S3CopyObjectOperator
from airflow.operators.python_operator import PythonOperator


default_args = {
    'owner': 'me',
}

s3_final_destination = {
    "bucket_name": "ingestion.archive.dev",
    "verification_failed": "validation_failed",
    "processing_failed": "processing_failed",
    "processing_success": "processing_success"
}


def print_var(file_path,
              file_split,
              source_bucket,
              source_path,
              file_name):
    data = {
        "file_path": file_path,
        "file_split": file_split,
        "source_bucket": source_bucket,
        "source_path": source_path,
        "file_name": file_name
    }

    print(data)


with DAG(
        f"test_s3_transfer",
        default_args=default_args,
        description='Test',
        schedule_interval=None,
        start_date=datetime(2021, 4, 24),
        tags=['ingestion', "test", "context"],

) as dag:
    # {"file_path": "ingestion.archive.dev/yolo/PMS_2_DXBTD_RTBD_2021032800000020210328000000SD_20210329052822.XML"}
    file_path = "{{ dag_run.conf['file_path'] }}"
    file_split = file_path.split('/')
    source_bucket = file_split[0]
    source_path = "/".join(file_split[1:])
    file_name = file_split[-1]

    test_var = PythonOperator(
        task_id="test_var",
        python_callable=print_var,
        op_kwargs={
            "file_path": file_path,
            "file_split": file_split,
            "source_bucket": source_bucket,
            "source_path": source_path,
            "file_name": file_name
        }
    )

    file_verification_fail_to_s3 = S3CopyObjectOperator(
        task_id="file_verification_fail_to_s3",
        source_bucket_key=source_bucket,
        source_bucket_name=source_path,
        dest_bucket_key=s3_final_destination["bucket_name"],
        dest_bucket_name=f'{s3_final_destination["verification_failed"]}/{file_name}'
    )

    test_var >> file_verification_fail_to_s3

I use the PythonOperator to check the value I got to debug. I have the right value in file_path but I got in file_split -> ['ingestion.archive.dev/yolo/PMS_2_DXBTD_RTBD_2021032800000020210328000000SD_20210329052822.XML'] It's my str in a list and not each part splited like ["ingestion.archive.dev", "yolo", "PMS_2_DXBTD_RTBD_2021032800000020210328000000SD_20210329052822.XML"].

So what's wrong here? So I started to read more about Jinja Templating and I find out this on Airflow : https://airflow.apache.org/docs/apache-airflow/stable/concepts/operators.html#rendering-fields-as-native-python-objects

And try to use render_template_as_native_obj=True to solve my issue, but I got an error when the scheduler picked up my dag saying that this args isn't in the DAG object. Effectivly in the documentation, you cannot find it either : https://airflow.apache.org/docs/apache-airflow/stable/_api/airflow/models/dag/index.html?highlight=dag#module-airflow.models.dag

I try to use this argument in the jinja_environment_kwargs arg, but it's not available again. So there is a regression and an error in the documentation.

But my real question is how to split my jinja str ?

For the doc issue. My bad I didn't check the version you are right. Good if I help out on something. When I try to fix the doc with the builtin button "Suggest a change on this page" I got a 404: https://github.com/apache/airflow/edit/devel/docs/apache-airflow/concepts/operators.rst

For the splitting issue, I knew that jinja was working at running level but I thought that the splitting would work. I made several variables like that because I will need those values like 5 times in the same DAG (the one I shared is for demonstration purposes). That's why I didn't want to write the logic multiple times and had to fix the same bug at differents places (kinda factorization). The PythonOperator is for demonstration here so I know how to use XCom. For the S3Operator I will use the templated values.

Thank you for the quick answer.

Hi @gfelot,

Addressing the errors you are observing:

• The documentation you are referring to is for version 2.1 (current stable) while you are running 2.0.2 (of which the documentation can be found here). The render_template_as_native_obj parameter wasn't introduced until 2.1 so that's why it's missing from the DAG object within your environment. Additionally, the jinja_environment_kwargs arguments refer to the initialization parameters for a jinja2.Environment.
• The documentation for 2.1 is missing for the render_template_as_native_obj parameter and we'll get that added. Thank you for catching this!

Pertaining to splitting Jinja strings:

• Jinja expressions are not evaluated/rendered in Airflow until runtime (i.e. within the operator's execute() method. As written, the parsing/splitting of the string is done outside of the operator. Only after the task begins executing is the Jinja expression evaluated and rendered. Therefore the value being initialized is [" {{ dag_run.conf['file_path'] }} "] as there is no / character in the literal string of {{ dag_run.conf['file_path'] }}.
• A couple options:
  • Assuming the "test_var" task is not for demonstration purposes, move the splitting of the file_path value into the print_var() function and use XComs or the Taskflow API to pass values to the "file_verification_fail_to_s3" task.
  • Since the source_bucket_key, source_bucket_name, and dest_bucket_name arguments can be templated values in the S3CopyObjectOperator, you could directly access the conf object as you do now and split the value as needed within the Jinja expression.

For similar questions in the future, please use GitHub discussions, not issues.

:green_heart: :blue_heart: :purple_heart: :yellow_heart: :heart:

Closes #241

The return value here is a 5 element tuple, rather than the 4 element tuple declared in the typespec:

The typespec:

https://github.com/elixir-ecto/db_connection/blob/4d7e789943f1c3a0b54b871f3807a04303e666dc/lib/db_connection/holder.ex#L50

The offenders:

https://github.com/elixir-ecto/db_connection/blob/4d7e789943f1c3a0b54b871f3807a04303e666dc/lib/db_connection/holder.ex#L57

https://github.com/elixir-ecto/db_connection/blob/4d7e789943f1c3a0b54b871f3807a04303e666dc/lib/db_connection/holder.ex#L77

Closes #241

Also - please rebase again - there was a change in main that makes the image build to fail without rebase.

One small request again. Can we simply limit the size of the file ? It's not huge, but for an example, I think just getting it down to like 100 records rather than then the whole 20K should be perfectly OK I think. Sorry for the back/forth but I think additional few megs to get is not good for people who have limited bandwidth.

The return value here is a 5 element tuple, rather than the 4 element tuple declared in the typespec:

https://github.com/elixir-ecto/db_connection/blob/master/lib/db_connection/holder.ex#L57

Airflow 1.10 has just (yesterday) reached End-Of-Life yesterday (!). It won't receive any more even critical security fixes. Please upgrade to 2.1+ at earliest convenience.

Description

We're still on Airflow v1.10.15 (moving to v2, but it's taking a while). However, Airflow v1 depends on requests < 2.24, i.e. the latest version that can be installed is 2.23.0, but this in turn requires urllib3<1.26. However, there is a vulnerability in urrlib3 before 1.26.5 https://github.com/advisories/GHSA-q2q7-5pp4-w6pg

It's probably a bit cheeky to ask, but you never know: can there be another v1 release that allows enough of a bump of requests to allow urllib3 v1.26.5?

Use case / motivation

Avoid vulnerabilities in Airflow v1

Are you willing to submit a PR?

No (sorry...)

Related Issues

No

<!--

Welcome to Apache Airflow! For a smooth issue process, try to answer the following questions. Don't worry if they're not all applicable; just try to include what you can :-)

If you need to include code snippets or logs, please put them in fenced code blocks. If they're super-long, please use the details tag like <details><summary>super-long log</summary> lots of stuff </details>

Please delete these comment blocks before submitting the issue.

-->

<!--

IMPORTANT!!!

PLEASE CHECK "SIMILAR TO X EXISTING ISSUES" OPTION IF VISIBLE NEXT TO "SUBMIT NEW ISSUE" BUTTON!!!

PLEASE CHECK IF THIS ISSUE HAS BEEN REPORTED PREVIOUSLY USING SEARCH!!!

Please complete the next sections or the issue will be closed. These questions are the first thing we need to know to understand the context.

-->

Apache Airflow version:

Kubernetes version (if you are using kubernetes) (use kubectl version):

Environment:

• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Others:

What happened:

<!-- (please include exact error messages if you can) -->

What you expected to happen:

<!-- What do you think went wrong? -->

How to reproduce it: <!---

As minimally and precisely as possible. Keep in mind we do not have access to your cluster or dags.

If you are using kubernetes, please attempt to recreate the issue using minikube or kind.

Install minikube/kind

• Minikube https://minikube.sigs.k8s.io/docs/start/
• Kind https://kind.sigs.k8s.io/docs/user/quick-start/

If this is a UI bug, please provide a screenshot of the bug or a link to a youtube video of the bug in action

You can include images using the .md style of

To record a screencast, mac users can use QuickTime and then create an unlisted youtube video with the resulting .mov file.

--->

Anything else we need to know:

<!--

How often does this problem occur? Once? Every time etc?

Any relevant logs to include? Put them here in side a detail tag: <details><summary>x.log</summary> lots of stuff </details>

-->

Nothing here.

Saved me couple of hours today already.

Hey @ashb @kaxil - regardless of the long-term solution, this is is a very nice automation that helped me to manage the provider issue with status of releases - both the first one and the follow up. It's rather nice IMHO and allows to make it manageable with testing status and pinging people automatically to test the release of providers. It's green, I could merge it :)

Not willingly, in any case.

You can check if that is the case by tracing exclusively on the init/terminate callbacks. Trace to a file and then check that it is full of init followed by terminate.

As new package RCs are released, I encourage those from the list who have not done so, to double-check if the packages are working fine with their changes. The voting time has been prolonged to Tue 22 Jun 21:29:59 CEST 2021.

Looking forward to your thumb-ups :)

I don't think it is fair to compare is_empty with nth, hd and tail. First, this is_empty function is total. It is even an isomorphism between ('a list, @) and (bool, &&), thus it does seem to respect the algebraic nature of lists.

I always thought of uppercase as an adjective here, not a verb (like escaped).

The last commit looks fine. I will have a last glance after this week-end, then I will merge on Monday.

I captured the state of two processes (actually the same process twice) that were getting high reductions:

#state{
	parent = <0.4943.0>,
	ref = MyAppWeb.Endpoint.HTTP,
	socket = #Port<0.4067799>,
	transport = :ranch_tcp,
	proxy_header = undefined,
	opts = #{} :: #{env =>
          #{dispatch =>
                [{'_',[],
                     [{'_',[],'Elixir.Phoenix.Endpoint.Cowboy2Handler',
                          {'Elixir.MyApp.Endpoint',[]}}]}]},
      idle_timeout => 70000,max_keepalive => 20000,
      stream_handlers => [cowboy_telemetry_h,cowboy_stream_h]},
	buffer = <<>>,
	overriden_opts = #{},
  peer = {{0,0,0,0,0,65535,44051,33230},1060},
  sock = {{0,0,0,0,0,65535,44049,7},4080},
	cert = undefined,
	timer = #Reference<0.327935682.2040528927.255668>,
	active = true,
	in_streamid = 286,
	in_state = #ps_request_line{empty_lines=0},
	flow = infinity,
	out_streamid = 286,
	out_state = wait,
	last_streamid = 20000,
	streams = [],
	children = []
}

#state{
	parent = <0.4943.0>,
	ref = MyAppWeb.Endpoint.HTTP,
	socket = #Port<0.3982394>,
	transport = :ranch_tcp,
	proxy_header = undefined,
	opts = #{} :: #{env =>
          #{dispatch =>
                [{'_',[],
                     [{'_',[],'Elixir.Phoenix.Endpoint.Cowboy2Handler',
                          {'Elixir.MyApp.Endpoint',[]}}]}]},
      idle_timeout => 70000,max_keepalive => 20000,
      stream_handlers => [cowboy_telemetry_h,cowboy_stream_h]},
	buffer = <<>>,
	overriden_opts = #{},
  peer = {{0,0,0,0,0,65535,44051,33230},54968},
  sock = {{0,0,0,0,0,65535,44049,7},4080},
	cert = undefined,
	timer = #Reference<0.327935682.2040528924.235998>,
	active = true,
	in_streamid = 1962,
	in_state = #ps_request_line{empty_lines=0},
	flow = infinity,
	out_streamid = 1961,
	out_state = wait,
	last_streamid = 20000,
	streams = [#stream{
        id = 1961,
        state = #cowboy_telemetry_h{...},
        method = "POST",
        version = "HTTP/1.1",
        te :: undefined ,
        local_expected_size = undefined,
        local_sent_size = 0 ,
        queue = []
    }],
	children = [#child{
        pid = #PID<0.22534.478>,
        streamid = 1961,
        shutdown = 5000,
        timer = undefined
    }]
}

Nothing really jumps out at me with those two processes. One thing I did notice this time though is that the process with the top number of reductions is often the logger:

iex(socialx@2a4a976d3e85)36> :recon.proc_window(:reductions, 5, 5000)
[
  {#PID<0.4621.0>, 1681468,
   [
     Logger,
     {:current_function, {:gen_event, :fetch_msg, 6}},
     {:initial_call, {:proc_lib, :init_p, 5}}
   ]},
  {#PID<0.29288.479>, 205943,
   [
     current_function: {:cowboy_http, :loop, 1},
     initial_call: {:proc_lib, :init_p, 5}
   ]},
   ...
]

This probably correlates with a massive uptick we're seeing in error logs coming from our telemetry code. Our current telemetry setup is: cowboy -> cowboy_telemetry -> telemetry -> some internal glue code -> Spandex

Spandex has an error log that we hit a lot during these:

Logger.error("Tried to start a trace over top of another trace. name: #{inspect(name)}")

https://github.com/spandex-project/spandex/blob/d14f78ce416ccedc2b16e0acd1927809e626ca30/lib/spandex.ex#L40

The error happens when there is already an open Spandex trace in the process dict (which I believe would be the cowboy_http process).

We attempt to continue the trace as a result of the telemetry event [cowboy, request, start] which cowboy_telemetry fires from the init call of the cowboy_telemetry_h stream handler.

All this brings me to ask - did something change in 2.9 that would cause init function of a handler to be called multiple times or the metrics_callback (which stops the trace) to not be called before the next init is called?

After pondering, this is more my personal opinion but I would say:

• assert to enforce invariants, and it is the responsibility of the callers to respect those.
• fatal_error to protect downstream users from an erroneous state that must not be propagated.