profile
viewpoint
If you are wondering where the data of this site comes from, please visit https://api.github.com/users/pap/events. GitMemory does not store any data, but only uses NGINX to cache data for a period of time. The idea behind GitMemory is simply to give users a better reading experience.

pap/30-days-of-elixir 2

A walk through the Elixir language in 30 exercises.

pap/basedevcontainer 1

Base development Docker image used by other development Docker images

pap/devcontainer-base 1

Base container for Visual Studio Code Remote Containers development

pap/devcontainer-elixir 1

Elixir container for Visual Studio Code Remote Containers development

pap/devcontainer-elixir-phoenix 1

Phoenix framework container for Visual Studio Code Remote Containers development

pap/a-little-elixir-goes-a-long-way 0

The Little Schemer in Elixir.

pap/ahoy 0

Never build an analytics platform from scratch again

pap/awesome-elixir 0

A curated list of amazingly awesome Elixir and Erlang libraries, resources and shiny things.

pap/awesome-microservices 0

A curated list of Microservice Architecture related principles and technologies.

PR opened apache/airflow

Fetch Helm Chart inventory from remote cache

Close: https://github.com/apache/airflow/issues/16533

We have a mechanism that retrieves missing inventories from the remote cache if they are not present locally. This helps to prevent the problem of eggs and chickens. For this to work, we need to determine if this package has versioning or not in the docs/exts/docs_build/fetch_inventories.py file.

<!-- Thank you for contributing! Please make sure that your code changes are covered with tests. And in case of new features or big changes remember to adjust the documentation.

Feel free to ping committers for the review!

In case of existing issue, reference it using one of the following:

closes: #ISSUE related: #ISSUE

How to write a good git commit message: http://chris.beams.io/posts/git-commit/ -->


^ Add meaningful description above

Read the Pull Request Guidelines for more information. In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed. In case of a new dependency, check compliance with the ASF 3rd Party License Policy. In case of backwards incompatible changes please leave a note in UPDATING.md.

+7 -6

0 comment

1 changed file

pr created time in 11 minutes

push eventapache/airflow

Jarek Potiuk

commit sha 0369ea5dc2b187fd531bd7a80581bdc76045fb43

Updated 3.9 constraints to non-conflicting version

view details

push time in 2 hours

pull request commentapache/airflow

Updating the DAG docstring to include `render_template_as_native_obj`

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst) Here are some useful points:

  • Pay attention to the quality of your code (flake8, pylint and type annotations). Our pre-commits will help you with that.
  • In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
  • Consider using Breeze environment for testing locally, it’s a heavy docker but it ships with a working Airflow and a lot of integrations.
  • Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
  • Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
  • Be sure to read the Airflow Coding style. Apache Airflow is a community-driven project and together we are making it better 🚀. In case of doubts contact the developers at: Mailing List: dev@airflow.apache.org Slack: https://s.apache.org/airflow-slack
josh-fell

comment created time in 2 hours

PR opened apache/airflow

Updating the DAG docstring to include `render_template_as_native_obj`

The current docstring for the airflow.models.dag.DAG class is missing the render_template_as_native_obj parameter and therefore missing in the public documentation as well. This PR updates the docstring accordingly.


^ Add meaningful description above

Read the Pull Request Guidelines for more information. In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed. In case of a new dependency, check compliance with the ASF 3rd Party License Policy. In case of backwards incompatible changes please leave a note in UPDATING.md.

+4 -0

0 comment

1 changed file

pr created time in 2 hours

issue closedapache/airflow

Cannot split Jinja str and render_template_as_native_obj doesn't exist

Apache Airflow version: 2.0.1

Environment:

  • Cloud provider or hardware configuration: AWS
  • OS (e.g. from /etc/os-release): Ubuntu 20.04

What happened:

I trigger my dag with the API from a lambda function with a trigger on a file upload. I get the file path from the lambda context i.e. : ingestion.archive.dev/yolo/PMS_2_DXBTD_RTBD_2021032800000020210328000000SD_20210329052822.XML

I put this variable in the API call to get it back as "{{ dag_run.conf['file_path'] }}"

At some point, I need to extract information from this string by splitting it by / so inside the DAG to use the S3CopyObjectOperator.

So here the first approach I had

from datetime import datetime

from airflow import DAG
from airflow.providers.amazon.aws.operators.s3_copy_object import S3CopyObjectOperator
from airflow.operators.python_operator import PythonOperator


default_args = {
    'owner': 'me',
}

s3_final_destination = {
    "bucket_name": "ingestion.archive.dev",
    "verification_failed": "validation_failed",
    "processing_failed": "processing_failed",
    "processing_success": "processing_success"
}


def print_var(file_path,
              file_split,
              source_bucket,
              source_path,
              file_name):
    data = {
        "file_path": file_path,
        "file_split": file_split,
        "source_bucket": source_bucket,
        "source_path": source_path,
        "file_name": file_name
    }

    print(data)


with DAG(
        f"test_s3_transfer",
        default_args=default_args,
        description='Test',
        schedule_interval=None,
        start_date=datetime(2021, 4, 24),
        tags=['ingestion', "test", "context"],

) as dag:
    # {"file_path": "ingestion.archive.dev/yolo/PMS_2_DXBTD_RTBD_2021032800000020210328000000SD_20210329052822.XML"}
    file_path = "{{ dag_run.conf['file_path'] }}"
    file_split = file_path.split('/')
    source_bucket = file_split[0]
    source_path = "/".join(file_split[1:])
    file_name = file_split[-1]

    test_var = PythonOperator(
        task_id="test_var",
        python_callable=print_var,
        op_kwargs={
            "file_path": file_path,
            "file_split": file_split,
            "source_bucket": source_bucket,
            "source_path": source_path,
            "file_name": file_name
        }
    )

    file_verification_fail_to_s3 = S3CopyObjectOperator(
        task_id="file_verification_fail_to_s3",
        source_bucket_key=source_bucket,
        source_bucket_name=source_path,
        dest_bucket_key=s3_final_destination["bucket_name"],
        dest_bucket_name=f'{s3_final_destination["verification_failed"]}/{file_name}'
    )

    test_var >> file_verification_fail_to_s3

I use the PythonOperator to check the value I got to debug. I have the right value in file_path but I got in file_split -> ['ingestion.archive.dev/yolo/PMS_2_DXBTD_RTBD_2021032800000020210328000000SD_20210329052822.XML'] It's my str in a list and not each part splited like ["ingestion.archive.dev", "yolo", "PMS_2_DXBTD_RTBD_2021032800000020210328000000SD_20210329052822.XML"].

So what's wrong here? So I started to read more about Jinja Templating and I find out this on Airflow : https://airflow.apache.org/docs/apache-airflow/stable/concepts/operators.html#rendering-fields-as-native-python-objects

And try to use render_template_as_native_obj=True to solve my issue, but I got an error when the scheduler picked up my dag saying that this args isn't in the DAG object. Effectivly in the documentation, you cannot find it either : https://airflow.apache.org/docs/apache-airflow/stable/_api/airflow/models/dag/index.html?highlight=dag#module-airflow.models.dag

I try to use this argument in the jinja_environment_kwargs arg, but it's not available again. So there is a regression and an error in the documentation.

But my real question is how to split my jinja str ?

closed time in 2 hours

gfelot

issue commentapache/airflow

Cannot split Jinja str and render_template_as_native_obj doesn't exist

For the doc issue. My bad I didn't check the version you are right. Good if I help out on something. When I try to fix the doc with the builtin button "Suggest a change on this page" I got a 404: https://github.com/apache/airflow/edit/devel/docs/apache-airflow/concepts/operators.rst

For the splitting issue, I knew that jinja was working at running level but I thought that the splitting would work. I made several variables like that because I will need those values like 5 times in the same DAG (the one I shared is for demonstration purposes). That's why I didn't want to write the logic multiple times and had to fix the same bug at differents places (kinda factorization). The PythonOperator is for demonstration here so I know how to use XCom. For the S3Operator I will use the templated values.

Thank you for the quick answer.

gfelot

comment created time in 2 hours

issue commentapache/airflow

Cannot split Jinja str and render_template_as_native_obj doesn't exist

Hi @gfelot,

Addressing the errors you are observing:

  • The documentation you are referring to is for version 2.1 (current stable) while you are running 2.0.2 (of which the documentation can be found here). The render_template_as_native_obj parameter wasn't introduced until 2.1 so that's why it's missing from the DAG object within your environment. Additionally, the jinja_environment_kwargs arguments refer to the initialization parameters for a jinja2.Environment.
  • The documentation for 2.1 is missing for the render_template_as_native_obj parameter and we'll get that added. Thank you for catching this!

Pertaining to splitting Jinja strings:

  • Jinja expressions are not evaluated/rendered in Airflow until runtime (i.e. within the operator's execute() method. As written, the parsing/splitting of the string is done outside of the operator. Only after the task begins executing is the Jinja expression evaluated and rendered. Therefore the value being initialized is [" {{ dag_run.conf['file_path'] }} "] as there is no / character in the literal string of {{ dag_run.conf['file_path'] }}.
  • A couple options:
    • Assuming the "test_var" task is not for demonstration purposes, move the splitting of the file_path value into the print_var() function and use XComs or the Taskflow API to pass values to the "file_verification_fail_to_s3" task.
    • Since the source_bucket_key, source_bucket_name, and dest_bucket_name arguments can be templated values in the S3CopyObjectOperator, you could directly access the conf object as you do now and split the value as needed within the Jinja expression.

For similar questions in the future, please use GitHub discussions, not issues.

gfelot

comment created time in 3 hours

delete branch elixir-ecto/db_connection

delete branch : josevalim-patch-1

delete time in 3 hours

pull request commentelixir-ecto/db_connection

Update spec

:green_heart: :blue_heart: :purple_heart: :yellow_heart: :heart:

josevalim

comment created time in 3 hours

push eventelixir-ecto/db_connection

José Valim

commit sha 6850853977a08ca9bbc7fc5d9d2b07f028720a6c

Update spec (#242) Closes #241

view details

push time in 3 hours

PR merged elixir-ecto/db_connection

Update spec

Closes #241

+4 -2

0 comment

1 changed file

josevalim

pr closed time in 3 hours

issue closedelixir-ecto/db_connection

Typespec Inconsistency

The return value here is a 5 element tuple, rather than the 4 element tuple declared in the typespec:

The typespec:

https://github.com/elixir-ecto/db_connection/blob/4d7e789943f1c3a0b54b871f3807a04303e666dc/lib/db_connection/holder.ex#L50

The offenders:

https://github.com/elixir-ecto/db_connection/blob/4d7e789943f1c3a0b54b871f3807a04303e666dc/lib/db_connection/holder.ex#L57

https://github.com/elixir-ecto/db_connection/blob/4d7e789943f1c3a0b54b871f3807a04303e666dc/lib/db_connection/holder.ex#L77

closed time in 3 hours

isaacsanders

PR opened elixir-ecto/db_connection

Update spec

Closes #241

+4 -2

0 comment

1 changed file

pr created time in 3 hours

create barnchelixir-ecto/db_connection

branch : josevalim-patch-1

created branch time in 3 hours

pull request commentapache/airflow

Added new pipeline example for the tutorial docs (Issue #11208)

Also - please rebase again - there was a change in main that makes the image build to fail without rebase.

Sanchit112

comment created time in 3 hours

pull request commentapache/airflow

Added new pipeline example for the tutorial docs (Issue #11208)

One small request again. Can we simply limit the size of the file ? It's not huge, but for an example, I think just getting it down to like 100 records rather than then the whole 20K should be perfectly OK I think. Sorry for the back/forth but I think additional few megs to get is not good for people who have limited bandwidth.

Sanchit112

comment created time in 3 hours

issue openedelixir-ecto/db_connection

Typespec Inconsistency

The return value here is a 5 element tuple, rather than the 4 element tuple declared in the typespec:

https://github.com/elixir-ecto/db_connection/blob/master/lib/db_connection/holder.ex#L57

created time in 3 hours

issue commentapache/airflow

Request: v1 release that allows urllib3 v1.26.5

Airflow 1.10 has just (yesterday) reached End-Of-Life yesterday (!). It won't receive any more even critical security fixes. Please upgrade to 2.1+ at earliest convenience.

michalc

comment created time in 4 hours

issue closedapache/airflow

Request: v1 release that allows urllib3 v1.26.5

Description

We're still on Airflow v1.10.15 (moving to v2, but it's taking a while). However, Airflow v1 depends on requests < 2.24, i.e. the latest version that can be installed is 2.23.0, but this in turn requires urllib3<1.26. However, there is a vulnerability in urrlib3 before 1.26.5 https://github.com/advisories/GHSA-q2q7-5pp4-w6pg

It's probably a bit cheeky to ask, but you never know: can there be another v1 release that allows enough of a bump of requests to allow urllib3 v1.26.5?

Use case / motivation

Avoid vulnerabilities in Airflow v1

Are you willing to submit a PR?

No (sorry...)

Related Issues

No

closed time in 4 hours

michalc

issue closedapache/airflow

Windows 10 bug Ubuntu

<!--

Welcome to Apache Airflow! For a smooth issue process, try to answer the following questions. Don't worry if they're not all applicable; just try to include what you can :-)

If you need to include code snippets or logs, please put them in fenced code blocks. If they're super-long, please use the details tag like <details><summary>super-long log</summary> lots of stuff </details>

Please delete these comment blocks before submitting the issue.

-->

<!--

IMPORTANT!!!

PLEASE CHECK "SIMILAR TO X EXISTING ISSUES" OPTION IF VISIBLE NEXT TO "SUBMIT NEW ISSUE" BUTTON!!!

PLEASE CHECK IF THIS ISSUE HAS BEEN REPORTED PREVIOUSLY USING SEARCH!!!

Please complete the next sections or the issue will be closed. These questions are the first thing we need to know to understand the context.

-->

Apache Airflow version:

Kubernetes version (if you are using kubernetes) (use kubectl version):

Environment:

  • Cloud provider or hardware configuration:
  • OS (e.g. from /etc/os-release):
  • Kernel (e.g. uname -a):
  • Install tools:
  • Others:

What happened:

<!-- (please include exact error messages if you can) -->

What you expected to happen:

<!-- What do you think went wrong? -->

How to reproduce it: <!---

As minimally and precisely as possible. Keep in mind we do not have access to your cluster or dags.

If you are using kubernetes, please attempt to recreate the issue using minikube or kind.

Install minikube/kind

  • Minikube https://minikube.sigs.k8s.io/docs/start/
  • Kind https://kind.sigs.k8s.io/docs/user/quick-start/

If this is a UI bug, please provide a screenshot of the bug or a link to a youtube video of the bug in action

You can include images using the .md style of alt text

To record a screencast, mac users can use QuickTime and then create an unlisted youtube video with the resulting .mov file.

--->

Anything else we need to know:

<!--

How often does this problem occur? Once? Every time etc?

Any relevant logs to include? Put them here in side a detail tag: <details><summary>x.log</summary> lots of stuff </details>

-->

closed time in 4 hours

Dolfino

issue commentapache/airflow

Windows 10 bug Ubuntu

Nothing here.

Dolfino

comment created time in 4 hours

pull request commentapache/airflow

Adds automated generation of provider issue to track test progress

Saved me couple of hours today already.

potiuk

comment created time in 4 hours

pull request commentapache/airflow

Adds automated generation of provider issue to track test progress

Hey @ashb @kaxil - regardless of the long-term solution, this is is a very nice automation that helped me to manage the provider issue with status of releases - both the first one and the follow up. It's rather nice IMHO and allows to make it manageable with testing status and pinging people automatically to test the release of providers. It's green, I could merge it :)

potiuk

comment created time in 4 hours

issue commentninenines/cowboy

Possible Regression/Leak in Cowboy 2.9

Not willingly, in any case.

You can check if that is the case by tracing exclusively on the init/terminate callbacks. Trace to a file and then check that it is full of init followed by terminate.

jeffutter

comment created time in 4 hours

issue commentapache/airflow

Status of testing Providers that were prepared on June 18, 2021

As new package RCs are released, I encourage those from the list who have not done so, to double-check if the packages are working fine with their changes. The voting time has been prolonged to Tue 22 Jun 21:29:59 CEST 2021.

Looking forward to your thumb-ups :)

potiuk

comment created time in 4 hours

pull request commentocaml/ocaml

stdlib: add `List.is_empty`

I don't think it is fair to compare is_empty with nth, hd and tail. First, this is_empty function is total. It is even an isomorphism between ('a list, @) and (bool, &&), thus it does seem to respect the algebraic nature of lists.

CraigFe

comment created time in 4 hours

Pull request review commentocaml/ocaml

Expose Oprint.escape_string as String.escape

 val escaped : string -> string     [escaped s] fails).      @raise Invalid_argument if the result is longer than-    {!Sys.max_string_length} bytes. *)+    {!Sys.max_string_length} bytes.++    @deprecated Use {!String.escape_ascii} instead. *)++val escape_ascii : string -> string+(** [escape_ascii s] is [s] with all characters outside the printable US-ASCII+    range represented by escape sequences.++    All characters outside the US-ASCII printable range \[0x20;0x7E\] are+    escaped, as well as backslash (0x2F) and double-quote (0x22).++    The function {!Scanf.unescaped} is a left inverse of [escaped],+    i.e. [Scanf.unescaped (escaped s) = s] for any string [s] (unless+    [escaped s] fails).++    @raise Invalid_argument if the result is longer than+    {!Sys.max_string_length} bytes.++    @since 4.14.0 *)++val escape : string -> string+(** [escape s] is [s] with special characters replaced with escape sequences,+    following the lexical conventions of OCaml.++    The following characters are escaped:+    double quote (['\"'], newline (['\n']), carriage return (['\r']),+    tab (['\t\]), backspace (['\b']), DEL (0x7F),+    and everything in the ['\x00' .. '\x1F'] range.++    @raise Invalid_argument if the result is longer than+    {!Sys.max_string_length} bytes.++    @since 4.14.0 *)

I always thought of uppercase as an adjective here, not a verb (like escaped).

dmbaturin

comment created time in 4 hours

pull request commentocaml/ocaml

Generate full error traces during module inclusion

The last commit looks fine. I will have a last glance after this week-end, then I will merge on Monday.

antalsz

comment created time in 4 hours

issue commentninenines/cowboy

Possible Regression/Leak in Cowboy 2.9

I captured the state of two processes (actually the same process twice) that were getting high reductions:

#state{
	parent = <0.4943.0>,
	ref = MyAppWeb.Endpoint.HTTP,
	socket = #Port<0.4067799>,
	transport = :ranch_tcp,
	proxy_header = undefined,
	opts = #{} :: #{env =>
          #{dispatch =>
                [{'_',[],
                     [{'_',[],'Elixir.Phoenix.Endpoint.Cowboy2Handler',
                          {'Elixir.MyApp.Endpoint',[]}}]}]},
      idle_timeout => 70000,max_keepalive => 20000,
      stream_handlers => [cowboy_telemetry_h,cowboy_stream_h]},
	buffer = <<>>,
	overriden_opts = #{},
  peer = {{0,0,0,0,0,65535,44051,33230},1060},
  sock = {{0,0,0,0,0,65535,44049,7},4080},
	cert = undefined,
	timer = #Reference<0.327935682.2040528927.255668>,
	active = true,
	in_streamid = 286,
	in_state = #ps_request_line{empty_lines=0},
	flow = infinity,
	out_streamid = 286,
	out_state = wait,
	last_streamid = 20000,
	streams = [],
	children = []
}
#state{
	parent = <0.4943.0>,
	ref = MyAppWeb.Endpoint.HTTP,
	socket = #Port<0.3982394>,
	transport = :ranch_tcp,
	proxy_header = undefined,
	opts = #{} :: #{env =>
          #{dispatch =>
                [{'_',[],
                     [{'_',[],'Elixir.Phoenix.Endpoint.Cowboy2Handler',
                          {'Elixir.MyApp.Endpoint',[]}}]}]},
      idle_timeout => 70000,max_keepalive => 20000,
      stream_handlers => [cowboy_telemetry_h,cowboy_stream_h]},
	buffer = <<>>,
	overriden_opts = #{},
  peer = {{0,0,0,0,0,65535,44051,33230},54968},
  sock = {{0,0,0,0,0,65535,44049,7},4080},
	cert = undefined,
	timer = #Reference<0.327935682.2040528924.235998>,
	active = true,
	in_streamid = 1962,
	in_state = #ps_request_line{empty_lines=0},
	flow = infinity,
	out_streamid = 1961,
	out_state = wait,
	last_streamid = 20000,
	streams = [#stream{
        id = 1961,
        state = #cowboy_telemetry_h{...},
        method = "POST",
        version = "HTTP/1.1",
        te :: undefined ,
        local_expected_size = undefined,
        local_sent_size = 0 ,
        queue = []
    }],
	children = [#child{
        pid = #PID<0.22534.478>,
        streamid = 1961,
        shutdown = 5000,
        timer = undefined
    }]
}

Nothing really jumps out at me with those two processes. One thing I did notice this time though is that the process with the top number of reductions is often the logger:

iex(socialx@2a4a976d3e85)36> :recon.proc_window(:reductions, 5, 5000)
[
  {#PID<0.4621.0>, 1681468,
   [
     Logger,
     {:current_function, {:gen_event, :fetch_msg, 6}},
     {:initial_call, {:proc_lib, :init_p, 5}}
   ]},
  {#PID<0.29288.479>, 205943,
   [
     current_function: {:cowboy_http, :loop, 1},
     initial_call: {:proc_lib, :init_p, 5}
   ]},
   ...
]

This probably correlates with a massive uptick we're seeing in error logs coming from our telemetry code. Our current telemetry setup is: cowboy -> cowboy_telemetry -> telemetry -> some internal glue code -> Spandex

Spandex has an error log that we hit a lot during these:

Logger.error("Tried to start a trace over top of another trace. name: #{inspect(name)}")

https://github.com/spandex-project/spandex/blob/d14f78ce416ccedc2b16e0acd1927809e626ca30/lib/spandex.ex#L40

The error happens when there is already an open Spandex trace in the process dict (which I believe would be the cowboy_http process).

We attempt to continue the trace as a result of the telemetry event [cowboy, request, start] which cowboy_telemetry fires from the init call of the cowboy_telemetry_h stream handler.

All this brings me to ask - did something change in 2.9 that would cause init function of a handler to be called multiple times or the metrics_callback (which stops the trace) to not be called before the next init is called?

jeffutter

comment created time in 4 hours

Pull request review commentocaml/ocaml

Generate full error traces during module inclusion

 type equality_error =  type moregen_error = { trace : comparison error } [@@unboxed] +let unification_error ~trace : unification_error =+  if trace = []+  then Misc.fatal_error "Unification error trace was empty"

After pondering, this is more my personal opinion but I would say:

  • assert to enforce invariants, and it is the responsibility of the callers to respect those.
  • fatal_error to protect downstream users from an erroneous state that must not be propagated.
antalsz

comment created time in 4 hours