profile
viewpoint
If you are wondering where the data of this site comes from, please visit https://api.github.com/users/mik-laj/events. GitMemory does not store any data, but only uses NGINX to cache data for a period of time. The idea behind GitMemory is simply to give users a better reading experience.
Kamil Breguła mik-laj @Snowflakedb Warsaw https://snowflake.com Software Engineer at @snowflakedb | Apache Airflow PMC Member & Committer

GoogleCloudPlatform/oozie-to-airflow 69

Oozie Workflow to Airflow DAGs migration tool

ad-m/petycja-faoo 0

Aplikacja petycji Fundacji AkademiiOrganizacjiObywatelskich w sprawie projektu nowelizacji ustawy o działalności pożytku publicznego i o wolontariacie

mik-laj/AdminLTE 0

Pi-hole Dashboard for stats and more

mik-laj/agata 0

A simple UI layer for visualizing Google Analytics v4 hits on Android apps

mik-laj/agenda-editor 0

Agenda Editor

mik-laj/airflow 0

Apache Airflow (Incubating)

push eventPolideaInternal/airflow

Kristen Malik

commit sha c955078b22ad797a48baf690693454c9b8ba516d

Added theScore to the list of companies using Apache Airflow (#19253) * Added theScore to the list of companies using Apache Airflow * Add space between Org name and GitHub handle Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com>

view details

Bhavani Ravi

commit sha fd569e714403176770b26cf595632812bd384bc0

#19223 add mongo_db param to MongoSensor (#19276)

view details

Xuan

commit sha 55abc2f620a96832661d1797442a834bf958bb3e

Support query timeout as an argument in CassandraToGCSOperator (#18927) Support query timeout as an argument in CassandraToGCSOperator (#18927)

view details

Edith Puclla

commit sha 8f5c222170c35d03320b7241fc4df7d40fe29238

Modify doc contributing (#19124)

view details

push time in 2 hours

push eventmik-laj/airflow

Kristen Malik

commit sha c955078b22ad797a48baf690693454c9b8ba516d

Added theScore to the list of companies using Apache Airflow (#19253) * Added theScore to the list of companies using Apache Airflow * Add space between Org name and GitHub handle Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com>

view details

Bhavani Ravi

commit sha fd569e714403176770b26cf595632812bd384bc0

#19223 add mongo_db param to MongoSensor (#19276)

view details

Xuan

commit sha 55abc2f620a96832661d1797442a834bf958bb3e

Support query timeout as an argument in CassandraToGCSOperator (#18927) Support query timeout as an argument in CassandraToGCSOperator (#18927)

view details

Edith Puclla

commit sha 8f5c222170c35d03320b7241fc4df7d40fe29238

Modify doc contributing (#19124)

view details

push time in 2 hours

push eventapache/airflow

Kristen Malik

commit sha c955078b22ad797a48baf690693454c9b8ba516d

Added theScore to the list of companies using Apache Airflow (#19253) * Added theScore to the list of companies using Apache Airflow * Add space between Org name and GitHub handle Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com>

view details

push time in 5 hours

PR merged apache/airflow

Added theScore to the list of companies using Apache Airflow okay to merge

Added theScore to INTHEWILD.md

+1 -0

2 comments

1 changed file

kristenmalikk

pr closed time in 5 hours

issue closedapache/airflow

Airflow Docker Compose Network Issue

Apache Airflow version

2.2.0 (latest released)

Operating System

Debian 11

Versions of Apache Airflow Providers

kafka-python==2.0.2 apache-airflow-providers-apache-spark==2.0.1

Deployment

Docker-Compose

Deployment details

  • Docker version 20.10.10, build b485636
  • docker-compose version 1.29.2, build 5becea4c
  • Debian GNU/Linux 11 (bullseye)

I'm using the official docker-compose file here: https://airflow.apache.org/docs/apache-airflow/2.2.0/docker-compose.yaml

I am using the build option to load a couple python packages. The Dockerfile looks like this:

FROM apache/airflow:2.2.0

RUN pip install --no-cache-dir kafka-python==2.0.2
RUN pip install --no-cache-dir apache-airflow-providers-apache-spark==2.0.1
RUN pip install --upgrade apache-airflow-providers-google

The only modification I've made to the compose file is to add this at the end:

networks:
  default:
    external: true
    name: ti-network

What happened

This works totally fine on my macbook, but when I try to run this on Debian 11 I get the following error:

[2021-10-28 02:53:56,997] {cli_action_loggers.py:105} WARNING - Failed to log action with (psycopg2.OperationalError) FATAL: password authentication failed for user "airflow"

If I remove the network block, it builds totally fine on Debian 11 too without error.

What you expected to happen

I expect to be able to use the external docker network. I need it as Airflow will be calling other containers on the same docker network such as a Kafka container.

How to reproduce

Add the external network block to the official docker-compose file: https://airflow.apache.org/docs/apache-airflow/2.2.0/docker-compose.yaml

Run it on a debian system:

docker-compose build
docker-compose up airflow-init 
docker-compose up

Anything else

Here is the full error log:

Attaching to airflow_postgres_1, airflow_redis_1, airflow_airflow-init_1, airflow_airflow-webserver_1, airflow_flower_1, airflow_airflow-worker_1, airflow_airflow-triggerer_1, airflow_airflow-scheduler_1
airflow-init_1       | The container is run as root user. For security, consider using a regular user account.
airflow-init_1       | .
airflow-init_1       | [2021-10-28 02:53:56,997] {cli_action_loggers.py:105} WARNING - Failed to log action with (psycopg2.OperationalError) FATAL:  password authentication failed for user "airflow"
airflow-init_1       | 
airflow-init_1       | (Background on this error at: http://sqlalche.me/e/13/e3q8)
airflow-init_1       | DB: postgresql+psycopg2://airflow:***@postgres/airflow
airflow-init_1       | Traceback (most recent call last):
airflow-init_1       |   File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2336, in _wrap_pool_connect
airflow-init_1       |     return fn()
airflow-init_1       |   File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 304, in unique_connection
airflow-init_1       |     return _ConnectionFairy._checkout(self)
airflow-init_1       |   File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 778, in _checkout
airflow-init_1       |     fairy = _ConnectionRecord.checkout(pool)
airflow-init_1       |   File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 495, in checkout
airflow-init_1       |     rec = pool._do_get()
airflow-init_1       |   File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/impl.py", line 140, in _do_get
airflow-init_1       |     self._dec_overflow()
airflow-init_1       |   File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
airflow-init_1       |     with_traceback=exc_tb,
airflow-init_1       |   File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
airflow-init_1       |     raise exception
airflow-init_1       |   File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/impl.py", line 137, in _do_get
airflow-init_1       |     return self._create_connection()
airflow-init_1       |   File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 309, in _create_connection
airflow-init_1       |     return _ConnectionRecord(self)
airflow-init_1       |   File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 440, in __init__
airflow-init_1       |     self.__connect(first_connect_check=True)
airflow-init_1       |   File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 661, in __connect
airflow-init_1       |     pool.logger.debug("Error on connect(): %s", e)
airflow-init_1       |   File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
airflow-init_1       |     with_traceback=exc_tb,
airflow-init_1       |   File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
airflow-init_1       |     raise exception
airflow-init_1       |   File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 656, in __connect
airflow-init_1       |     connection = pool._invoke_creator(self)
airflow-init_1       |   File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
airflow-init_1       |     return dialect.connect(*cargs, **cparams)
airflow-init_1       |   File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 508, in connect
airflow-init_1       |     return self.dbapi.connect(*cargs, **cparams)
airflow-init_1       |   File "/home/airflow/.local/lib/python3.6/site-packages/psycopg2/__init__.py", line 122, in connect
airflow-init_1       |     conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
airflow-init_1       | psycopg2.OperationalError: FATAL:  password authentication failed for user "airflow"
airflow-init_1       | 
airflow-init_1       | 
airflow-init_1       | The above exception was the direct cause of the following exception:
airflow-init_1       | 
airflow-init_1       | Traceback (most recent call last):
airflow-init_1       |   File "/home/airflow/.local/bin/airflow", line 8, in <module>
airflow-init_1       |     sys.exit(main())
airflow-init_1       |   File "/home/airflow/.local/lib/python3.6/site-packages/airflow/__main__.py", line 40, in main
airflow-init_1       |     args.func(args)
airflow-init_1       |   File "/home/airflow/.local/lib/python3.6/site-packages/airflow/cli/cli_parser.py", line 48, in command
airflow-init_1       |     return func(*args, **kwargs)
airflow-init_1       |   File "/home/airflow/.local/lib/python3.6/site-packages/airflow/utils/cli.py", line 92, in wrapper
airflow-init_1       |     return f(*args, **kwargs)
airflow-init_1       |   File "/home/airflow/.local/lib/python3.6/site-packages/airflow/cli/commands/db_command.py", line 48, in upgradedb
airflow-init_1       |     db.upgradedb()
airflow-init_1       |   File "/home/airflow/.local/lib/python3.6/site-packages/airflow/utils/session.py", line 70, in wrapper
airflow-init_1       |     return func(*args, session=session, **kwargs)
airflow-init_1       |   File "/home/airflow/.local/lib/python3.6/site-packages/airflow/utils/db.py", line 813, in upgradedb
airflow-init_1       |     for err in _check_migration_errors(session=session):
airflow-init_1       |   File "/home/airflow/.local/lib/python3.6/site-packages/airflow/utils/db.py", line 799, in _check_migration_errors
airflow-init_1       |     yield from check_fn(session)
airflow-init_1       |   File "/home/airflow/.local/lib/python3.6/site-packages/airflow/utils/db.py", line 705, in check_run_id_null
airflow-init_1       |     metadata.reflect(only=["dag_run"])
airflow-init_1       |   File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/sql/schema.py", line 4559, in reflect
airflow-init_1       |     with bind.connect() as conn:
airflow-init_1       |   File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2263, in connect
airflow-init_1       |     return self._connection_cls(self, **kwargs)
airflow-init_1       |   File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 104, in __init__
airflow-init_1       |     else engine.raw_connection()
airflow-init_1       |   File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2370, in raw_connection
airflow-init_1       |     self.pool.unique_connection, _connection
airflow-init_1       |   File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2340, in _wrap_pool_connect
airflow-init_1       |     e, dialect, self
airflow-init_1       |   File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1584, in _handle_dbapi_exception_noconnection
airflow-init_1       |     sqlalchemy_exception, with_traceback=exc_info[2], from_=e
airflow-init_1       |   File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
airflow-init_1       |     raise exception
airflow-init_1       |   File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2336, in _wrap_pool_connect
airflow-init_1       |     return fn()
airflow-init_1       |   File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 304, in unique_connection
airflow-init_1       |     return _ConnectionFairy._checkout(self)
airflow-init_1       |   File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 778, in _checkout
airflow-init_1       |     fairy = _ConnectionRecord.checkout(pool)
airflow-init_1       |   File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 495, in checkout
airflow-init_1       |     rec = pool._do_get()
airflow-init_1       |   File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/impl.py", line 140, in _do_get
airflow-init_1       |     self._dec_overflow()
airflow-init_1       |   File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
airflow-init_1       |     with_traceback=exc_tb,
airflow-init_1       |   File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
airflow-init_1       |     raise exception
airflow-init_1       |   File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/impl.py", line 137, in _do_get
airflow-init_1       |     return self._create_connection()
airflow-init_1       |   File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 309, in _create_connection
airflow-init_1       |     return _ConnectionRecord(self)
airflow-init_1       |   File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 440, in __init__
airflow-init_1       |     self.__connect(first_connect_check=True)
airflow-init_1       |   File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 661, in __connect
airflow-init_1       |     pool.logger.debug("Error on connect(): %s", e)
airflow-init_1       |   File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
airflow-init_1       |     with_traceback=exc_tb,
airflow-init_1       |   File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
airflow-init_1       |     raise exception
airflow-init_1       |   File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 656, in __connect
airflow-init_1       |     connection = pool._invoke_creator(self)
airflow-init_1       |   File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
airflow-init_1       |     return dialect.connect(*cargs, **cparams)
airflow-init_1       |   File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 508, in connect
airflow-init_1       |     return self.dbapi.connect(*cargs, **cparams)
airflow-init_1       |   File "/home/airflow/.local/lib/python3.6/site-packages/psycopg2/__init__.py", line 122, in connect
airflow-init_1       |     conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
airflow-init_1       | sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) FATAL:  password authentication failed for user "airflow"
airflow-init_1       | 
airflow-init_1       | (Background on this error at: http://sqlalche.me/e/13/e3q8)

Are you willing to submit PR?

  • [x] Yes I am willing to submit a PR!

Code of Conduct

closed time in 6 hours

buffaloDeveloper

issue commentapache/airflow

Airflow Docker Compose Network Issue

DO NOT expect the Docker Compose below will be enough to run production-ready Docker Compose Airflow installation using it. This is a truly quick-start docker-compose for you to get Airflow up and running locally and get your hands dirty with Airflow. Configuring a Docker-Compose installation that is ready for production requires an intrinsic knowledge of Docker Compose, a lot of customization, and possibly even writing the Docker Compose file that will suit your needs from the scratch. It's probably OK if you want to run Docker Compose-based deployment, but short of becoming a Docker Compose expert, it's highly unlikely you will get robust deployment with it.

If you want to get an easy to configure Docker-based deployment that Airflow Community develops, supports and can provide support with deployment, you should consider using Kubernetes and deploying Airflow using Official Airflow Community Helm Chart.

buffaloDeveloper

comment created time in 6 hours

issue closedapache/airflow

Airflow Triggerer Pod has AttributeError: module 'asyncio' has no attribute 'run'

Apache Airflow version

2.2.0 (latest released)

Operating System

Docker Image

Versions of Apache Airflow Providers

No response

Deployment

Official Apache Airflow Helm Chart

Deployment details

1.2.0 helm chart and 2.2.0 Airflow

What happened

I see scheduler pods and new triggerer pods have restarts. Just upgraded from 2.1.4 to 2.2.0.

In the triggerer pod logs:


____ |( )_______ / /________ __ ____ /| |_ /__ / / __ / __ _ | /| / / ___ ___ | / _ / _ / _ / / // / |/ |/ / // |// // // // _/____/|__/ [2021-10-28 04:31:26,124] {triggerer_job.py:101} INFO - Starting the triggerer [2021-10-28 04:36:16,431] {triggerer_job.py:111} INFO - Waiting for triggers to clean up [2021-10-28 04:36:16,431] {triggerer_job.py:117} INFO - Exited trigger loop Exception in thread Thread-1: Traceback (most recent call last): File "/usr/local/lib/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/home/airflow/.local/lib/python3.6/site-packages/airflow/jobs/triggerer_job.py", line 231, in run asyncio.run(self.arun()) # pylint: disable=no-member AttributeError: module 'asyncio' has no attribute 'run'

Scheduler logs:


____ |( )_______ / /________ __ ____ /| |_ /__ / / __ / __ _ | /| / / ___ ___ | / _ / _ / _ / / // / |/ |/ / // |// // // // _/____/|__/ [2021-10-28 04:31:31,031] {scheduler_job.py:555} INFO - Starting the scheduler [2021-10-28 04:31:31,031] {scheduler_job.py:560} INFO - Processing each file at most -1 times [2021-10-28 04:31:31,258] {manager.py:163} INFO - Launched DagFileProcessorManager with pid: 31 [2021-10-28 04:31:31,264] {scheduler_job.py:1066} INFO - Resetting orphaned tasks for active dag runs [2021-10-28 04:31:31,272] {settings.py:52} INFO - Configured default timezone Timezone('UTC') [2021-10-28 04:31:31,276] {settings.py:462} INFO - Loaded airflow_local_settings from /opt/airflow/config/airflow_local_settings.py . Process DagFileProcessor322-Process: Traceback (most recent call last): File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2336, in _wrap_pool_connect return fn() File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 364, in connect return _ConnectionFairy._checkout(self) File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 778, in _checkout fairy = _ConnectionRecord.checkout(pool) File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 495, in checkout rec = pool._do_get() File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/impl.py", line 140, in _do_get self.dec_overflow() File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 70, in exit with_traceback=exc_tb, File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 182, in raise raise exception File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/impl.py", line 137, in _do_get return self._create_connection() File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 309, in _create_connection return _ConnectionRecord(self) File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 440, in init self.__connect(first_connect_check=True) File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 661, in _connect pool.logger.debug("Error on connect(): %s", e) File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 70, in exit with_traceback=exc_tb, File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 182, in raise raise exception File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 656, in __connect connection = pool._invoke_creator(self) File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect return dialect.connect(*cargs, **cparams) File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 508, in connect return self.dbapi.connect(*cargs, **cparams) File "/home/airflow/.local/lib/python3.6/site-packages/psycopg2/init.py", line 122, in connect conn = _connect(dsn, connection_factory=connection_factory, **kwasync) psycopg2.OperationalError: server closed the connection unexpectedly This probably means the server terminated abnormally before or while processing the request.

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/usr/local/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap self.run() File "/usr/local/lib/python3.6/multiprocessing/process.py", line 93, in run self._target(*self._args, **self._kwargs) File "/home/airflow/.local/lib/python3.6/site-packages/airflow/dag_processing/processor.py", line 168, in _run_file_processor callback_requests=callback_requests, File "/home/airflow/.local/lib/python3.6/site-packages/airflow/utils/session.py", line 70, in wrapper return func(*args, session=session, **kwargs) File "/home/airflow/.local/lib/python3.6/site-packages/airflow/dag_processing/processor.py", line 629, in process_file self.deactivate_missing_dags(session, dagbag, file_path) File "/home/airflow/.local/lib/python3.6/site-packages/airflow/dag_processing/processor.py", line 665, in deactivate_missing_dags .update({DagModel.is_active: False}, synchronize_session="fetch") File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 4063, in update update_op.exec() File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/orm/persistence.py", line 1696, in exec self._do_pre_synchronize() File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/orm/persistence.py", line 1806, in _do_pre_synchronize select_stmt, mapper=self.mapper, params=query._params File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 1295, in execute return self._connection_for_bind(bind, close_with_result=True).execute( File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 1151, in _connection_for_bind engine, execution_options File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 433, in _connection_for_bind conn = bind._contextual_connect() File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2302, in _contextual_connect self._wrap_pool_connect(self.pool.connect, None), File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2340, in wrap_pool_connect e, dialect, self File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1584, in handle_dbapi_exception_noconnection sqlalchemy_exception, with_traceback=exc_info[2], from=e File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 182, in raise raise exception File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2336, in _wrap_pool_connect return fn() File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 364, in connect return _ConnectionFairy._checkout(self) File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 778, in _checkout fairy = _ConnectionRecord.checkout(pool) File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 495, in checkout rec = pool._do_get() File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/impl.py", line 140, in _do_get self.dec_overflow() File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 70, in exit with_traceback=exc_tb, File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 182, in raise raise exception File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/impl.py", line 137, in _do_get return self._create_connection() File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 309, in _create_connection return _ConnectionRecord(self) File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 440, in init self.__connect(first_connect_check=True) File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 661, in _connect pool.logger.debug("Error on connect(): %s", e) File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 70, in exit with_traceback=exc_tb, File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 182, in raise raise exception File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 656, in __connect connection = pool._invoke_creator(self) File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect return dialect.connect(*cargs, **cparams) File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 508, in connect return self.dbapi.connect(*cargs, **cparams) File "/home/airflow/.local/lib/python3.6/site-packages/psycopg2/init.py", line 122, in connect conn = _connect(dsn, connection_factory=connection_factory, **kwasync) sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) server closed the connection unexpectedly This probably means the server terminated abnormally before or while processing the request.

(Background on this error at: http://sqlalche.me/e/13/e3q8) [2021-10-28 04:36:17,379] {scheduler_job.py:161} INFO - Exiting gracefully upon receiving signal 15 [2021-10-28 04:36:18,382] {process_utils.py:100} INFO - Sending Signals.SIGTERM to GPID 31 [2021-10-28 04:36:18,889] {process_utils.py:66} INFO - Process psutil.Process(pid=956, status='terminated', started='04:36:17') (956) terminated with exit code None [2021-10-28 04:36:19,108] {process_utils.py:212} INFO - Waiting up to 5 seconds for processes to exit... [2021-10-28 04:36:19,142] {process_utils.py:66} INFO - Process psutil.Process(pid=31, status='terminated', exitcode=0, started='04:31:30') (31) terminated with exit code 0 [2021-10-28 04:36:19,143] {process_utils.py:66} INFO - Process psutil.Process(pid=957, status='terminated', started='04:36:17') (957) terminated with exit code None [2021-10-28 04:36:19,143] {process_utils.py:100} INFO - Sending Signals.SIGTERM to GPID 31 [2021-10-28 04:36:19,144] {scheduler_job.py:614} INFO - Exited execute loop

What you expected to happen

Upgraded from 2.1.4 to 2.2.0. Noticed the new triggerer pod has restarted a few times and scheduler pods has restarted the same number of times. I don't think any dag execution have occurred. Don't believe there was any issues with 2.1.x version like this that I noticed.

How to reproduce

No response

Anything else

No response

Are you willing to submit PR?

  • [ ] Yes I am willing to submit a PR!

Code of Conduct

closed time in 6 hours

kn6405

issue commentapache/airflow

Airflow Triggerer Pod has AttributeError: module 'asyncio' has no attribute 'run'

Please update python version. Fixed by https://github.com/apache/airflow/pull/18926

kn6405

comment created time in 6 hours

pull request commentapache/airflow

Chart: add resources for cleanup and createuser jobs

Yes. Exactly.

Aakcht

comment created time in 6 hours

push eventPolideaInternal/airflow

Tony Zhang

commit sha e4aa377da7fe1801dff04d54c5ed4fd525af94c4

Update BigQueryCreateExternalTableOperator doc and parameters (#18676)

view details

push time in 7 hours

push eventmik-laj/airflow

Tony Zhang

commit sha e4aa377da7fe1801dff04d54c5ed4fd525af94c4

Update BigQueryCreateExternalTableOperator doc and parameters (#18676)

view details

push time in 7 hours

push eventPolideaInternal/airflow

Daniel Standish

commit sha 60fade2c65f0da77c99d6f4fc715330714aee41f

Rename variable `serialize_dag` to `serialized_dag` (#19265) The expression `serialize_dag` represents an action, but the variable is actually a dag that has been _serialized_.

view details

push time in 11 hours

push eventmik-laj/airflow

Daniel Standish

commit sha 60fade2c65f0da77c99d6f4fc715330714aee41f

Rename variable `serialize_dag` to `serialized_dag` (#19265) The expression `serialize_dag` represents an action, but the variable is actually a dag that has been _serialized_.

view details

push time in 11 hours

Pull request review commentapache/airflow

Swtich both PROD and CI images to use virtual environments

 You should be aware, about a few things:   `best practices of Dockerfiles <https://docs.docker.com/develop/develop-images/dockerfile_best-practices/>`_   to make sure your image is lean and small. -* The PyPI dependencies in Apache Airflow are installed in the user library, of the "airflow" user, so-  PIP packages are installed to ``~/.local`` folder as if the ``--user`` flag was specified when running PIP.-  Note also that using ``--no-cache-dir`` is a good idea that can help to make your image smaller.+* Using ``--no-cache-dir`` is a good idea that can help to make your image smaller.++* The PyPI dependencies in Apache Airflow are installed in the ``/.venv`` folder. This is virtualenv where+  airflow and all dependent packages are installed, following latest ``pip`` recommendations and upcoming+  `PEP 668 <https://www.python.org/dev/peps/pep-0668/>`_. The ``PATH`` inside the image is set to point first
  :pep:`668`. The ``PATH`` inside the image is set to point first

I am not sure if it will be of any benefit to us, but Sphinx implemented it for some reason (I believe). See: https://www.sphinx-doc.org/en/master/usage/restructuredtext/roles.html

potiuk

comment created time in 12 hours

PullRequestReviewEvent

Pull request review commentapache/airflow

Swtich both PROD and CI images to use virtual environments

 ENV INSTALL_MYSQL_CLIENT=${INSTALL_MYSQL_CLIENT} \     AIRFLOW_CONSTRAINTS_REFERENCE=${AIRFLOW_CONSTRAINTS_REFERENCE} \     AIRFLOW_CONSTRAINTS_LOCATION=${AIRFLOW_CONSTRAINTS_LOCATION} \     DEFAULT_CONSTRAINTS_BRANCH=${DEFAULT_CONSTRAINTS_BRANCH} \-    PATH=${PATH}:/root/.local/bin \+    PATH=${PATH}:/.venv/bin \     AIRFLOW_PIP_VERSION=${AIRFLOW_PIP_VERSION} \

In the following code block, we need only two variables: INSTALL_MYSQL_CLIENT, INSTALL_MSSQL_CLIENT https://github.com/apache/airflow/blob/6686cf94924bc1397a1e21494025247873ef2234/Dockerfile#L192-L209 We can move all other variables to: https://github.com/apache/airflow/blob/6686cf94924bc1397a1e21494025247873ef2234/Dockerfile#L209-L215

Thanks to this, we will have system dependencies and Airflow dependencies separated by layers, so these layers will be invalidated much less frequently.

potiuk

comment created time in 13 hours

PullRequestReviewEvent

Pull request review commentapache/airflow

Swtich both PROD and CI images to use virtual environments

 ENV INSTALL_MYSQL_CLIENT=${INSTALL_MYSQL_CLIENT} \     AIRFLOW_CONSTRAINTS_REFERENCE=${AIRFLOW_CONSTRAINTS_REFERENCE} \     AIRFLOW_CONSTRAINTS_LOCATION=${AIRFLOW_CONSTRAINTS_LOCATION} \     DEFAULT_CONSTRAINTS_BRANCH=${DEFAULT_CONSTRAINTS_BRANCH} \-    PATH=${PATH}:/root/.local/bin \+    PATH=${PATH}:/.venv/bin \     AIRFLOW_PIP_VERSION=${AIRFLOW_PIP_VERSION} \

I am particularly surprised that AIRFLOW_VERSION is defined in the section below, but AIRFLOW_CONSTRAINTS is defined in this section when AIRFLOW_VERSION is included in the AIRFLOW_CONSTRAINTS.

potiuk

comment created time in 13 hours

PullRequestReviewEvent

Pull request review commentapache/airflow

Swtich both PROD and CI images to use virtual environments

 ENV INSTALL_MYSQL_CLIENT=${INSTALL_MYSQL_CLIENT} \     AIRFLOW_CONSTRAINTS_REFERENCE=${AIRFLOW_CONSTRAINTS_REFERENCE} \     AIRFLOW_CONSTRAINTS_LOCATION=${AIRFLOW_CONSTRAINTS_LOCATION} \     DEFAULT_CONSTRAINTS_BRANCH=${DEFAULT_CONSTRAINTS_BRANCH} \-    PATH=${PATH}:/root/.local/bin \+    PATH=/.venv/bin:${PATH} \     AIRFLOW_PIP_VERSION=${AIRFLOW_PIP_VERSION} \     PIP_PROGRESS_BAR=${PIP_PROGRESS_BAR} \-    # Install Airflow with "--user" flag, so that we can copy the whole .local folder to the final image+    # Install Airflow in a virtual environment, so that we can copy the whole+    # .venv folder to the final image     # from the build image and always in non-editable mode-    AIRFLOW_INSTALL_USER_FLAG="--user" \     AIRFLOW_INSTALL_EDITABLE_FLAG="" \     UPGRADE_TO_NEWER_DEPENDENCIES=${UPGRADE_TO_NEWER_DEPENDENCIES} -COPY scripts/docker/*.sh /scripts/docker/+# Only copy mysql/mssql installation scripts for now - so that changing the other+# scripts which are needed much later will not invalidate the docker layer here+COPY scripts/docker/common.sh scripts/docker/install_mysql.sh scripts/docker/install_mssql.sh \

Do we need common.sh? As far as I can see it is not used by scripts/docker/install_mysql.sh or scripts/docker/install_mssql.sh.

potiuk

comment created time in 14 hours

PullRequestReviewEvent

Pull request review commentapache/airflow

Swtich both PROD and CI images to use virtual environments

 ENV INSTALL_MYSQL_CLIENT=${INSTALL_MYSQL_CLIENT} \     AIRFLOW_CONSTRAINTS_REFERENCE=${AIRFLOW_CONSTRAINTS_REFERENCE} \     AIRFLOW_CONSTRAINTS_LOCATION=${AIRFLOW_CONSTRAINTS_LOCATION} \     DEFAULT_CONSTRAINTS_BRANCH=${DEFAULT_CONSTRAINTS_BRANCH} \-    PATH=${PATH}:/root/.local/bin \+    PATH=${PATH}:/.venv/bin \     AIRFLOW_PIP_VERSION=${AIRFLOW_PIP_VERSION} \

What do you think about moving PIP_* and PATH env variables to section below? https://github.com/apache/airflow/blob/6686cf94924bc1397a1e21494025247873ef2234/Dockerfile#L209-L215

potiuk

comment created time in 14 hours

PullRequestReviewEvent

pull request commentapache/airflow

Chart: add resources for cleanup and createuser jobs

@Aakcht We have reference documentation that allows a Helm / Kubernetes expert to take advantage of this option. However, we lack a guide that will allow a less experienced user to use this feature.

For example, Kubernetes has a page that describes how to create a deployment object. https://kubernetes.io/docs/concepts/workloads/controllers/deployment/ Butt they have a reference page for this object type: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.22/#deployment-v1-apps

Aakcht

comment created time in 14 hours

push eventmik-laj/airflow

Amit Ran

commit sha bce888306f28d0c56c6a45c874a045fa304c90b7

sqlite_default has been hard-coded to /tmp, usegettempdir instead, (#19255) respecting tempdir enviroment variables. See #19208, #19229.

view details

Bazzan

commit sha d51c1495a1da2dc37fa66c1a9d25e2909f1e21ef

Edit permalinks in OpenApi description file (#19244)

view details

deedmitrij

commit sha 20847fdbf8ecd3be394d24d47ce151c26d018ea1

Add value to "namespaceId" of query (#19163)

view details

Étienne Tremblay

commit sha 83eb80d4eccb881775dd44bab19818e50e42270b

SFTP hook to prefer the SSH paramiko key over the key file path (#18988)

view details

push time in 14 hours

push eventPolideaInternal/airflow

Amit Ran

commit sha bce888306f28d0c56c6a45c874a045fa304c90b7

sqlite_default has been hard-coded to /tmp, usegettempdir instead, (#19255) respecting tempdir enviroment variables. See #19208, #19229.

view details

Bazzan

commit sha d51c1495a1da2dc37fa66c1a9d25e2909f1e21ef

Edit permalinks in OpenApi description file (#19244)

view details

deedmitrij

commit sha 20847fdbf8ecd3be394d24d47ce151c26d018ea1

Add value to "namespaceId" of query (#19163)

view details

Étienne Tremblay

commit sha 83eb80d4eccb881775dd44bab19818e50e42270b

SFTP hook to prefer the SSH paramiko key over the key file path (#18988)

view details

push time in 14 hours

pull request commentapache/airflow

Chart: add resources for cleanup and createuser jobs

CC: @simplylizz

Aakcht

comment created time in 15 hours

pull request commentapache/airflow

Chart: add resources for cleanup and createuser jobs

We have a few options for resource configuration, but no documentation about it.

workers.kerberosSidecar.resources
workers.resources
workers.logGroomerSidecar.resources
scheduler.resources
scheduler.logGroomerSidecar.resources
triggerer.resources
migrateDatabaseJob.resources
webserver.resources
flower.resources
statsd.resources
pgbouncer.resources
pgbouncer.metricsExporterSidecar.resources
redis.resources
dags.gitSync.resources

Would you take the time to describe it?

Aakcht

comment created time in 15 hours

push eventapache/airflow

Bazzan

commit sha d51c1495a1da2dc37fa66c1a9d25e2909f1e21ef

Edit permalinks in OpenApi description file (#19244)

view details

push time in 16 hours