profile
viewpoint
nodejs Spring Boot React Rust tensorflow
If you are wondering where the data of this site comes from, please visit https://api.github.com/users/thedrow/events. GitMemory does not store any data, but only uses NGINX to cache data for a period of time. The idea behind GitMemory is simply to give users a better reading experience.
Omer Katz thedrow Israel http://omerkatz.com

repos

493

gists

94

followers

246

following

1543

not-kennethreitz/envoy 2255

Python Subprocesses for Humans™.

oauthlib/oauthlib 2246

A generic, spec-compliant, thorough implementation of the OAuth request-signing logic

stefanfoulis/django-phonenumber-field 1112

A django model and form field for normalised phone numbers using python-phonenumbers

bear/parsedatetime 611

Parse human-readable date/time strings

azazel75/macropy 30

Macros in Python: quasiquotes, case classes, LINQ and more!

0chroma/question-party 4

questionparty.com doesn't seem to exist anymore, but we liked the idea, so we decided to redo it.

pytransitions/transitions-anyio 3

An extension for the transitions state machine library which runs asynchronous state machines using anyio.

celery/pytest-celery 2

celery/bootsteps 0

Program Initialization Toolkit

jheld/kombu 0

Messaging library for Python.

PR closed celery/celery

Reviewers
Migrate to Poetry Category: Project Governance PR Type: Enhancement

Note: Before submitting this pull request, please review our contributing guidelines.

Description

We want to migrate to Poetry so that we'll be able to use pyproject.toml and get rid of most of our requirement files.

<!-- Please describe your pull request.

NOTE: All patches should be made against master, not a maintenance branch like 3.1, 2.5, etc. That is unless the bug is already fixed in master, but not in that version series.

If it fixes a bug or resolves a feature request, be sure to link to that issue via (Fixes #4412) for example. -->

+110 -199
1 commit in celery/celery/tree/poetry celery/celery/tree/master

6 comments

3 changed files

thedrow

pr closed time in 20 hours

pull request commentcelery/celery

Remove celery.task references in modules, docs

Could you please address the rest of the review's comments so we can merge this? Thanks!

ferdnyc

comment created time in 21 hours

PullRequestReviewEvent

Pull request review commentcelery/celery

test on win32 py3.9 with pycurl windows wheels from https://www.lfd.uci.edu/~gohlke/pythonlibs/ 

 commands =     unit: pytest --maxfail=10 -v --cov=celery --cov-report=xml --cov-report term {posargs}     integration: pytest -xsv t/integration {posargs} setenv =+    PIP_EXTRA_INDEX_URL=https://graingert.co.uk/celery-wheelhouse/repo/simple/

I guess being explicit is better so I like the original name.

graingert

comment created time in 21 hours

PullRequestReviewEvent

issue closedcelery/celery

soft_time_limit invalid

I am learning to use celery.

System version: macOS 11.3.1 (20E241)

Python  3.8.8
celery  5.1.2

When I run the following code and wait for it to finish, I repeat the operation several times (random times) and find that soft_time_limit does not work in the celery output log.

code:

from celery import Celery, group
from celery.exceptions import SoftTimeLimitExceeded
import time
import logging
app = Celery(
    "celery_app",
    broker="redis://localhost",
    backend="redis://localhost",
)
logger = logging.getLogger(__name__)


@app.task(bind=True, soft_time_limit=2)
def demo(self, n):
    try:
        for i in range(5):
            time.sleep(1)
            print(f"No.{self.request.id}: {i}")
    except SoftTimeLimitExceeded as e:
        logger.warn(str(e))


if __name__ == '__main__':
    tasks = []
    for a in range(1):
        tasks.append(demo.s(a))

    result = group(tasks).apply_async()
    result.join()

celery log:

❯ celery -A celery_app.app worker -c 10 -l info
 
 -------------- [email protected] v5.1.2 (sun-harmonics)
--- ***** ----- 
-- ******* ---- macOS-10.16-x86_64-i386-64bit 2021-07-24 15:35:34
- *** --- * --- 
- ** ---------- [config]
- ** ---------- .> app:         celery_app:0x7ffc26f19040
- ** ---------- .> transport:   redis://localhost:6379//
- ** ---------- .> results:     redis://localhost/
- *** --- * --- .> concurrency: 10 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** ----- 
 -------------- [queues]
                .> celery           exchange=celery(direct) key=celery
                

[tasks]
  . celery_app.demo

[2021-07-24 15:35:34,929: INFO/MainProcess] Connected to redis://localhost:6379//
[2021-07-24 15:35:34,936: INFO/MainProcess] mingle: searching for neighbors
[2021-07-24 15:35:35,952: INFO/MainProcess] mingle: all alone
[2021-07-24 15:35:35,964: INFO/MainProcess] [email protected] ready.
[2021-07-24 15:35:37,340: INFO/MainProcess] Task celery_app.demo[7e0245e0-797a-42e6-85dd-547de39c63fb] received
[2021-07-24 15:35:38,343: WARNING/ForkPoolWorker-8] No.7e0245e0-797a-42e6-85dd-547de39c63fb: 0
[2021-07-24 15:35:38,344: WARNING/ForkPoolWorker-8] 

[2021-07-24 15:35:39,345: WARNING/MainProcess] Soft time limit (2s) exceeded for celery_app.demo[7e0245e0-797a-42e6-85dd-547de39c63fb]
[2021-07-24 15:35:39,345: WARNING/ForkPoolWorker-8] No.7e0245e0-797a-42e6-85dd-547de39c63fb: 1
[2021-07-24 15:35:39,346: WARNING/ForkPoolWorker-8] SoftTimeLimitExceeded()
[2021-07-24 15:35:39,360: INFO/ForkPoolWorker-8] Task celery_app.demo[7e0245e0-797a-42e6-85dd-547de39c63fb] succeeded in 2.0180346350000002s: None
[2021-07-24 15:35:43,045: INFO/MainProcess] Task celery_app.demo[dfa18112-38f6-4e0e-a952-78967d9cfcb8] received
[2021-07-24 15:35:44,046: WARNING/ForkPoolWorker-8] No.dfa18112-38f6-4e0e-a952-78967d9cfcb8: 0
[2021-07-24 15:35:44,047: WARNING/ForkPoolWorker-8] 

[2021-07-24 15:35:45,048: WARNING/ForkPoolWorker-8] No.dfa18112-38f6-4e0e-a952-78967d9cfcb8: 1
[2021-07-24 15:35:45,049: WARNING/ForkPoolWorker-8] 

[2021-07-24 15:35:45,049: WARNING/MainProcess] Soft time limit (2s) exceeded for celery_app.demo[dfa18112-38f6-4e0e-a952-78967d9cfcb8]
[2021-07-24 15:35:45,050: WARNING/ForkPoolWorker-8] SoftTimeLimitExceeded()
[2021-07-24 15:35:45,050: INFO/ForkPoolWorker-8] Task celery_app.demo[dfa18112-38f6-4e0e-a952-78967d9cfcb8] succeeded in 2.0052167459999986s: None
[2021-07-24 15:35:46,897: INFO/MainProcess] Task celery_app.demo[34bbc48f-3723-42b0-8f0c-f20132a17029] received
[2021-07-24 15:35:47,898: WARNING/ForkPoolWorker-8] No.34bbc48f-3723-42b0-8f0c-f20132a17029: 0
[2021-07-24 15:35:47,898: WARNING/ForkPoolWorker-8] 

[2021-07-24 15:35:48,899: WARNING/ForkPoolWorker-8] No.34bbc48f-3723-42b0-8f0c-f20132a17029: 1
[2021-07-24 15:35:48,900: WARNING/ForkPoolWorker-8] 

[2021-07-24 15:35:48,901: WARNING/MainProcess] Soft time limit (2s) exceeded for celery_app.demo[34bbc48f-3723-42b0-8f0c-f20132a17029]
[2021-07-24 15:35:48,901: WARNING/ForkPoolWorker-8] SoftTimeLimitExceeded()
[2021-07-24 15:35:48,904: INFO/ForkPoolWorker-8] Task celery_app.demo[34bbc48f-3723-42b0-8f0c-f20132a17029] succeeded in 2.006053979999999s: None
[2021-07-24 15:35:50,743: INFO/MainProcess] Task celery_app.demo[ef742335-6e4e-4599-aa36-724c6e991a9e] received
[2021-07-24 15:35:51,744: WARNING/ForkPoolWorker-8] No.ef742335-6e4e-4599-aa36-724c6e991a9e: 0
[2021-07-24 15:35:51,745: WARNING/ForkPoolWorker-8] 

[2021-07-24 15:35:52,747: WARNING/MainProcess] Soft time limit (2s) exceeded for celery_app.demo[ef742335-6e4e-4599-aa36-724c6e991a9e]
[2021-07-24 15:35:52,747: WARNING/ForkPoolWorker-8] No.ef742335-6e4e-4599-aa36-724c6e991a9e: 1
[2021-07-24 15:35:52,747: WARNING/ForkPoolWorker-8] SoftTimeLimitExceeded()
[2021-07-24 15:35:52,748: INFO/ForkPoolWorker-8] Task celery_app.demo[ef742335-6e4e-4599-aa36-724c6e991a9e] succeeded in 2.0043429249999996s: None
[2021-07-24 15:35:54,657: INFO/MainProcess] Task celery_app.demo[6f13d05c-a189-4730-91bd-c57f0785bd1c] received
[2021-07-24 15:35:55,658: WARNING/ForkPoolWorker-8] No.6f13d05c-a189-4730-91bd-c57f0785bd1c: 0
[2021-07-24 15:35:55,658: WARNING/ForkPoolWorker-8] 

[2021-07-24 15:35:56,660: WARNING/ForkPoolWorker-8] No.6f13d05c-a189-4730-91bd-c57f0785bd1c: 1
[2021-07-24 15:35:56,661: WARNING/ForkPoolWorker-8] 

[2021-07-24 15:35:56,662: WARNING/MainProcess] Soft time limit (2s) exceeded for celery_app.demo[6f13d05c-a189-4730-91bd-c57f0785bd1c]
[2021-07-24 15:35:56,663: WARNING/ForkPoolWorker-8] SoftTimeLimitExceeded()
[2021-07-24 15:35:56,665: INFO/ForkPoolWorker-8] Task celery_app.demo[6f13d05c-a189-4730-91bd-c57f0785bd1c] succeeded in 2.007433097s: None
[2021-07-24 15:35:58,442: INFO/MainProcess] Task celery_app.demo[540b2b53-8589-4afc-8094-0e961aba5af8] received
[2021-07-24 15:35:59,443: WARNING/ForkPoolWorker-8] No.540b2b53-8589-4afc-8094-0e961aba5af8: 0
[2021-07-24 15:35:59,444: WARNING/ForkPoolWorker-8] 

[2021-07-24 15:36:00,445: WARNING/MainProcess] Soft time limit (2s) exceeded for celery_app.demo[540b2b53-8589-4afc-8094-0e961aba5af8]
[2021-07-24 15:36:00,476: WARNING/ForkPoolWorker-8] 

[2021-07-24 15:36:01,477: WARNING/ForkPoolWorker-8] No.540b2b53-8589-4afc-8094-0e961aba5af8: 2
[2021-07-24 15:36:01,478: WARNING/ForkPoolWorker-8] 

[2021-07-24 15:36:02,479: WARNING/ForkPoolWorker-8] No.540b2b53-8589-4afc-8094-0e961aba5af8: 3
[2021-07-24 15:36:02,479: WARNING/ForkPoolWorker-8] 

[2021-07-24 15:36:03,481: WARNING/ForkPoolWorker-8] No.540b2b53-8589-4afc-8094-0e961aba5af8: 4
[2021-07-24 15:36:03,481: WARNING/ForkPoolWorker-8] 

[2021-07-24 15:36:03,482: INFO/ForkPoolWorker-8] Task celery_app.demo[540b2b53-8589-4afc-8094-0e961aba5af8] succeeded in 5.039028163999998s: None

closed time in 3 days

pppigg

issue commentcelery/celery

soft_time_limit invalid

As @f0xxx1 already explained, this works as expected.

pppigg

comment created time in 3 days

startedvlado/activerecord-cte

started time in 3 days

startedortelius/ortelius

started time in 3 days

startedschemahero/schemahero

started time in 3 days

startedrjyo/next-boost

started time in 4 days

startedvercel/swr

started time in 4 days

startedquickwit-inc/quickwit

started time in 4 days

starteddermesser/leveldb-rs

started time in 4 days

startedquirrel-dev/owl

started time in 5 days

Pull request review commentcelery/celery

Remove celery.task references in modules, docs 

 chain breaks:      .. code-block:: python -        from celery.task import task+        from celery import app -        @task(queue='hipri')+        @app.task(queue='hipri')         def hello(to):             return 'hello {0}'.format(to)  Abstract Tasks ============== -All tasks created using the :meth:`~@task` decorator+All tasks created using the :meth:`[email protected]` decorator

Does this render properly?

ferdnyc

comment created time in 5 days

Pull request review commentcelery/celery

Remove celery.task references in modules, docs 

 format:  .. code-block:: javascript +    // XXX: No longer applicable?

The response is still the same but the ping task is elsewhere.

ferdnyc

comment created time in 5 days

Pull request review commentcelery/celery

Remove celery.task references in modules, docs 

 Here's an example using Celery in single-mode:  .. code-block:: python +    # XXX: No longer valid?

Yes this is no longer valid.

ferdnyc

comment created time in 5 days

PullRequestReviewEvent
PullRequestReviewEvent
PullRequestReviewEvent
PullRequestReviewEvent

Pull request review commentcelery/celery

Add message properties to app.tasks.Context (#6777) 

 def __init__(self, message, on_ack=noop,                  maybe_make_aware=maybe_make_aware,                  maybe_iso8601=maybe_iso8601, **opts):         self._message = message-        self._request_dict = message.headers if headers is None else headers+        self._request_dict = (message.headers.copy() if headers is None

Why are we copying the headers?

InvalidInterrupt

comment created time in 5 days

PullRequestReviewEvent

push eventcelery/celery

Thomas Grainger

commit sha ea1df2ba82e2492657c2e6c512f85a188ecdec18

import celery lazilly in pytest plugin and unignore flake8 F821, "undefined name '...'" (#6872) * unignore f821 * defer celery imports in celery pytest plugin

view details

push time in 5 days

PR merged celery/celery

import celery lazilly in pytest plugin and unignore flake8 F821, "undefined name '...'"

fixes #6283

+45 -26
2 commit in graingert/celery/tree/unignore-f821 celery/celery/tree/master

3 comments

13 changed files

graingert

pr closed time in 5 days

issue closedcelery/celery

unignore `F821, # undefined name '...'`

F821 is one of the most useful flake8 rules, however it's currently disabled because type comments reference undefined names.

it can be fixed with a conditional dependency on typing or importing typing in an if False: block, see discussion here: https://github.com/PyCQA/pyflakes/issues/501

conditional dep

# requirements.txt
typing; python_version < '3.5'

# some_annotated.py
from typing import Any, Tuple

if typing.TYPE_CHECKING:
    from celery import Celery

if False:

# some_annotated.py

if False:
    from typing import Any, Tuple
    from celery import Celery

@auvipy @thedrow do you have a preferred typing import approach? mine is to add the conditional dep

<!-- Please fill this template entirely and do not erase parts of it. We reserve the right to close without a response enhancement requests which are incomplete. -->

Checklist

<!-- To check an item on the list replace [ ] with [x]. -->

  • [ ] I have checked the issues list for similar or identical enhancement to an existing feature.
  • [ ] I have checked the pull requests list for existing proposed enhancements.
  • [ ] I have checked the commit log to find out if the if the same enhancement was already implemented in the master branch.
  • [ ] I have included all related issues and possible duplicate issues in this issue (If there are none, check this box anyway).

Related Issues and Possible Duplicates

<!-- Please make sure to search and mention any related issues or possible duplicates to this issue as requested by the checklist above.

This may or may not include issues in other repositories that the Celery project maintains or other repositories that are dependencies of Celery.

If you don't know how to mention issues, please refer to Github's documentation on the subject: https://help.github.com/en/articles/autolinked-references-and-urls#issues-and-pull-requests -->

Related Issues

  • None

Possible Duplicates

  • None

Brief Summary

<!-- Please include a brief summary of what the enhancement is and why it is needed. -->

Design

Architectural Considerations

<!-- If more components other than Celery are involved, describe them here and the effect it would have on Celery. --> None

Proposed Behavior

<!-- Please describe in detail how this enhancement is going to change the behavior of an existing feature. Describe what happens in case of failures as well if applicable. -->

Proposed UI/UX

<!-- Please provide your ideas for the API, CLI options, configuration key names etc. that will be adjusted for this enhancement. -->

Diagrams

<!-- Please include any diagrams that might be relevant to the implementation of this enhancement such as:

  • Class Diagrams
  • Sequence Diagrams
  • Activity Diagrams You can drag and drop images into the text box to attach them to this issue. --> N/A

Alternatives

<!-- If you have considered any alternative implementations describe them in detail below. --> None

closed time in 5 days

graingert
PullRequestReviewEvent
PullRequestReviewEvent

pull request commentcelery/celery

Fix UnboundLocalError when running celery worker in foreground

@maybe-sybr Thanks for helping me with the investigation. ❤️

michael-k

comment created time in 6 days