profile
viewpoint
Peter Lamut plamut Independent contractor Remote

chaoflow/dicttree 2

dicttree serves as use case for metachao

plamut/eestec.portal 1

The main repository for the eestec.net portal, holding in buildout configs, site policy, content types, email notifications, workflows, template overrides, tests, etc.

plamut/ggrc-core 1

GGRC Core

plamut/Patterns 1

Library for quickly applying rich interaction patterns without the need to write any JavaScript.

plamut/Products.TinyMCE 1

Google Summer of Code 2011 - TinyMCE improvements for Plone integration

NejcZupec/pr-stats 0

Pull requests stats for Github projects

plamut/Booktype 0

Booktype is a free, open source platform that produces beautiful, engaging books formatted for print, Amazon, iBooks and almost any ereader within minutes.

Pull request review commentgoogleapis/python-pubsub

feat: exactly-once changes (draft)

 def ack(self, items: Sequence[requests.AckRequest]) -> None:          # We must potentially split the request into multiple smaller requests         # to avoid the server-side max request size limit.-        ack_ids = (item.ack_id for item in items)+        items_gen = (i for i in items)+        ack_ids_gen = (item.ack_id for item in items)         total_chunks = int(math.ceil(len(items) / _ACK_IDS_BATCH_SIZE))          for _ in range(total_chunks):-            request = gapic_types.StreamingPullRequest(-                ack_ids=itertools.islice(ack_ids, _ACK_IDS_BATCH_SIZE)-            )-            self._manager.send(request)--        # Remove the message from lease management.-        self.drop(items)+            future_reqs_dict = {req.ack_id: req for req in itertools.islice(items_gen, _ACK_IDS_BATCH_SIZE) if req.future}+            requests_completed, requests_to_retry =  self._manager.send_unary_ack(+                ack_ids=list(itertools.islice(ack_ids_gen, _ACK_IDS_BATCH_SIZE)),+                future_reqs_dict=future_reqs_dict)++            # Remove the completed messages from lease management.+            self.drop(requests_completed)++            # retry acks+            # todo: figure out if retry delays/limits are needed+            while requests_to_retry:

OK good then. :+1: For a moment I got worried that we forgot about this. :)

pradn

comment created time in 2 days

PullRequestReviewEvent

Pull request review commentgoogleapis/python-pubsub

feat: exactly-once changes (draft)

 def ack(self, items: Sequence[requests.AckRequest]) -> None:          # We must potentially split the request into multiple smaller requests         # to avoid the server-side max request size limit.-        ack_ids = (item.ack_id for item in items)+        items_gen = (i for i in items)+        ack_ids_gen = (item.ack_id for item in items)         total_chunks = int(math.ceil(len(items) / _ACK_IDS_BATCH_SIZE))          for _ in range(total_chunks):-            request = gapic_types.StreamingPullRequest(-                ack_ids=itertools.islice(ack_ids, _ACK_IDS_BATCH_SIZE)-            )-            self._manager.send(request)--        # Remove the message from lease management.-        self.drop(items)+            future_reqs_dict = {req.ack_id: req for req in itertools.islice(items_gen, _ACK_IDS_BATCH_SIZE) if req.future}+            requests_completed, requests_to_retry =  self._manager.send_unary_ack(+                ack_ids=list(itertools.islice(ack_ids_gen, _ACK_IDS_BATCH_SIZE)),+                future_reqs_dict=future_reqs_dict)++            # Remove the completed messages from lease management.+            self.drop(requests_completed)++            # retry acks+            # todo: figure out if retry delays/limits are needed+            while requests_to_retry:

@pradn This seems to have slipped through, are we sure this will behave gracefully in all cases? Won't it block the thread for too long?

pradn

comment created time in 3 days

PullRequestReviewEvent
PullRequestReviewEvent

Pull request review commentgoogleapis/python-pubsub

feat: exactly-once changes (draft)

 def test__on_response_modifies_ack_deadline():         manager._on_response(response)      dispatcher.modify_ack_deadline.assert_called_once_with(-        [requests.ModAckRequest("ack_1", 18), requests.ModAckRequest("ack_2", 18)]+        [requests.ModAckRequest("ack_1", 18, None), requests.ModAckRequest("ack_2", 18, None)]+    )+++def test__on_response_modifies_ack_deadline_with_exactly_once_min_lease():+    # exactly_once is disabled by default.+    manager, _, dispatcher, leaser, _, scheduler = make_running_manager()+    manager._callback = mock.sentinel.callback++    # make p99 value smaller than exactly_once min lease+    manager.ack_histogram.add(10)++    # adjust message bookkeeping in leaser+    fake_leaser_add(leaser, init_msg_count=0, assumed_msg_size=42)++    # Set up the response with the first set of messages and exactly_once not+    # enabled.+    response1 = gapic_types.StreamingPullResponse(+        received_messages=[+            gapic_types.ReceivedMessage(+                ack_id="ack_1",+                message=gapic_types.PubsubMessage(data=b"foo", message_id="1"),+            ),+            gapic_types.ReceivedMessage(+                ack_id="ack_2",+                message=gapic_types.PubsubMessage(data=b"bar", message_id="2"),+            ),+        ],+        subscription_properties = gapic_types.StreamingPullResponse.SubscriptionProperties(+            exactly_once_delivery_enabled = False+        )++    )++    # Set up the response with the second set of messages and exactly_once enabled.+    response2 = gapic_types.StreamingPullResponse(+        received_messages=[+            gapic_types.ReceivedMessage(+                ack_id="ack_3",+                message=gapic_types.PubsubMessage(data=b"foo", message_id="1"),+            ),+            gapic_types.ReceivedMessage(+                ack_id="ack_4",+                message=gapic_types.PubsubMessage(data=b"bar", message_id="2"),+            ),+        ],+        subscription_properties = gapic_types.StreamingPullResponse.SubscriptionProperties(+            exactly_once_delivery_enabled = True+        )++    )++    dispatcher.assert_has_calls(+        # mod-acks called with histogram-based lease value+        dispatcher.modify_ack_deadline([requests.ModAckRequest("ack_1", 10, None), requests.ModAckRequest("ack_2", 10, None)]),++        # mod-acks called with 60 sec min lease value for exactly_once subscriptions+        dispatcher.modify_ack_deadline([requests.ModAckRequest("ack_3", 60, None), requests.ModAckRequest("ack_4", 60, None)])     ) +    # exactly_once is still disabled b/c subscription_properties says so+    manager._on_response(response1)++    # exactly_once should be enabled after this request b/c subscription_properties says so+    manager._on_response(response2)

Sorry, I might be missing something, which comments were added? What is the role of manager._on_response(...) calls here, what errors/assertions do they trigger?

pradn

comment created time in 3 days

Pull request review commentgoogleapis/python-pubsub

feat: exactly-once changes (draft)

 def test_modify_ack_deadline_splitting_large_payload():      items = [         # use realistic lengths for ACK IDs (max 176 bytes)-        requests.ModAckRequest(ack_id=str(i).zfill(176), seconds=60)+        requests.ModAckRequest(ack_id=str(i).zfill(176), seconds=60, future=None)         for i in range(5001)     ]+    manager.send_unary_modack.return_value = (items, [])     dispatcher_.modify_ack_deadline(items) -    calls = manager.send.call_args_list+    calls = manager.send_unary_modack.call_args_list     assert len(calls) == 3      all_ack_ids = {item.ack_id for item in items}     sent_ack_ids = collections.Counter()      for call in calls:-        message = call.args[0]-        assert message._pb.ByteSize() <= 524288  # server-side limit (2**19)-        sent_ack_ids.update(message.modify_deadline_ack_ids)+        modack_ackids = call[1]["modify_deadline_ack_ids"]+        assert len(modack_ackids) <= dispatcher._ACK_IDS_BATCH_SIZE+        sent_ack_ids.update(modack_ackids)

True, merely summing the individual strings would be a bit off. But what if we actually build a ModifyAckDeadlineRequest instance in the test and check the size of that? Since that's the payload that is sent to the backend?

pradn

comment created time in 3 days

PullRequestReviewEvent
PullRequestReviewEvent

Pull request review commentgoogleapis/python-pubsub

feat: exactly-once changes (draft)

 def ack(self) -> None:                 byte_size=self.size,                 time_to_ack=time_to_ack,                 ordering_key=self.ordering_key,+                future=None             )         ) +    def ack_with_response(self) -> None:+        """Acknowledge the given message.

I agree with 3 or more, while two instances of the same are sometimes borderline. I would personally still go with de-duplication here, but I also understand a different view.

pradn

comment created time in 3 days

Pull request review commentgoogleapis/python-pubsub

feat: exactly-once changes (draft)

 def _schedule_message_on_hold(         assert self._callback is not None         self._scheduler.schedule(self._callback, msg) -    def _send_unary_request(self, request: gapic_types.StreamingPullRequest) -> None:++    def send_unary_ack(self, ack_ids, future_reqs_dict) -> tuple[list[requests.AckRequest], list[requests.AckRequest]]:         """Send a request using a separate unary request instead of over the stream. -        Args:-            request: The stream request to be mapped into unary requests.+        If a RetryError occurs, the manager shutdown is triggered, and the+        error is re-raised.         """-        if request.ack_ids:-            self._client.acknowledge(  # type: ignore-                subscription=self._subscription, ack_ids=list(request.ack_ids)+        assert ack_ids++        success = True+        ack_errors_dict = None+        try:+            self._client.acknowledge(+                subscription=self._subscription, ack_ids=ack_ids+            )+        except exceptions.GoogleAPICallError as exc:+            _LOGGER.debug(+                "Exception while sending unary RPC. This is typically "+                "non-fatal as stream requests are best-effort.",+                exc_info=True,+            )+            ack_errors_dict = _get_ack_errors(exc)+        except exceptions.RetryError as exc:+            _LOGGER.debug(+                "RetryError while sending unary RPC. Waiting on a transient "+                "error resolution for too long, will now trigger shutdown.",+                exc_info=False,             )+            # The underlying channel has been suffering from a retryable error+            # for too long, time to give up and shut the streaming pull down.+            self._on_rpc_done(exc)+            raise -        if request.modify_deadline_ack_ids:+        requests_completed, requests_to_retry = _process_errors(future_reqs_dict, ack_errors_dict)

_process_futures() is a good choice IMO. :+1:

pradn

comment created time in 3 days

PullRequestReviewEvent

push eventplamut/python-pubsub

Peter Lamut

commit sha 7c210c7bc2cf417ab6f9d200aa33f33c8143837d

Disable mypy_samples session until blockers resolved

view details

push time in 3 days

pull request commentgoogleapis/python-pubsub

fix: refactor client classes for safer type checking

@tswast I confused the type-check errors with BigQuery, they were only fixed there.

There are two classes of errors reported by Pub/Sub samples type check:

  • Generated proto types are injected into google.cloud.pubsub_v1.types dynamically at import time, and mypy cannot know about them.
    Module has no attribute "ProtoMessageType"
    
  • Untyped objects from python-api-core - not all of them have been annotated, for example api_core.retry.if_exception_type and api_core.retry.Retry from the retry module.
    Call to untyped function "if_exception_type" in typed context
    

While the latter can be solved by adding the missing annotations to api-core, the former would require quite some acrobatics to make it work.

We would have to avoid dynamically injected content, perhaps by some sort of a preprocessor that would copy/paste the generated types directly into google/cloud/pubsub_v1/types.py. In principle doable, but would require changes to the process and an update to tooling, which is too large a task to fit into this PR... :confused:

cc: @pradn

plamut

comment created time in 3 days

push eventplamut/python-pubsub

Peter Lamut

commit sha 67a12c4873e00cbecaff8b2529d0d3bfbbc23fa7

Remove redundant type hint casts

view details

push time in 3 days

push eventplamut/python-pubsub

Peter Lamut

commit sha 8e6b76b52f76252d52c581994c6333b61a597695

Assure that mypy_samples is not commented out

view details

push time in 3 days

push eventplamut/python-pubsub

gcf-owl-bot[bot]

commit sha 7fd7694d7710742c0b51cb53d4df3f1ea7556301

chore(python): Noxfile recognizes that tests can live in a folder (#566) Source-Link: https://github.com/googleapis/synthtool/commit/4760d8dce1351d93658cb11d02a1b7ceb23ae5d7 Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:f0e4b51deef56bed74d3e2359c583fc104a8d6367da3984fc5c66938db738828 Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>

view details

Peter Lamut

commit sha 1e7bd04ea4d09eecf477e39d9459720bdaa1d212

Merge branch 'main' into iss-536-refactor-composition

view details

push time in 3 days

pull request commentgoogleapis/python-pubsub

fix: refactor client classes for safer type checking

Getting a few mypy errors on the samples

This used to pass, will see what changes are needed to bring it back to green state.

plamut

comment created time in 3 days

Pull request review commentgoogleapis/python-pubsub

fix: refactor client classes for safer type checking

 def mypy(session): # ---------------------------------------------------------------------------- s.replace(     "noxfile.py",-    r'"pytype",',+    r'    "mypy",',     '\g<0>\n    # "mypy_samples",  # TODO: uncomment when the checks pass',

Indeed, will remove it so that owlbot will not add it back.

plamut

comment created time in 3 days

PullRequestReviewEvent

pull request commentgoogleapis/python-pubsub

fix: refactor client classes for safer type checking

@tswast Applied the suggestions.

Sorry for the force push, rebased just the relevant commits onto the latest main after #551 was merged.

plamut

comment created time in 4 days

push eventplamut/python-pubsub

Anthonios Partheniou

commit sha 17b5d143f2ffcab7e7efea8bb2018148c00ce1ee

chore: update .repo-metadata.json (#554)

view details

gcf-owl-bot[bot]

commit sha 1de3a528d2bc1bd9daa99e9d521847d57d2ed168

chore: use gapic-generator-python 0.58.4 (#555) * chore: use gapic-generator-python 0.58.4 fix: provide appropriate mock values for message body fields committer: dovs PiperOrigin-RevId: 419025932 Source-Link: https://github.com/googleapis/googleapis/commit/73da6697f598f1ba30618924936a59f8e457ec89 Source-Link: https://github.com/googleapis/googleapis-gen/commit/46df624a54b9ed47c1a7eefb7a49413cf7b82f98 Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiNDZkZjYyNGE1NGI5ZWQ0N2MxYTdlZWZiN2E0OTQxM2NmN2I4MmY5OCJ9 * 🦉 Updates from OwlBot See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>

view details

gcf-owl-bot[bot]

commit sha 03a63fd2b94cfc1a95d90d5179f5112fbced4b62

chore(samples): Add check for tests in directory (#557) Source-Link: https://github.com/googleapis/synthtool/commit/52aef91f8d25223d9dbdb4aebd94ba8eea2101f3 Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:36a95b8f494e4674dc9eee9af98961293b51b86b3649942aac800ae6c1f796d4 Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>

view details

Tianzi Cai

commit sha 1f1f7f33ad418b6605490ecde1b718d42cbae1ea

chore: update library type in .repo-metadata.json (#559)

view details

Tianzi Cai

commit sha 5067ad75f95e9bf310a55efe9030e8efd8c06a04

build: upgrade pip in samples build script (#561) * test: lite version update * try 1.2 * upgrade pip

view details

gcf-owl-bot[bot]

commit sha 273fa8cdc3838884df5ccbe67d320c02957f24b7

build: switch to release-please for tagging (#560) Source-Link: https://github.com/googleapis/synthtool/commit/f8077d237e0df2cb0066dfc6e09fc41e1c59646a Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:dfa9b663b32de8b5b327e32c1da665a80de48876558dd58091d8160c60ad7355 Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com> Co-authored-by: Anthonios Partheniou <partheniou@google.com>

view details

gcf-owl-bot[bot]

commit sha 37ce364f720000cd8ad29fb2b2a1340d71cdca55

chore(python): update release.sh to use keystore (#562) Source-Link: https://github.com/googleapis/synthtool/commit/69fda12e2994f0b595a397e8bb6e3e9f380524eb Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:ae600f36b6bc972b368367b6f83a1d91ec2c82a4a116b383d67d547c56fe6de3 Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com> Co-authored-by: Anthonios Partheniou <partheniou@google.com>

view details

Peter Lamut

commit sha a861848b146d2a24ae016b69e771f690e32ed919

process: add mypy session for samples (#551) * Add mypy session for samples to main noxfile * Make samples type hints good with mypy * Make samples type hints good with mypy (part 2) * Temporarily disable mypy_samples session * Move type-checked depepndency pins to noxfile The version pins that require recent enough depndencies for the sake of static type analysis should be placed in the corresponding nox session definition. They should not interfere with dependency versions used for testing the code at runtime, i.e. when running the tests. * Update google/cloud/pubsub_v1/publisher/futures.py Co-authored-by: Tim Swast <swast@google.com> Co-authored-by: Tianzi Cai <tianzi@google.com> Co-authored-by: Anthonios Partheniou <partheniou@google.com>

view details

Peter Lamut

commit sha 3c67b13e0853b9cde20183736d907311f0fdfedc

Directly subclass generated PublisherClient

view details

Peter Lamut

commit sha 1daefdebab2b749a2475974bc246142c187584b9

Directly subclass generated SubscriberClient

view details

Peter Lamut

commit sha 1eefc6dfaf0d029b02767825d76e339d56c673d8

Remove unneded GAPIC helper

view details

Peter Lamut

commit sha 5fe9c35787e15c05940e72ee4cde16963c18704f

Remove pytype There are no real advantages over mypy, but at the same time several downsides such as being slow, producing more false positives, etc.

view details

Peter Lamut

commit sha 048d109d61d4956fb71996dfe35dc89450dde014

Re-enable mypy_samples nox session

view details

Peter Lamut

commit sha ace0a039a183458ae935465af35ebd5d30a59ade

Convert a comment to docstring in publisher client

view details

Peter Lamut

commit sha 4457779c100ac4c836b152250851d6c0c7680b1b

Add api property back, but deprecated

view details

push time in 4 days

push eventplamut/python-pubsub

gcf-owl-bot[bot]

commit sha 1de3a528d2bc1bd9daa99e9d521847d57d2ed168

chore: use gapic-generator-python 0.58.4 (#555) * chore: use gapic-generator-python 0.58.4 fix: provide appropriate mock values for message body fields committer: dovs PiperOrigin-RevId: 419025932 Source-Link: https://github.com/googleapis/googleapis/commit/73da6697f598f1ba30618924936a59f8e457ec89 Source-Link: https://github.com/googleapis/googleapis-gen/commit/46df624a54b9ed47c1a7eefb7a49413cf7b82f98 Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiNDZkZjYyNGE1NGI5ZWQ0N2MxYTdlZWZiN2E0OTQxM2NmN2I4MmY5OCJ9 * 🦉 Updates from OwlBot See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>

view details

gcf-owl-bot[bot]

commit sha 03a63fd2b94cfc1a95d90d5179f5112fbced4b62

chore(samples): Add check for tests in directory (#557) Source-Link: https://github.com/googleapis/synthtool/commit/52aef91f8d25223d9dbdb4aebd94ba8eea2101f3 Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:36a95b8f494e4674dc9eee9af98961293b51b86b3649942aac800ae6c1f796d4 Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>

view details

Tianzi Cai

commit sha 1f1f7f33ad418b6605490ecde1b718d42cbae1ea

chore: update library type in .repo-metadata.json (#559)

view details

Tianzi Cai

commit sha 5067ad75f95e9bf310a55efe9030e8efd8c06a04

build: upgrade pip in samples build script (#561) * test: lite version update * try 1.2 * upgrade pip

view details

gcf-owl-bot[bot]

commit sha 273fa8cdc3838884df5ccbe67d320c02957f24b7

build: switch to release-please for tagging (#560) Source-Link: https://github.com/googleapis/synthtool/commit/f8077d237e0df2cb0066dfc6e09fc41e1c59646a Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:dfa9b663b32de8b5b327e32c1da665a80de48876558dd58091d8160c60ad7355 Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com> Co-authored-by: Anthonios Partheniou <partheniou@google.com>

view details

gcf-owl-bot[bot]

commit sha 37ce364f720000cd8ad29fb2b2a1340d71cdca55

chore(python): update release.sh to use keystore (#562) Source-Link: https://github.com/googleapis/synthtool/commit/69fda12e2994f0b595a397e8bb6e3e9f380524eb Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:ae600f36b6bc972b368367b6f83a1d91ec2c82a4a116b383d67d547c56fe6de3 Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com> Co-authored-by: Anthonios Partheniou <partheniou@google.com>

view details

Peter Lamut

commit sha a861848b146d2a24ae016b69e771f690e32ed919

process: add mypy session for samples (#551) * Add mypy session for samples to main noxfile * Make samples type hints good with mypy * Make samples type hints good with mypy (part 2) * Temporarily disable mypy_samples session * Move type-checked depepndency pins to noxfile The version pins that require recent enough depndencies for the sake of static type analysis should be placed in the corresponding nox session definition. They should not interfere with dependency versions used for testing the code at runtime, i.e. when running the tests. * Update google/cloud/pubsub_v1/publisher/futures.py Co-authored-by: Tim Swast <swast@google.com> Co-authored-by: Tianzi Cai <tianzi@google.com> Co-authored-by: Anthonios Partheniou <partheniou@google.com>

view details

push time in 4 days

delete branch plamut/python-pubsub

delete branch : iss-536-samples

delete time in 4 days

PullRequestReviewEvent

pull request commentgoogleapis/python-bigquery

test: make GC check resilient to broken weakrefs

@tswast I don't have write access anymore, please re-run the tests and merge the fix, thanks!

msuozzo

comment created time in 5 days

PullRequestReviewEvent

Pull request review commentgoogleapis/python-bigquery

test: make GC check resilient to broken weakrefs

 def test_does_not_keep_cursor_instances_alive(self):         # Connections should not hold strong references to the Cursor instances         # they created, unnecessarily keeping them alive.         gc.collect()-        cursors = [obj for obj in gc.get_objects() if isinstance(obj, Cursor)]-        self.assertEqual(len(cursors), 2)+        cursor_count = 0+        for obj in gc.get_objects():+          try:+            if isinstance(obj, Cursor):+              cursor_count += 1+          except ReferenceError:  # pragma: NO COVER+            pass+        self.assertEqual(cursor_count, 2)

The weakref obj is unwrapped when calling isinstance

You mean weakref.proxy? Because when initially locally testing with a dead weakref.weakref, it simply returned None and isinstance(wref, Foo) returned False (no ReferenceError). However, the error is indeed reproducible when using a dead weakref.proxy instead.

I suppose obj returned by gc.get_objects() is actually a proxy, which explains the behavior. Thanks!

msuozzo

comment created time in 5 days

PullRequestReviewEvent

Pull request review commentgoogleapis/python-pubsub

fix: refactor client classes for safer type checking

     "lint_setup_py",     "blacken",     "mypy",-    "pytype",+    # "mypy_samples",  # TODO: uncomment when the checks pass

Definitely, should have been enabled here already. Good catch.

plamut

comment created time in 9 days

more