profile
viewpoint
If you are wondering where the data of this site comes from, please visit https://api.github.com/users/jparise/events. GitMemory does not store any data, but only uses NGINX to cache data for a period of time. The idea behind GitMemory is simply to give users a better reading experience.
Jon Parise jparise Pinterest Oakland, CA https://www.indelible.org/ Scientia Potentia Est

jparise/chrome-utm-stripper 608

Browser extension that strips Google Analytics (UTM) parameters, and various other click tracking tokens, from URL query strings

jparise/AFHTTPClientLogger 40

Configurable HTTP request logger for AFNetworking

jparise/apprankings 36

App Store Rankings Scraper

jparise/dotfiles 6

My public dotfiles

jparise/django-ink 3

Ink is a blogging application for Django.

jparise/AFHTTPRequestOperationLogger 2

AFNetworking Extension for HTTP Request Logging

jparise/dammit-winamp 2

DAMMIT WinAMP plugin that Soco and I wrote at Computer Science House in the fall of 1998

jparise/ActionSheetPicker 1

Quickly reproduce the dropdown UIPickerView / ActionSheet functionality from Safari on iPhone/ iOS / CocoaTouch.

jparise/AFNetworking 1

A delightful iOS networking library with NSOperations and block-based callbacks

issue commentpinterest/pymemcache

setup of test_bench_get[pylibmc] fails on openSUSE Tumbleweed

When I disable pylibmc tests, I get

pymemcache/test/test_benchmark.py::test_bench_get[memcache] PASSED       [  0%]
pymemcache/test/test_benchmark.py::test_bench_get[pymemcache] ERROR      [  0%]

==================================== ERRORS ====================================
_________________ ERROR at setup of test_bench_get[pymemcache] _________________

request = <SubRequest 'client' for <Function test_bench_get[pymemcache]>>
host = 'localhost', port = 11211

    @pytest.fixture(params=[
        "pylibmc",
        "memcache",
        "pymemcache",
    ])
    def client(request, host, port):
        if request.param == "pylibmc":
            if not HAS_PYLIBMC:
                pytest.skip("requires pylibmc")
            client = pylibmc.Client(['{0}:{1}'.format(host, port)])
            client.behaviors = {"tcp_nodelay": True}
    
        elif request.param == "memcache":
            if not HAS_MEMCACHE:
                pytest.skip("requires python-memcached")
            client = memcache.Client(['{0}:{1}'.format(host, port)])
    
        elif request.param == "pymemcache":
            if not HAS_PYMEMCACHE:
                pytest.skip("requires pymemcache")
            client = pymemcache.client.Client((host, port))
    
        else:
            pytest.skip("unknown library {0}".format(request.param))
    
>       client.flush_all()

pymemcache/test/test_benchmark.py:64: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
pymemcache/client/base.py:803: in flush_all
    results = self._misc_cmd([cmd], b'flush_all', noreply)
pymemcache/client/base.py:1002: in _misc_cmd
    self._connect()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <pymemcache.client.base.Client object at 0x7f1323681430>

    def _connect(self):
        self.close()
    
        s = self.socket_module
    
        if not isinstance(self.server, tuple):
            sockaddr = self.server
            sock = s.socket(s.AF_UNIX, s.SOCK_STREAM)
    
        else:
            sock = None
            error = None
            host, port = self.server
            info = s.getaddrinfo(host, port, s.AF_UNSPEC, s.SOCK_STREAM,
                                 s.IPPROTO_TCP)
            for family, socktype, proto, _, sockaddr in info:
                try:
                    sock = s.socket(family, socktype, proto)
                    if self.no_delay:
                        sock.setsockopt(s.IPPROTO_TCP, s.TCP_NODELAY, 1)
                    if self.tls_context:
                        context = self.tls_context
                        sock = context.wrap_socket(sock, server_hostname=host)
                except Exception as e:
                    error = e
                    if sock is not None:
                        sock.close()
                        sock = None
                else:
                    break
    
            if error is not None:
                raise error
    
        try:
            sock.settimeout(self.connect_timeout)
>           sock.connect(sockaddr)
E           ConnectionRefusedError: [Errno 111] Connection refused
jayvdb

comment created time in 20 minutes

issue openedpinterest/pymemcache

setup of test_bench_get[pylibmc] fails on openSUSE Tumbleweed

openSUSE Tumbleweed currently has the latest version of pylibmc, v1.6.1, and I am seeing the following when running the bench tests.

__________________ ERROR at setup of test_bench_get[pylibmc] ___________________

request = <SubRequest 'client' for <Function test_bench_get[pylibmc]>>
host = 'localhost', port = 11211

    @pytest.fixture(params=[
        "pylibmc",
        "memcache",
        "pymemcache",
    ])
    def client(request, host, port):
        if request.param == "pylibmc":
            if not HAS_PYLIBMC:
                pytest.skip("requires pylibmc")
            client = pylibmc.Client(['{0}:{1}'.format(host, port)])
            client.behaviors = {"tcp_nodelay": True}
    
        elif request.param == "memcache":
            if not HAS_MEMCACHE:
                pytest.skip("requires python-memcached")
            client = memcache.Client(['{0}:{1}'.format(host, port)])
    
        elif request.param == "pymemcache":
            if not HAS_PYMEMCACHE:
                pytest.skip("requires pymemcache")
            client = pymemcache.client.Client((host, port))
    
        else:
            pytest.skip("unknown library {0}".format(request.param))
    
>       client.flush_all()
E       pylibmc.SomeErrors: error 19 from flush_all: (0x5588f11ed030) CONNECTION FAILURE(Connection refused),  host: localhost:11211 -> libmemcached/connect.cc:156

created time in 24 minutes

PR opened pinterest/pymemcache

Client API for server shutdown protocol command

This change proposes the addition of a client API wrapper for the shutdown text protocol command.

Minimal example usage:

from pymemcache.client.base import Client


c = Client('localhost:11211')
c.shutdown(False)

When shutdown is not enabled on the server:

<28 new auto-negotiating client connection
28: Client using the ascii protocol
<28 shutdown
>28 ERROR: shutdown not enabled
<28 connection closed.
Traceback (most recent call last):
  File "driver.py", line 5, in <module>
    c.shutdown(False)
  File "/home/kiwi/sync/code/external/pymemcache/pymemcache/client/base.py", line 845, in shutdown
    self._misc_cmd([cmd], b'shutdown', False)
  File "/home/kiwi/sync/code/external/pymemcache/pymemcache/client/base.py", line 1045, in _misc_cmd
    self._raise_errors(line, cmd_name)
  File "/home/kiwi/sync/code/external/pymemcache/pymemcache/client/base.py", line 853, in _raise_errors
    raise MemcacheUnknownCommandError(name)
pymemcache.exceptions.MemcacheUnknownCommandError: b'shutdown'

When --enable-shutdown is specified at server runtime:

<28 new auto-negotiating client connection
28: Client using the ascii protocol
<28 shutdown
Signal handled: Interrupt.
<28 connection closed.
stopped assoc
asking workers to stop
asking background threads to stop
stopped lru crawler
stopped maintainer
stopped slab mover
stopped logger thread
stopped idle timeout thread
closing connections
<26 connection closed.
<27 connection closed.
reaping worker threads
all background threads stopped
+49 -0

0 comment

2 changed files

pr created time in 4 hours

pull request commentmicrosoft/thrifty

Add http transport

Codecov Report

Merging #449 (af78d7d) into master (e218285) will increase coverage by 0.00%. The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff            @@
##             master     #449   +/-   ##
=========================================
  Coverage     59.52%   59.53%           
  Complexity      796      796           
=========================================
  Files            60       60           
  Lines          5534     5535    +1     
  Branches        876      876           
=========================================
+ Hits           3294     3295    +1     
  Misses         1996     1996           
  Partials        244      244           
Impacted Files Coverage Δ
.../com/microsoft/thrifty/kgen/KotlinCodeGenerator.kt 82.35% <0.00%> (+0.01%) :arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update e218285...af78d7d. Read the comment docs.

luqasn

comment created time in 7 hours

PR opened microsoft/thrifty

Add http transport
+178 -0

0 comment

2 changed files

pr created time in 7 hours

pull request commentpinterest/pymemcache

[WIP] Adding retry management

Fresh paint! Does not yet work, please don't spend time on it for now.

4383

comment created time in 11 hours

pull request commentpinterest/pymemcache

Implement RetryingClient

It's not a problem for me if you decide to continue with this version (that of @martinnj). Maybe we could converge our implementations to design and co-author something, it's up to you to decide. I propose you dispose.

Wild timing, I'll just wait with spending more time until a collaborator decides either way. :)

martinnj

comment created time in 12 hours

Pull request review commentpinterest/pymemcache

[WIP] Adding retry management

+import collections+import functools+import inspect+import socket+import time+import logging++import pymemcache.exceptions as pymemexception++logger = logging.getLogger(__name__)++def retry(func):+    @functools.wraps(func)+    def func_call(*args, **kwargs):+        ntries=15+        while ntries:+            try:+                return func(*args, **kwargs)+            except pymemexception.MemcacheUnexpectedCloseError as e:

Same thing here, I think that we need to provide strategies. Either we allow handle all the kinds of exceptions, or only the pymemcache exceptions, or again only one specific type of exception passed by user.

For now it's just a dumb PoC for my own usage.

4383

comment created time in 14 hours

Pull request review commentpinterest/pymemcache

[WIP] Adding retry management

+import collections+import functools+import inspect+import socket+import time+import logging++import pymemcache.exceptions as pymemexception++logger = logging.getLogger(__name__)++def retry(func):+    @functools.wraps(func)+    def func_call(*args, **kwargs):+        ntries=15

Still WIP so hard coded, but the goal is to provide some configurable options (backoff, number of retry, duration, etc...)

4383

comment created time in 14 hours

Pull request review commentpinterest/pymemcache

[WIP] Adding retry management

+import collections+import functools+import inspect+import socket+import time+import logging++import pymemcache.exceptions as pymemexception++logger = logging.getLogger(__name__)++def retry(func):+    @functools.wraps(func)+    def func_call(*args, **kwargs):+        ntries=15

We need more robust strategy, like a backoff strategy with a range of possible value to avoid to flood the server and manage system workload more finely. If many clients try to reconnect at the same time the backoff strategy will help to spread the load.

4383

comment created time in 14 hours

pull request commentpinterest/pymemcache

Implement RetryingClient

Also notice that Wednesday I also started to implement similar features, however, I didn't yet proposed them before today through a pull request. #326

By introspection the passed clients, I dynamically decorate their methods to make them "retryable".

It's not a problem for me if you decide to continue with this version (that of @martinnj). Maybe we could converge our implementations to design and co-author something, it's up to you to decide. I propose you dispose.

martinnj

comment created time in 14 hours

pull request commentpinterest/pymemcache

Implement RetryingClient

There are a couple existing "retry" libraries on PyPI, and it is probably worth either implementing their full set of functionality, or just using them directly. Here are the two I found:

https://pypi.org/project/retry/ https://pypi.org/project/retrying/

Tenacity is an interesting implementation and already well used on openstack.

However, I agree with @martinnj , if possible, it would be more interesting to keep pymemcache as a pure python library (I voluntarly ignore six). Maybe by only providing specific retry strategies without tons of configurations. Less is more.

martinnj

comment created time in 14 hours

pull request commentpinterest/pymemcache

[WIP] Adding retry management

Related to #307

4383

comment created time in 14 hours

PR opened pinterest/pymemcache

[WIP] Adding retry management

Still work in progress.

The goal is to decorate all passed client dynamically to avoid technical debt and API changes.

This module will only decorate passed client to manage uniformized and standardized retries. It will be possible to configure the retry strategies and mechanismes.

Here is just a PoC for now.

+49 -0

0 comment

2 changed files

pr created time in 14 hours

release Mantle/Mantle

2.2.0

released time in 15 hours

pull request commentmicrosoft/thrifty

Add Server support

Server generation now always generates its own service interface in a server namespace that is always coroutine-based. First, I tried for allowing both "plain" service handlers and coroutine ones, but that was a pain in the back because I had to duplicate a lot of code because suspend functions have different signatures. So in the end I opted to just always generate coroutine based servers, because it allows for more freedom for the user, e.g. to use async constructs, either magic from Arrow or whatever. If you choose to not use any coroutine specific stuff in your handler, that should be totally fine and not impact performance much, all you have is an extraneous parameter on our methods for the coroutine continuation (or maybe the compiler even optimizes that away if you never suspend).

luqasn

comment created time in 16 hours

Pull request review commentmicrosoft/thrifty

Add Server support

 class KotlinCodeGenerator(             }         } +        if (generateServer) {+            check(coroutineServiceClients) { "Server feature currently requires coroutineServiceClients to be enabled" }

This should be fixed now.

luqasn

comment created time in 16 hours

release usablica/intro.js

v4.1.0

released time in 16 hours

release backstage/backstage

release-2021-06-18

released time in 17 hours

PR opened pinterest/arcanist-linters

Reviewers
Support autofix with eslint, and remove fix flag

When there are autofixable lint errors detected by eslint, the order of operations is:

  1. arc diff
  2. if there are uncommited changes, amend or commit
  3. arc lint, which autofixes and returns successful
  4. autofixed changes are not committed nor staged
  5. diff is pushed to phabricator without changes

This means that arc diff fails to properly apply lint changes before creating a diff.

See messages for a file provided by eslint@6.8.0

        "messages": [
            {
                "ruleId": "prettier/prettier",
                "severity": 2,
                "message": "Replace `(flow.component·&&·flow.component.archived)` with `flow.component·&&·flow.component.archived`",
                "line": 61,
                "column": 10,
                "nodeType": null,
                "messageId": "replace",
                "endLine": 61,
                "endColumn": 53,
                "fix": {
                    "range": [
                        1462,
                        1505
                    ],
                    "text": "flow.component && flow.component.archived"
                }
            },
            {
                "ruleId": "prettier/prettier",
                "severity": 2,
                "message": "Insert `(⏎··········`",
                "line": 65,
                "column": 13,
                "nodeType": null,
                "messageId": "insert",
                "endLine": 65,
                "endColumn": 13,
                "fix": {
                    "range": [
                        1675,
                        1675
                    ],
                    "text": "(\n          "
                }
            },
            {
                "ruleId": "prettier/prettier",
                "severity": 2,
                "message": "Replace `⏎⏎⏎········` with `········)`",
                "line": 66,
                "column": 1,
                "nodeType": null,
                "messageId": "replace",
                "endLine": 69,
                "endColumn": 9,
                "fix": {
                    "range": [
                        1685,
                        1696
                    ],
                    "text": "        )"
                }
            }
        ],

Summary of changes:

  1. Parse the fix fields in each message, and build ArcanistLintMessage with originalText and replacementText populated, along with a severity of ArcanistLintSeverity::SEVERITY_AUTOFIX. This means that the user will receive a prompt asking if they'd like to apply the fixes.
  2. Remove the eslint.fix flag because it likely causes more confusion that it helps.
+47 -11

0 comment

1 changed file

pr created time in a day

issue openedpinterest/pymemcache

Three unit tests failing on isolated VM

When running the tests in an isolated VM for openSUSE rpm builds, disconnected from the internet, I am encountering the following three test failures. The openSUSE package is currently at v3.3.0

======================================================================
ERROR: test_socket_close (pymemcache.test.test_client.TestClientSocketConnect)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/abuild/rpmbuild/BUILD/pymemcache-3.4.4/pymemcache/test/test_client.py", line 1160, in test_socket_close
    client._connect()
  File "/home/abuild/rpmbuild/BUILD/pymemcache-3.4.4/pymemcache/client/base.py", line 314, in _connect
    s.IPPROTO_TCP)
  File "/usr/lib64/python3.6/socket.py", line 745, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution

======================================================================
ERROR: test_socket_close_exception (pymemcache.test.test_client.TestClientSocketConnect)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/abuild/rpmbuild/BUILD/pymemcache-3.4.4/pymemcache/test/test_client.py", line 1171, in test_socket_close_exception
    client._connect()
  File "/home/abuild/rpmbuild/BUILD/pymemcache-3.4.4/pymemcache/client/base.py", line 314, in _connect
    s.IPPROTO_TCP)
  File "/usr/lib64/python3.6/socket.py", line 745, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution

======================================================================
FAIL: test_socket_connect_closes_on_failure (pymemcache.test.test_client.TestClientSocketConnect)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/abuild/rpmbuild/BUILD/pymemcache-3.4.4/pymemcache/test/test_client.py", line 1152, in test_socket_connect_closes_on_failure
    assert len(socket_module.sockets) == 1
AssertionError

I am also experiencing errors with the gevent and pylibmc unit tests, and the integration and benchmark tests, but I am still investigating those.

created time in a day

Pull request review commentpinterest/pymemcache

Implement RetryingClient

+""" Module containing the RetryingClient wrapper class. """++from collections import Iterable+from time import sleep+++def _ensure_tuple_argument(argument_name, argument):+    """+    Helper function to ensure the given arguments are tuples of Exceptions (or subclasses),+    or can at least be converted to such.++    Args:+      argument_name: str, name of the argument we're checking, only used for raising meaningfull+        exceptions.+      argument: any, the argument itself.++    Returns:+      tuple[Exception]: A tuple with the elements from the argument if they are valid.++    Exceptions:+      ValueError: If the argument was not None, tuple or Iterable.+      ValueError: If any of the elements of the argument is not a subclass of Exception.+    """++    argument_tuple = None++    # Ensure the argument is a tuple, or raise an error if we can't convert it.+    if argument is None:+        return tuple()+    elif isinstance(argument, tuple):+        argument_tuple = argument+    elif isinstance(argument, Iterable) and not isinstance(argument, (str, bytes, bytearray)):+        argument_tuple = tuple(argument)+    else:+        raise ValueError("%s was not a tuple or Iterable." % argument_name)++    # Check that all the elements are actually inherited from Exception. (Catchable)+    if not all([issubclass(arg, Exception) for arg in argument_tuple]):+        raise ValueError("%s contained elements other than Exceptions." % argument_name)++class RetryingClient(object):+    """+    Client that allows retrying calls for the other clients.+    """++    def __init__(+        self,+        client,+        attempts=2,+        retry_delay=0,+        retry_for=None,+        do_not_retry_for=None+    ):+        self.client = client+        self.attempts = attempts+        self.retry_delay = retry_delay+        self.retry_for = _ensure_tuple_argument("retry_for", retry_for)+        self.do_not_retry_for = _ensure_tuple_argument("do_not_retry_for", do_not_retry_for)+++    # This is the real magic soup of the class, we catch anything that isn't strictly defined for+    # ourselves and pass it on to whatever client we've been given.+    def __getattr__(self, name, *args, **kwargs):+        # For no retry:+        # return self.client.__getattribute__(name, *args, **kwargs)+        for attempt in range(self.attempts):+            try:+                return self.client.__getattribute__(name, *args, **kwargs)+            except Exception as exc:

Gotcha!

martinnj

comment created time in a day

pull request commentpinterest/pymemcache

Implement RetryingClient

There are a couple existing "retry" libraries on PyPI, and it is probably worth either implementing their full set of functionality, or just using them directly. Here are the two I found:

https://pypi.org/project/retry/ https://pypi.org/project/retrying/

Was there any specific features from those libraries you where wanting? Currently the timeout, retry count and exception based actions is implemented.

I didn't look for existing libraries because adding dependencies for something like this would be a shame in my eyes.

martinnj

comment created time in a day

Pull request review commentpinterest/pymemcache

Implement RetryingClient

+""" Module containing the RetryingClient wrapper class. """++from collections import Iterable+from time import sleep+++def _ensure_tuple_argument(argument_name, argument):+    """+    Helper function to ensure the given arguments are tuples of Exceptions (or subclasses),+    or can at least be converted to such.++    Args:+      argument_name: str, name of the argument we're checking, only used for raising meaningfull+        exceptions.+      argument: any, the argument itself.++    Returns:+      tuple[Exception]: A tuple with the elements from the argument if they are valid.++    Exceptions:+      ValueError: If the argument was not None, tuple or Iterable.+      ValueError: If any of the elements of the argument is not a subclass of Exception.+    """++    argument_tuple = None++    # Ensure the argument is a tuple, or raise an error if we can't convert it.+    if argument is None:+        return tuple()+    elif isinstance(argument, tuple):+        argument_tuple = argument+    elif isinstance(argument, Iterable) and not isinstance(argument, (str, bytes, bytearray)):+        argument_tuple = tuple(argument)+    else:+        raise ValueError("%s was not a tuple or Iterable." % argument_name)++    # Check that all the elements are actually inherited from Exception. (Catchable)+    if not all([issubclass(arg, Exception) for arg in argument_tuple]):+        raise ValueError("%s contained elements other than Exceptions." % argument_name)++class RetryingClient(object):+    """+    Client that allows retrying calls for the other clients.+    """++    def __init__(+        self,+        client,+        attempts=2,+        retry_delay=0,+        retry_for=None,+        do_not_retry_for=None+    ):+        self.client = client+        self.attempts = attempts+        self.retry_delay = retry_delay+        self.retry_for = _ensure_tuple_argument("retry_for", retry_for)+        self.do_not_retry_for = _ensure_tuple_argument("do_not_retry_for", do_not_retry_for)+++    # This is the real magic soup of the class, we catch anything that isn't strictly defined for+    # ourselves and pass it on to whatever client we've been given.+    def __getattr__(self, name, *args, **kwargs):+        # For no retry:+        # return self.client.__getattribute__(name, *args, **kwargs)+        for attempt in range(self.attempts):+            try:+                return self.client.__getattribute__(name, *args, **kwargs)+            except Exception as exc:

Of course, a user of the library should be able to retry on their own AttributeErrors! So we need logic that ignores AttributeError thrown by our code only.

martinnj

comment created time in a day

Pull request review commentpinterest/pymemcache

Implement RetryingClient

+""" Module containing the RetryingClient wrapper class. """++from collections import Iterable+from time import sleep+++def _ensure_tuple_argument(argument_name, argument):+    """+    Helper function to ensure the given arguments are tuples of Exceptions (or subclasses),+    or can at least be converted to such.++    Args:+      argument_name: str, name of the argument we're checking, only used for raising meaningfull+        exceptions.+      argument: any, the argument itself.++    Returns:+      tuple[Exception]: A tuple with the elements from the argument if they are valid.++    Exceptions:+      ValueError: If the argument was not None, tuple or Iterable.+      ValueError: If any of the elements of the argument is not a subclass of Exception.+    """++    argument_tuple = None++    # Ensure the argument is a tuple, or raise an error if we can't convert it.+    if argument is None:+        return tuple()+    elif isinstance(argument, tuple):+        argument_tuple = argument+    elif isinstance(argument, Iterable) and not isinstance(argument, (str, bytes, bytearray)):+        argument_tuple = tuple(argument)+    else:+        raise ValueError("%s was not a tuple or Iterable." % argument_name)++    # Check that all the elements are actually inherited from Exception. (Catchable)+    if not all([issubclass(arg, Exception) for arg in argument_tuple]):+        raise ValueError("%s contained elements other than Exceptions." % argument_name)++class RetryingClient(object):+    """+    Client that allows retrying calls for the other clients.+    """++    def __init__(+        self,+        client,+        attempts=2,+        retry_delay=0,+        retry_for=None,+        do_not_retry_for=None+    ):+        self.client = client+        self.attempts = attempts+        self.retry_delay = retry_delay+        self.retry_for = _ensure_tuple_argument("retry_for", retry_for)+        self.do_not_retry_for = _ensure_tuple_argument("do_not_retry_for", do_not_retry_for)+++    # This is the real magic soup of the class, we catch anything that isn't strictly defined for+    # ourselves and pass it on to whatever client we've been given.+    def __getattr__(self, name, *args, **kwargs):+        # For no retry:+        # return self.client.__getattribute__(name, *args, **kwargs)+        for attempt in range(self.attempts):+            try:+                return self.client.__getattribute__(name, *args, **kwargs)+            except Exception as exc:

We need to make sure to never retry on an AttributeError.

martinnj

comment created time in a day

Pull request review commentpinterest/pymemcache

Implement RetryingClient

+""" Module containing the RetryingClient wrapper class. """++from collections import Iterable+from time import sleep+++def _ensure_tuple_argument(argument_name, argument):+    """+    Helper function to ensure the given arguments are tuples of Exceptions (or subclasses),+    or can at least be converted to such.++    Args:+      argument_name: str, name of the argument we're checking, only used for raising meaningfull+        exceptions.+      argument: any, the argument itself.++    Returns:+      tuple[Exception]: A tuple with the elements from the argument if they are valid.++    Exceptions:+      ValueError: If the argument was not None, tuple or Iterable.+      ValueError: If any of the elements of the argument is not a subclass of Exception.+    """++    argument_tuple = None++    # Ensure the argument is a tuple, or raise an error if we can't convert it.+    if argument is None:+        return tuple()+    elif isinstance(argument, tuple):+        argument_tuple = argument+    elif isinstance(argument, Iterable) and not isinstance(argument, (str, bytes, bytearray)):+        argument_tuple = tuple(argument)+    else:+        raise ValueError("%s was not a tuple or Iterable." % argument_name)++    # Check that all the elements are actually inherited from Exception. (Catchable)+    if not all([issubclass(arg, Exception) for arg in argument_tuple]):+        raise ValueError("%s contained elements other than Exceptions." % argument_name)++class RetryingClient(object):+    """+    Client that allows retrying calls for the other clients.+    """++    def __init__(+        self,+        client,+        attempts=2,+        retry_delay=0,+        retry_for=None,+        do_not_retry_for=None+    ):+        self.client = client+        self.attempts = attempts+        self.retry_delay = retry_delay+        self.retry_for = _ensure_tuple_argument("retry_for", retry_for)+        self.do_not_retry_for = _ensure_tuple_argument("do_not_retry_for", do_not_retry_for)+++    # This is the real magic soup of the class, we catch anything that isn't strictly defined for+    # ourselves and pass it on to whatever client we've been given.+    def __getattr__(self, name, *args, **kwargs):+        # For no retry:+        # return self.client.__getattribute__(name, *args, **kwargs)+        for attempt in range(self.attempts):+            try:+                return self.client.__getattribute__(name, *args, **kwargs)+            except Exception as exc:++                if attempt == self.attempts:

Nice catch. Thanks. 👍

martinnj

comment created time in a day

Pull request review commentpinterest/pymemcache

Implement RetryingClient

+""" Module containing the RetryingClient wrapper class. """++from collections import Iterable+from time import sleep+++def _ensure_tuple_argument(argument_name, argument):+    """+    Helper function to ensure the given arguments are tuples of Exceptions (or subclasses),+    or can at least be converted to such.++    Args:+      argument_name: str, name of the argument we're checking, only used for raising meaningfull+        exceptions.+      argument: any, the argument itself.++    Returns:+      tuple[Exception]: A tuple with the elements from the argument if they are valid.++    Exceptions:+      ValueError: If the argument was not None, tuple or Iterable.+      ValueError: If any of the elements of the argument is not a subclass of Exception.+    """++    argument_tuple = None++    # Ensure the argument is a tuple, or raise an error if we can't convert it.+    if argument is None:+        return tuple()+    elif isinstance(argument, tuple):+        argument_tuple = argument+    elif isinstance(argument, Iterable) and not isinstance(argument, (str, bytes, bytearray)):+        argument_tuple = tuple(argument)+    else:+        raise ValueError("%s was not a tuple or Iterable." % argument_name)++    # Check that all the elements are actually inherited from Exception. (Catchable)+    if not all([issubclass(arg, Exception) for arg in argument_tuple]):+        raise ValueError("%s contained elements other than Exceptions." % argument_name)++class RetryingClient(object):+    """+    Client that allows retrying calls for the other clients.+    """++    def __init__(+        self,+        client,+        attempts=2,+        retry_delay=0,+        retry_for=None,+        do_not_retry_for=None+    ):+        self.client = client+        self.attempts = attempts+        self.retry_delay = retry_delay+        self.retry_for = _ensure_tuple_argument("retry_for", retry_for)+        self.do_not_retry_for = _ensure_tuple_argument("do_not_retry_for", do_not_retry_for)+++    # This is the real magic soup of the class, we catch anything that isn't strictly defined for+    # ourselves and pass it on to whatever client we've been given.+    def __getattr__(self, name, *args, **kwargs):+        # For no retry:+        # return self.client.__getattribute__(name, *args, **kwargs)+        for attempt in range(self.attempts):+            try:+                return self.client.__getattribute__(name, *args, **kwargs)+            except Exception as exc:++                if attempt == self.attempts:+                    # If we're at the limit, just raise the exception to caller.+                    # Exception type doesn't matter.+                    raise exc+                elif isinstance(exc, self.retry_for) or not isinstance(exc, self.do_not_retry_for):

This one will be clarified with documentation. Case 4 is a little dicey yes. My intuition was that if you haven't asked specified any go/no-go exceptions, revert to simpler behavior: Everything is just getting retried.

martinnj

comment created time in a day

Pull request review commentpinterest/pymemcache

Implement RetryingClient

+""" Module containing the RetryingClient wrapper class. """++from collections import Iterable+from time import sleep+++def _ensure_tuple_argument(argument_name, argument):+    """+    Helper function to ensure the given arguments are tuples of Exceptions (or subclasses),+    or can at least be converted to such.++    Args:+      argument_name: str, name of the argument we're checking, only used for raising meaningfull+        exceptions.+      argument: any, the argument itself.++    Returns:+      tuple[Exception]: A tuple with the elements from the argument if they are valid.++    Exceptions:+      ValueError: If the argument was not None, tuple or Iterable.+      ValueError: If any of the elements of the argument is not a subclass of Exception.+    """++    argument_tuple = None++    # Ensure the argument is a tuple, or raise an error if we can't convert it.+    if argument is None:+        return tuple()+    elif isinstance(argument, tuple):+        argument_tuple = argument+    elif isinstance(argument, Iterable) and not isinstance(argument, (str, bytes, bytearray)):+        argument_tuple = tuple(argument)+    else:+        raise ValueError("%s was not a tuple or Iterable." % argument_name)++    # Check that all the elements are actually inherited from Exception. (Catchable)+    if not all([issubclass(arg, Exception) for arg in argument_tuple]):+        raise ValueError("%s contained elements other than Exceptions." % argument_name)++class RetryingClient(object):+    """+    Client that allows retrying calls for the other clients.+    """++    def __init__(+        self,+        client,+        attempts=2,+        retry_delay=0,+        retry_for=None,+        do_not_retry_for=None+    ):+        self.client = client+        self.attempts = attempts+        self.retry_delay = retry_delay+        self.retry_for = _ensure_tuple_argument("retry_for", retry_for)+        self.do_not_retry_for = _ensure_tuple_argument("do_not_retry_for", do_not_retry_for)+++    # This is the real magic soup of the class, we catch anything that isn't strictly defined for+    # ourselves and pass it on to whatever client we've been given.+    def __getattr__(self, name, *args, **kwargs):+        # For no retry:+        # return self.client.__getattribute__(name, *args, **kwargs)+        for attempt in range(self.attempts):+            try:+                return self.client.__getattribute__(name, *args, **kwargs)+            except Exception as exc:

Just to be sure, you mean catch AttributeError? Propegating it seems like the right call since you've essentially tried to call a specific method from the underlying client, that method doesn't exist. Which would also give the AttributeError if called directly.

Or did I misunderstand the comment? :)

martinnj

comment created time in a day

Pull request review commentpinterest/pymemcache

Implement RetryingClient

+""" Module containing the RetryingClient wrapper class. """++from collections import Iterable+from time import sleep+++def _ensure_tuple_argument(argument_name, argument):+    """+    Helper function to ensure the given arguments are tuples of Exceptions (or subclasses),+    or can at least be converted to such.++    Args:+      argument_name: str, name of the argument we're checking, only used for raising meaningfull+        exceptions.+      argument: any, the argument itself.++    Returns:+      tuple[Exception]: A tuple with the elements from the argument if they are valid.++    Exceptions:+      ValueError: If the argument was not None, tuple or Iterable.+      ValueError: If any of the elements of the argument is not a subclass of Exception.+    """++    argument_tuple = None++    # Ensure the argument is a tuple, or raise an error if we can't convert it.+    if argument is None:+        return tuple()+    elif isinstance(argument, tuple):+        argument_tuple = argument+    elif isinstance(argument, Iterable) and not isinstance(argument, (str, bytes, bytearray)):+        argument_tuple = tuple(argument)+    else:+        raise ValueError("%s was not a tuple or Iterable." % argument_name)++    # Check that all the elements are actually inherited from Exception. (Catchable)+    if not all([issubclass(arg, Exception) for arg in argument_tuple]):+        raise ValueError("%s contained elements other than Exceptions." % argument_name)++class RetryingClient(object):+    """+    Client that allows retrying calls for the other clients.+    """++    def __init__(+        self,+        client,+        attempts=2,+        retry_delay=0,+        retry_for=None,+        do_not_retry_for=None+    ):+        self.client = client+        self.attempts = attempts+        self.retry_delay = retry_delay+        self.retry_for = _ensure_tuple_argument("retry_for", retry_for)+        self.do_not_retry_for = _ensure_tuple_argument("do_not_retry_for", do_not_retry_for)+++    # This is the real magic soup of the class, we catch anything that isn't strictly defined for+    # ourselves and pass it on to whatever client we've been given.+    def __getattr__(self, name, *args, **kwargs):+        # For no retry:+        # return self.client.__getattribute__(name, *args, **kwargs)

I would just leave this comment out, since it doesn't add anything to the understanding.

martinnj

comment created time in a day