profile
viewpoint
Sergei Lebedev superbobry Software Engineer at @deepmind London, UK

pytries/marisa-trie 725

Static memory-efficient Trie-like structures for Python (2.x and 3.x) based on marisa-trie C++ library.

pytries/datrie 419

Fast, efficiently stored Trie for Python. Uses libdatrie.

slacy/minimongo 312

A lightweight, Pythonic, Object Oriented Interface to MongoDB.

deepmind/tree 41

tree is a library for working with nested data structures

selectel/pat 23

Pat, the only SMTP postman!

superbobry/color-theme-desert 12

A port of VIM's desert color theme to Emacs.

demiurg906/pyte_gui 2

gui for Pyte project

superbobry/cuisine 2

Chef-like functionality for Fabric

lambda-llama/icfpc2016 1

λ-llama code for ICFP contest 2016

issue commentsphinx-doc/sphinx

Option for not unfolding aliases

This is doable with zero alterations to the Sphinx codebase. All you need is the annotations future-import (introduced in PEP 563 and available in Python >=3.7) and a monkey-patched typing.get_type_hints:

import typing
typing.get_type_hints = lambda obj, *unused: obj

Sphinx uses typing.get_type_hints to resolve string annotations, so the future-import alone is not sufficient. Making this behavior configurable will allow to avoid monkey-patching.

rhkubiak

comment created time in a day

Pull request review commenttensorflow/community

RFC: TensorFlow Canonical Type System

+# TensorFlow Canonical Type System++| Status        | Proposed       |+:-------------- |:---------------------------------------------------- |+| **RFC #**     | [208](https://github.com/tensorflow/community/pull/208)+| **Author(s)** | Dan Moldovan (mdan@google.com) |+| **Sponsor**   | Gaurav Jain (gjn@google.com)                 |+| **Updated**   | 2020-02-19                                           |++## Objective++This RFC proposes a new TensorFlow module and namespace (`tf.types`) dedicated to storing implementation-free type definitions, similar to C++ header files. This module has no other dependencies inside TensorFlow, so any other internal module can depend on it to ensure interoperability without the risk of creating circular dependencies. These definitions can also be used by external users, for example in pytype annotations.+The RFC focuses on the Python API, however the design should be reviewed with cross-language consistency in mind.+## Motivation++**Interoperability and composability**. A set of standard types that formalize an interface and decouples it from implementation ensures composability between components, especially when multiple implementations are involved.++**Supports the [acyclic dependencies principle](https://en.wikipedia.org/wiki/Acyclic_dependencies_principle)**. In many instances, circular dependencies are caused between low-level complex components that need to compose (e.g. autograph needs to recognize datasets, and datasets need to use autograph). Interface extraction is a common pattern for breaking such cycles.++**Supports pytype**. A set of static types that is consistent under Python’s `isinstance`/`issubclass` is required to support [PEP-484 type annotations](https://www.python.org/dev/peps/pep-0484/) in TensorFlow. This module can serve as the basis for that.++**Helps formalize requirements for new APIs**. Having a formal, implementation-independent definition for things such as tensors, variables, iterables, iterators makes it easy to document and test compatibility between APIs.++## User Benefit++Application developers may use these canonical definitions for pytype annotations.++Library developers can more easily define their API interfaces by referring to this namespace.++Developers of modules internal to TensorFlow can use this module to avoid creating circular dependencies.++## Design Proposal++### The `tf.types` Namespace / Module+All the declarations exposed under the `tf.types` namespace reside in the `python/types/*.py` module. These are [abstract base classes](https://docs.python.org/3.7/library/abc.html) with a bare minimum of method definitions and minimal or no implementation, which serve to formalize and document the contract of common types such as `Tensor`, `Variable`, etc.++These definitions may be used as PEP 484 type hints, although in some cases they may be type- or shape- erased (for example, `tf.types.Tensor` may not necessarily be parametrized by `dtype` or `shape`). Note however that designs which parametrize on shape do exist, see for instance [tensorflow#31579](https://github.com/tensorflow/tensorflow/issues/31579).++The type definitions are consistent with `isinstance` and `issubclass`. For example, `isinstance(tf.Tensor, tf.types.Tensor) == True`.++### General Principles+This module should not contain any implementation code. An advantage of that is that users exploring the implementation of specific types will not need to inspect this module. However, users who do not wish to inspect the code may visit the documentation of these generic types to better understand specifically what are the concrete subclasses of this type expected to do.++The `tf.types` module may depend on external packages (such as `numpy`) _strictly for the purpose of defining type annotations and documentation_. No dependencies to other TensorFlow interfaces are allowed. Any dependencies on external packages which themselves depend on TensorFlow are expressly forbidden.++Changes definitions inside `tf.types` must be approved by TensorFlow leads, and typically should be accompanied by an RFC.++All type declarations are compatible with [pytype](https://github.com/google/pytype).++It is recommended that internal and external type annotations, `isinstance` and `issubclass` checks use these types, eventually deprecating helpers like `tf.is_tensor`. However, concrete types continue to exist - for example, variables are instances of `tf.Variable`, which is now a subclass of `tf.types.Variable`.++Class type definitions define a minimum of abstract methods and properties which are required for pytype compatibility.++### Support for `tf.function`'s `input_signature`+The type system listed here can be expanded to allow input signatures using type annotations, see for instance [this thread](https://github.com/tensorflow/tensorflow/issues/31579).++### Initial Type Hierarchy+TensorFlow generally adopts an incremental development method. This RFC aims to remain consistent with that.++Below are listed the major types presently used in TensorFlow. All types included in this list are subject to [normal compatibility rules](https://www.tensorflow.org/guide/versions), so they are unlikely to change in the future. It is therefore preferable to maintain a strict minimum of orthogonal declarations and carefully vet any additions.++Most of these symbols will not be initially exported as public symbols. Only internal submodules will be able to use unexported types. The unexported types may be gradually exposed under `tf.types` or under `tf.types.experimental`.++The initial type hierarchy is focused on V2 symbols. We expect to encounter places where these symbols would not be compatible with V1 code; in such cases, the V1 symbols will not be affected.++#### Types created by this RFC++These types will be added with the initial creation of the `tf.types` namespace.++* Core tensor types++  * `DType`+  * `Shape`+  * `Tensor` - generic dense tensor++    * `Symbol` - the regular graph tensor+    * `Value` - eager tensors++  * `TensorLike` - any type that can be implicitly converted to `Tensor` (see for example https://github.com/tensorflow/addons/blob/master/tensorflow_addons/utils/types.py)

I think that regardless of the implementation, the relationship between TensorLike, tf.convert_to_tensor and tf.register_tensor_conversion_function should be specified to avoid confusion.

Personally, I'm fine with TensorLike being non-extensible, i.e. if you want your custom type to be TensorLike, convert it to such explicitly.

mdanatg

comment created time in 2 days

Pull request review commenttensorflow/community

RFC: TensorFlow Canonical Type System

+# TensorFlow Canonical Type System++| Status        | Proposed       |+:-------------- |:---------------------------------------------------- |+| **RFC #**     | [208](https://github.com/tensorflow/community/pull/208)+| **Author(s)** | Dan Moldovan (mdan@google.com) |+| **Sponsor**   | Gaurav Jain (gjn@google.com)                 |+| **Updated**   | 2020-02-19                                           |++## Objective++This RFC proposes a new TensorFlow module and namespace (`tf.types`) dedicated to storing implementation-free type definitions, similar to C++ header files. This module has no other dependencies inside TensorFlow, so any other internal module can depend on it to ensure interoperability without the risk of creating circular dependencies. These definitions can also be used by external users, for example in pytype annotations.+The RFC focuses on the Python API, however the design should be reviewed with cross-language consistency in mind.+## Motivation++**Interoperability and composability**. A set of standard types that formalize an interface and decouples it from implementation ensures composability between components, especially when multiple implementations are involved.++**Supports the [acyclic dependencies principle](https://en.wikipedia.org/wiki/Acyclic_dependencies_principle)**. In many instances, circular dependencies are caused between low-level complex components that need to compose (e.g. autograph needs to recognize datasets, and datasets need to use autograph). Interface extraction is a common pattern for breaking such cycles.++**Supports pytype**. A set of static types that is consistent under Python’s `isinstance`/`issubclass` is required to support [PEP-484 type annotations](https://www.python.org/dev/peps/pep-0484/) in TensorFlow. This module can serve as the basis for that.++**Helps formalize requirements for new APIs**. Having a formal, implementation-independent definition for things such as tensors, variables, iterables, iterators makes it easy to document and test compatibility between APIs.++## User Benefit++Application developers may use these canonical definitions for pytype annotations.++Library developers can more easily define their API interfaces by referring to this namespace.++Developers of modules internal to TensorFlow can use this module to avoid creating circular dependencies.++## Design Proposal++### The `tf.types` Namespace / Module+All the declarations exposed under the `tf.types` namespace reside in the `python/types/*.py` module. These are [abstract base classes](https://docs.python.org/3.7/library/abc.html) with a bare minimum of method definitions and minimal or no implementation, which serve to formalize and document the contract of common types such as `Tensor`, `Variable`, etc.++These definitions may be used as PEP 484 type hints, although in some cases they may be type- or shape- erased (for example, `tf.types.Tensor` may not necessarily be parametrized by `dtype` or `shape`). Note however that designs which parametrize on shape do exist, see for instance [tensorflow#31579](https://github.com/tensorflow/tensorflow/issues/31579).++The type definitions are consistent with `isinstance` and `issubclass`. For example, `isinstance(tf.Tensor, tf.types.Tensor) == True`.++### General Principles+This module should not contain any implementation code. An advantage of that is that users exploring the implementation of specific types will not need to inspect this module. However, users who do not wish to inspect the code may visit the documentation of these generic types to better understand specifically what are the concrete subclasses of this type expected to do.++The `tf.types` module may depend on external packages (such as `numpy`) _strictly for the purpose of defining type annotations and documentation_. No dependencies to other TensorFlow interfaces are allowed. Any dependencies on external packages which themselves depend on TensorFlow are expressly forbidden.++Changes definitions inside `tf.types` must be approved by TensorFlow leads, and typically should be accompanied by an RFC.++All type declarations are compatible with [pytype](https://github.com/google/pytype).++It is recommended that internal and external type annotations, `isinstance` and `issubclass` checks use these types, eventually deprecating helpers like `tf.is_tensor`. However, concrete types continue to exist - for example, variables are instances of `tf.Variable`, which is now a subclass of `tf.types.Variable`.++Class type definitions define a minimum of abstract methods and properties which are required for pytype compatibility.++### Support for `tf.function`'s `input_signature`+The type system listed here can be expanded to allow input signatures using type annotations, see for instance [this thread](https://github.com/tensorflow/tensorflow/issues/31579).++### Initial Type Hierarchy+TensorFlow generally adopts an incremental development method. This RFC aims to remain consistent with that.++Below are listed the major types presently used in TensorFlow. All types included in this list are subject to [normal compatibility rules](https://www.tensorflow.org/guide/versions), so they are unlikely to change in the future. It is therefore preferable to maintain a strict minimum of orthogonal declarations and carefully vet any additions.++Most of these symbols will not be initially exported as public symbols. Only internal submodules will be able to use unexported types. The unexported types may be gradually exposed under `tf.types` or under `tf.types.experimental`.++The initial type hierarchy is focused on V2 symbols. We expect to encounter places where these symbols would not be compatible with V1 code; in such cases, the V1 symbols will not be affected.++#### Types created by this RFC++These types will be added with the initial creation of the `tf.types` namespace.++* Core tensor types++  * `DType`+  * `Shape`+  * `Tensor` - generic dense tensor++    * `Symbol` - the regular graph tensor+    * `Value` - eager tensors++  * `TensorLike` - any type that can be implicitly converted to `Tensor` (see for example https://github.com/tensorflow/addons/blob/master/tensorflow_addons/utils/types.py)++  * `Variable`++#### Potential types for subsequent implementation++These types are raised for discussion by this RFC, but are not part of the original implementation, unless they are strictly required for consistency (to be determined during the initial submission).++Many of these are expected to be required when breaking the cyclic dependencies that currently exist between submodules. However, it is hoped that opening them up for discussion early can help create a more coherent type system.++* Container types++  * `Composite` - low-level static structure (opaque to GraphDef/IR)+  * `Module` - builder for structures of `Variables` (invisible to GraphDef/IR)+  * `Optional` - basic programming construct+  * `List` - superclass for `TensorArray`, `Queue`, etc. (opaque to GraphDef/IR)++* Higher-level types+  * `Dataset` - ETL pipeline+  * `Iterator` - basic stateful programming construct+  * `Iterable` - basic stateless programming construct+  * `Function` - basic programming construct+  * `Error` - superclass of all TF-specific errors++ * Distributed types+   * `DistributedDataset` - collective ETL+   * `DistributedIterator` - collective iterator+   + * Low-level execution primitives+   * `Graph` - GraphDef/IR program+     * `FunctionGraph` - IR of a single concrete function++### Alternatives Considered+* N/A++### Performance Implications+* No performance implications expected. At most, we are adding a small number of levels to the class tree of some objects.

One option is to define these types without abc.ABCMeta as a metaclass, i.e. make them abstract by convention. The way abc.ABCMeta enforces an implementation (at instantiation time instead of definition time) is likely not relevant/useful for tf.types. Note however, that this also makes these type non-extensible which could be an issue for TensorLike.

mdanatg

comment created time in 2 days

Pull request review commenttensorflow/community

RFC: TensorFlow Canonical Type System

+# TensorFlow Canonical Type System++| Status        | Proposed       |+:-------------- |:---------------------------------------------------- |+| **RFC #**     | [208](https://github.com/tensorflow/community/pull/208)+| **Author(s)** | Dan Moldovan (mdan@google.com) |+| **Sponsor**   | Gaurav Jain (gjn@google.com)                 |+| **Updated**   | 2020-02-19                                           |++## Objective++This RFC proposes a new TensorFlow module and namespace (`tf.types`) dedicated to storing implementation-free type definitions, similar to C++ header files. This module has no other dependencies inside TensorFlow, so any other internal module can depend on it to ensure interoperability without the risk of creating circular dependencies. These definitions can also be used by external users, for example in pytype annotations.+The RFC focuses on the Python API, however the design should be reviewed with cross-language consistency in mind.+## Motivation++**Interoperability and composability**. A set of standard types that formalize an interface and decouples it from implementation ensures composability between components, especially when multiple implementations are involved.++**Supports the [acyclic dependencies principle](https://en.wikipedia.org/wiki/Acyclic_dependencies_principle)**. In many instances, circular dependencies are caused between low-level complex components that need to compose (e.g. autograph needs to recognize datasets, and datasets need to use autograph). Interface extraction is a common pattern for breaking such cycles.++**Supports pytype**. A set of static types that is consistent under Python’s `isinstance`/`issubclass` is required to support [PEP-484 type annotations](https://www.python.org/dev/peps/pep-0484/) in TensorFlow. This module can serve as the basis for that.++**Helps formalize requirements for new APIs**. Having a formal, implementation-independent definition for things such as tensors, variables, iterables, iterators makes it easy to document and test compatibility between APIs.++## User Benefit++Application developers may use these canonical definitions for pytype annotations.++Library developers can more easily define their API interfaces by referring to this namespace.++Developers of modules internal to TensorFlow can use this module to avoid creating circular dependencies.++## Design Proposal++### The `tf.types` Namespace / Module+All the declarations exposed under the `tf.types` namespace reside in the `python/types/*.py` module. These are [abstract base classes](https://docs.python.org/3.7/library/abc.html) with a bare minimum of method definitions and minimal or no implementation, which serve to formalize and document the contract of common types such as `Tensor`, `Variable`, etc.++These definitions may be used as PEP 484 type hints, although in some cases they may be type- or shape- erased (for example, `tf.types.Tensor` may not necessarily be parametrized by `dtype` or `shape`). Note however that designs which parametrize on shape do exist, see for instance [tensorflow#31579](https://github.com/tensorflow/tensorflow/issues/31579).++The type definitions are consistent with `isinstance` and `issubclass`. For example, `isinstance(tf.Tensor, tf.types.Tensor) == True`.++### General Principles+This module should not contain any implementation code. An advantage of that is that users exploring the implementation of specific types will not need to inspect this module. However, users who do not wish to inspect the code may visit the documentation of these generic types to better understand specifically what are the concrete subclasses of this type expected to do.++The `tf.types` module may depend on external packages (such as `numpy`) _strictly for the purpose of defining type annotations and documentation_. No dependencies to other TensorFlow interfaces are allowed. Any dependencies on external packages which themselves depend on TensorFlow are expressly forbidden.++Changes definitions inside `tf.types` must be approved by TensorFlow leads, and typically should be accompanied by an RFC.++All type declarations are compatible with [pytype](https://github.com/google/pytype).++It is recommended that internal and external type annotations, `isinstance` and `issubclass` checks use these types, eventually deprecating helpers like `tf.is_tensor`. However, concrete types continue to exist - for example, variables are instances of `tf.Variable`, which is now a subclass of `tf.types.Variable`.++Class type definitions define a minimum of abstract methods and properties which are required for pytype compatibility.++### Support for `tf.function`'s `input_signature`+The type system listed here can be expanded to allow input signatures using type annotations, see for instance [this thread](https://github.com/tensorflow/tensorflow/issues/31579).++### Initial Type Hierarchy+TensorFlow generally adopts an incremental development method. This RFC aims to remain consistent with that.++Below are listed the major types presently used in TensorFlow. All types included in this list are subject to [normal compatibility rules](https://www.tensorflow.org/guide/versions), so they are unlikely to change in the future. It is therefore preferable to maintain a strict minimum of orthogonal declarations and carefully vet any additions.++Most of these symbols will not be initially exported as public symbols. Only internal submodules will be able to use unexported types. The unexported types may be gradually exposed under `tf.types` or under `tf.types.experimental`.++The initial type hierarchy is focused on V2 symbols. We expect to encounter places where these symbols would not be compatible with V1 code; in such cases, the V1 symbols will not be affected.++#### Types created by this RFC++These types will be added with the initial creation of the `tf.types` namespace.++* Core tensor types++  * `DType`+  * `Shape`+  * `Tensor` - generic dense tensor++    * `Symbol` - the regular graph tensor+    * `Value` - eager tensors++  * `TensorLike` - any type that can be implicitly converted to `Tensor` (see for example https://github.com/tensorflow/addons/blob/master/tensorflow_addons/utils/types.py)++  * `Variable`++#### Potential types for subsequent implementation++These types are raised for discussion by this RFC, but are not part of the original implementation, unless they are strictly required for consistency (to be determined during the initial submission).++Many of these are expected to be required when breaking the cyclic dependencies that currently exist between submodules. However, it is hoped that opening them up for discussion early can help create a more coherent type system.++* Container types++  * `Composite` - low-level static structure (opaque to GraphDef/IR)+  * `Module` - builder for structures of `Variables` (invisible to GraphDef/IR)+  * `Optional` - basic programming construct+  * `List` - superclass for `TensorArray`, `Queue`, etc. (opaque to GraphDef/IR)++* Higher-level types+  * `Dataset` - ETL pipeline+  * `Iterator` - basic stateful programming construct+  * `Iterable` - basic stateless programming construct+  * `Function` - basic programming construct+  * `Error` - superclass of all TF-specific errors++ * Distributed types+   * `DistributedDataset` - collective ETL+   * `DistributedIterator` - collective iterator+   + * Low-level execution primitives+   * `Graph` - GraphDef/IR program+     * `FunctionGraph` - IR of a single concrete function++### Alternatives Considered+* N/A++### Performance Implications+* No performance implications expected. At most, we are adding a small number of levels to the class tree of some objects.

One caveat here is that isinstance checks involving ABCs are significantly slower than that with non-abstract classes. This is because abc.ABCMeta implements __instancecheck__ so isinstance has to jump back to pure Python instead of doing MRO traversal in C:

>>> import abc                                                                                                                                                                                
>>> class A(abc.ABC): pass                                                                                                                                                                                            
>>> class B: pass                                                                                                                                                                             
>>> class SA(A): pass                                                                                                                                                                         
>>> class SB(B): pass                                                                                                                                                                         
>>> sa = SA()                                                                                                                                                                                 
>>> sb = SB()                                                                                                                                                                                 
>>> %timeit isinstance(sa, A)                                                                                                                                                                 
237 ns ± 0.45 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
>>> %timeit isinstance(sb, B)                                                                                                                                                                 
51.2 ns ± 0.245 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)

While this is of course an artificial benchmark, and a single isintance check is still "just" ~200ns (on my fairly slow laptop), TensorFlow does a lot of such checks, and the slowdown might as well be noticeable in more realistic workloads.

mdanatg

comment created time in 2 days

Pull request review commenttensorflow/community

RFC: TensorFlow Canonical Type System

+# TensorFlow Canonical Type System++| Status        | Proposed       |+:-------------- |:---------------------------------------------------- |+| **RFC #**     | [208](https://github.com/tensorflow/community/pull/208)+| **Author(s)** | Dan Moldovan (mdan@google.com) |+| **Sponsor**   | Gaurav Jain (gjn@google.com)                 |+| **Updated**   | 2020-02-19                                           |++## Objective++This RFC proposes a new TensorFlow module and namespace (`tf.types`) dedicated to storing implementation-free type definitions, similar to C++ header files. This module has no other dependencies inside TensorFlow, so any other internal module can depend on it to ensure interoperability without the risk of creating circular dependencies. These definitions can also be used by external users, for example in pytype annotations.+The RFC focuses on the Python API, however the design should be reviewed with cross-language consistency in mind.+## Motivation++**Interoperability and composability**. A set of standard types that formalize an interface and decouples it from implementation ensures composability between components, especially when multiple implementations are involved.++**Supports the [acyclic dependencies principle](https://en.wikipedia.org/wiki/Acyclic_dependencies_principle)**. In many instances, circular dependencies are caused between low-level complex components that need to compose (e.g. autograph needs to recognize datasets, and datasets need to use autograph). Interface extraction is a common pattern for breaking such cycles.++**Supports pytype**. A set of static types that is consistent under Python’s `isinstance`/`issubclass` is required to support [PEP-484 type annotations](https://www.python.org/dev/peps/pep-0484/) in TensorFlow. This module can serve as the basis for that.++**Helps formalize requirements for new APIs**. Having a formal, implementation-independent definition for things such as tensors, variables, iterables, iterators makes it easy to document and test compatibility between APIs.++## User Benefit++Application developers may use these canonical definitions for pytype annotations.++Library developers can more easily define their API interfaces by referring to this namespace.++Developers of modules internal to TensorFlow can use this module to avoid creating circular dependencies.++## Design Proposal++### The `tf.types` Namespace / Module+All the declarations exposed under the `tf.types` namespace reside in the `python/types/*.py` module. These are [abstract base classes](https://docs.python.org/3.7/library/abc.html) with a bare minimum of method definitions and minimal or no implementation, which serve to formalize and document the contract of common types such as `Tensor`, `Variable`, etc.++These definitions may be used as PEP 484 type hints, although in some cases they may be type- or shape- erased (for example, `tf.types.Tensor` may not necessarily be parametrized by `dtype` or `shape`). Note however that designs which parametrize on shape do exist, see for instance [tensorflow#31579](https://github.com/tensorflow/tensorflow/issues/31579).++The type definitions are consistent with `isinstance` and `issubclass`. For example, `isinstance(tf.Tensor, tf.types.Tensor) == True`.++### General Principles+This module should not contain any implementation code. An advantage of that is that users exploring the implementation of specific types will not need to inspect this module. However, users who do not wish to inspect the code may visit the documentation of these generic types to better understand specifically what are the concrete subclasses of this type expected to do.++The `tf.types` module may depend on external packages (such as `numpy`) _strictly for the purpose of defining type annotations and documentation_. No dependencies to other TensorFlow interfaces are allowed. Any dependencies on external packages which themselves depend on TensorFlow are expressly forbidden.++Changes definitions inside `tf.types` must be approved by TensorFlow leads, and typically should be accompanied by an RFC.++All type declarations are compatible with [pytype](https://github.com/google/pytype).

Do you expect them to also be compatible with other type checkers, e.g. mypy or pyre?

mdanatg

comment created time in 2 days

Pull request review commenttensorflow/community

RFC: TensorFlow Canonical Type System

+# TensorFlow Canonical Type System++| Status        | Proposed       |+:-------------- |:---------------------------------------------------- |+| **RFC #**     | [208](https://github.com/tensorflow/community/pull/208)+| **Author(s)** | Dan Moldovan (mdan@google.com) |+| **Sponsor**   | Gaurav Jain (gjn@google.com)                 |+| **Updated**   | 2020-02-19                                           |++## Objective++This RFC proposes a new TensorFlow module and namespace (`tf.types`) dedicated to storing implementation-free type definitions, similar to C++ header files. This module has no other dependencies inside TensorFlow, so any other internal module can depend on it to ensure interoperability without the risk of creating circular dependencies. These definitions can also be used by external users, for example in pytype annotations.+The RFC focuses on the Python API, however the design should be reviewed with cross-language consistency in mind.+## Motivation++**Interoperability and composability**. A set of standard types that formalize an interface and decouples it from implementation ensures composability between components, especially when multiple implementations are involved.++**Supports the [acyclic dependencies principle](https://en.wikipedia.org/wiki/Acyclic_dependencies_principle)**. In many instances, circular dependencies are caused between low-level complex components that need to compose (e.g. autograph needs to recognize datasets, and datasets need to use autograph). Interface extraction is a common pattern for breaking such cycles.++**Supports pytype**. A set of static types that is consistent under Python’s `isinstance`/`issubclass` is required to support [PEP-484 type annotations](https://www.python.org/dev/peps/pep-0484/) in TensorFlow. This module can serve as the basis for that.++**Helps formalize requirements for new APIs**. Having a formal, implementation-independent definition for things such as tensors, variables, iterables, iterators makes it easy to document and test compatibility between APIs.++## User Benefit++Application developers may use these canonical definitions for pytype annotations.++Library developers can more easily define their API interfaces by referring to this namespace.++Developers of modules internal to TensorFlow can use this module to avoid creating circular dependencies.++## Design Proposal++### The `tf.types` Namespace / Module+All the declarations exposed under the `tf.types` namespace reside in the `python/types/*.py` module. These are [abstract base classes](https://docs.python.org/3.7/library/abc.html) with a bare minimum of method definitions and minimal or no implementation, which serve to formalize and document the contract of common types such as `Tensor`, `Variable`, etc.++These definitions may be used as PEP 484 type hints, although in some cases they may be type- or shape- erased (for example, `tf.types.Tensor` may not necessarily be parametrized by `dtype` or `shape`). Note however that designs which parametrize on shape do exist, see for instance [tensorflow#31579](https://github.com/tensorflow/tensorflow/issues/31579).++The type definitions are consistent with `isinstance` and `issubclass`. For example, `isinstance(tf.Tensor, tf.types.Tensor) == True`.++### General Principles+This module should not contain any implementation code. An advantage of that is that users exploring the implementation of specific types will not need to inspect this module. However, users who do not wish to inspect the code may visit the documentation of these generic types to better understand specifically what are the concrete subclasses of this type expected to do.++The `tf.types` module may depend on external packages (such as `numpy`) _strictly for the purpose of defining type annotations and documentation_. No dependencies to other TensorFlow interfaces are allowed. Any dependencies on external packages which themselves depend on TensorFlow are expressly forbidden.++Changes definitions inside `tf.types` must be approved by TensorFlow leads, and typically should be accompanied by an RFC.++All type declarations are compatible with [pytype](https://github.com/google/pytype).++It is recommended that internal and external type annotations, `isinstance` and `issubclass` checks use these types, eventually deprecating helpers like `tf.is_tensor`. However, concrete types continue to exist - for example, variables are instances of `tf.Variable`, which is now a subclass of `tf.types.Variable`.++Class type definitions define a minimum of abstract methods and properties which are required for pytype compatibility.++### Support for `tf.function`'s `input_signature`+The type system listed here can be expanded to allow input signatures using type annotations, see for instance [this thread](https://github.com/tensorflow/tensorflow/issues/31579).++### Initial Type Hierarchy+TensorFlow generally adopts an incremental development method. This RFC aims to remain consistent with that.++Below are listed the major types presently used in TensorFlow. All types included in this list are subject to [normal compatibility rules](https://www.tensorflow.org/guide/versions), so they are unlikely to change in the future. It is therefore preferable to maintain a strict minimum of orthogonal declarations and carefully vet any additions.++Most of these symbols will not be initially exported as public symbols. Only internal submodules will be able to use unexported types. The unexported types may be gradually exposed under `tf.types` or under `tf.types.experimental`.++The initial type hierarchy is focused on V2 symbols. We expect to encounter places where these symbols would not be compatible with V1 code; in such cases, the V1 symbols will not be affected.++#### Types created by this RFC

Are the specifics of each type out of scope for this RFC? i.e. which methods will be available on a Shape or Tensor?

mdanatg

comment created time in 2 days

Pull request review commenttensorflow/community

RFC: TensorFlow Canonical Type System

+# TensorFlow Canonical Type System++| Status        | Proposed       |+:-------------- |:---------------------------------------------------- |+| **RFC #**     | [208](https://github.com/tensorflow/community/pull/208)+| **Author(s)** | Dan Moldovan (mdan@google.com) |+| **Sponsor**   | Gaurav Jain (gjn@google.com)                 |+| **Updated**   | 2020-02-19                                           |++## Objective++This RFC proposes a new TensorFlow module and namespace (`tf.types`) dedicated to storing implementation-free type definitions, similar to C++ header files. This module has no other dependencies inside TensorFlow, so any other internal module can depend on it to ensure interoperability without the risk of creating circular dependencies. These definitions can also be used by external users, for example in pytype annotations.+The RFC focuses on the Python API, however the design should be reviewed with cross-language consistency in mind.+## Motivation++**Interoperability and composability**. A set of standard types that formalize an interface and decouples it from implementation ensures composability between components, especially when multiple implementations are involved.++**Supports the [acyclic dependencies principle](https://en.wikipedia.org/wiki/Acyclic_dependencies_principle)**. In many instances, circular dependencies are caused between low-level complex components that need to compose (e.g. autograph needs to recognize datasets, and datasets need to use autograph). Interface extraction is a common pattern for breaking such cycles.++**Supports pytype**. A set of static types that is consistent under Python’s `isinstance`/`issubclass` is required to support [PEP-484 type annotations](https://www.python.org/dev/peps/pep-0484/) in TensorFlow. This module can serve as the basis for that.++**Helps formalize requirements for new APIs**. Having a formal, implementation-independent definition for things such as tensors, variables, iterables, iterators makes it easy to document and test compatibility between APIs.++## User Benefit++Application developers may use these canonical definitions for pytype annotations.++Library developers can more easily define their API interfaces by referring to this namespace.++Developers of modules internal to TensorFlow can use this module to avoid creating circular dependencies.++## Design Proposal++### The `tf.types` Namespace / Module+All the declarations exposed under the `tf.types` namespace reside in the `python/types/*.py` module. These are [abstract base classes](https://docs.python.org/3.7/library/abc.html) with a bare minimum of method definitions and minimal or no implementation, which serve to formalize and document the contract of common types such as `Tensor`, `Variable`, etc.++These definitions may be used as PEP 484 type hints, although in some cases they may be type- or shape- erased (for example, `tf.types.Tensor` may not necessarily be parametrized by `dtype` or `shape`). Note however that designs which parametrize on shape do exist, see for instance [tensorflow#31579](https://github.com/tensorflow/tensorflow/issues/31579).++The type definitions are consistent with `isinstance` and `issubclass`. For example, `isinstance(tf.Tensor, tf.types.Tensor) == True`.++### General Principles+This module should not contain any implementation code. An advantage of that is that users exploring the implementation of specific types will not need to inspect this module. However, users who do not wish to inspect the code may visit the documentation of these generic types to better understand specifically what are the concrete subclasses of this type expected to do.++The `tf.types` module may depend on external packages (such as `numpy`) _strictly for the purpose of defining type annotations and documentation_. No dependencies to other TensorFlow interfaces are allowed. Any dependencies on external packages which themselves depend on TensorFlow are expressly forbidden.++Changes definitions inside `tf.types` must be approved by TensorFlow leads, and typically should be accompanied by an RFC.++All type declarations are compatible with [pytype](https://github.com/google/pytype).++It is recommended that internal and external type annotations, `isinstance` and `issubclass` checks use these types, eventually deprecating helpers like `tf.is_tensor`. However, concrete types continue to exist - for example, variables are instances of `tf.Variable`, which is now a subclass of `tf.types.Variable`.++Class type definitions define a minimum of abstract methods and properties which are required for pytype compatibility.++### Support for `tf.function`'s `input_signature`+The type system listed here can be expanded to allow input signatures using type annotations, see for instance [this thread](https://github.com/tensorflow/tensorflow/issues/31579).++### Initial Type Hierarchy+TensorFlow generally adopts an incremental development method. This RFC aims to remain consistent with that.++Below are listed the major types presently used in TensorFlow. All types included in this list are subject to [normal compatibility rules](https://www.tensorflow.org/guide/versions), so they are unlikely to change in the future. It is therefore preferable to maintain a strict minimum of orthogonal declarations and carefully vet any additions.++Most of these symbols will not be initially exported as public symbols. Only internal submodules will be able to use unexported types. The unexported types may be gradually exposed under `tf.types` or under `tf.types.experimental`.++The initial type hierarchy is focused on V2 symbols. We expect to encounter places where these symbols would not be compatible with V1 code; in such cases, the V1 symbols will not be affected.++#### Types created by this RFC++These types will be added with the initial creation of the `tf.types` namespace.++* Core tensor types++  * `DType`+  * `Shape`+  * `Tensor` - generic dense tensor++    * `Symbol` - the regular graph tensor+    * `Value` - eager tensors++  * `TensorLike` - any type that can be implicitly converted to `Tensor` (see for example https://github.com/tensorflow/addons/blob/master/tensorflow_addons/utils/types.py)

How will TensorLike interact with custom ->Tensor conversion machinery? If a user registers a new conversion function, should they also register a "virtual" subclass of TensorLike?

mdanatg

comment created time in 2 days

Pull request review commenttensorflow/community

RFC: TensorFlow Canonical Type System

+# TensorFlow Canonical Type System++| Status        | Proposed       |+:-------------- |:---------------------------------------------------- |+| **RFC #**     | [208](https://github.com/tensorflow/community/pull/208)+| **Author(s)** | Dan Moldovan (mdan@google.com) |+| **Sponsor**   | Gaurav Jain (gjn@google.com)                 |+| **Updated**   | 2020-02-19                                           |++## Objective++This RFC proposes a new TensorFlow module and namespace (`tf.types`) dedicated to storing implementation-free type definitions, similar to C++ header files. This module has no other dependencies inside TensorFlow, so any other internal module can depend on it to ensure interoperability without the risk of creating circular dependencies. These definitions can also be used by external users, for example in pytype annotations.+The RFC focuses on the Python API, however the design should be reviewed with cross-language consistency in mind.+## Motivation++**Interoperability and composability**. A set of standard types that formalize an interface and decouples it from implementation ensures composability between components, especially when multiple implementations are involved.++**Supports the [acyclic dependencies principle](https://en.wikipedia.org/wiki/Acyclic_dependencies_principle)**. In many instances, circular dependencies are caused between low-level complex components that need to compose (e.g. autograph needs to recognize datasets, and datasets need to use autograph). Interface extraction is a common pattern for breaking such cycles.++**Supports pytype**. A set of static types that is consistent under Python’s `isinstance`/`issubclass` is required to support [PEP-484 type annotations](https://www.python.org/dev/peps/pep-0484/) in TensorFlow. This module can serve as the basis for that.++**Helps formalize requirements for new APIs**. Having a formal, implementation-independent definition for things such as tensors, variables, iterables, iterators makes it easy to document and test compatibility between APIs.++## User Benefit++Application developers may use these canonical definitions for pytype annotations.++Library developers can more easily define their API interfaces by referring to this namespace.++Developers of modules internal to TensorFlow can use this module to avoid creating circular dependencies.++## Design Proposal++### The `tf.types` Namespace / Module+All the declarations exposed under the `tf.types` namespace reside in the `python/types/*.py` module. These are [abstract base classes](https://docs.python.org/3.7/library/abc.html) with a bare minimum of method definitions and minimal or no implementation, which serve to formalize and document the contract of common types such as `Tensor`, `Variable`, etc.++These definitions may be used as PEP 484 type hints, although in some cases they may be type- or shape- erased (for example, `tf.types.Tensor` may not necessarily be parametrized by `dtype` or `shape`). Note however that designs which parametrize on shape do exist, see for instance [tensorflow#31579](https://github.com/tensorflow/tensorflow/issues/31579).++The type definitions are consistent with `isinstance` and `issubclass`. For example, `isinstance(tf.Tensor, tf.types.Tensor) == True`.++### General Principles+This module should not contain any implementation code. An advantage of that is that users exploring the implementation of specific types will not need to inspect this module. However, users who do not wish to inspect the code may visit the documentation of these generic types to better understand specifically what are the concrete subclasses of this type expected to do.++The `tf.types` module may depend on external packages (such as `numpy`) _strictly for the purpose of defining type annotations and documentation_. No dependencies to other TensorFlow interfaces are allowed. Any dependencies on external packages which themselves depend on TensorFlow are expressly forbidden.++Changes definitions inside `tf.types` must be approved by TensorFlow leads, and typically should be accompanied by an RFC.++All type declarations are compatible with [pytype](https://github.com/google/pytype).++It is recommended that internal and external type annotations, `isinstance` and `issubclass` checks use these types, eventually deprecating helpers like `tf.is_tensor`. However, concrete types continue to exist - for example, variables are instances of `tf.Variable`, which is now a subclass of `tf.types.Variable`.++Class type definitions define a minimum of abstract methods and properties which are required for pytype compatibility.++### Support for `tf.function`'s `input_signature`+The type system listed here can be expanded to allow input signatures using type annotations, see for instance [this thread](https://github.com/tensorflow/tensorflow/issues/31579).++### Initial Type Hierarchy+TensorFlow generally adopts an incremental development method. This RFC aims to remain consistent with that.++Below are listed the major types presently used in TensorFlow. All types included in this list are subject to [normal compatibility rules](https://www.tensorflow.org/guide/versions), so they are unlikely to change in the future. It is therefore preferable to maintain a strict minimum of orthogonal declarations and carefully vet any additions.++Most of these symbols will not be initially exported as public symbols. Only internal submodules will be able to use unexported types. The unexported types may be gradually exposed under `tf.types` or under `tf.types.experimental`.++The initial type hierarchy is focused on V2 symbols. We expect to encounter places where these symbols would not be compatible with V1 code; in such cases, the V1 symbols will not be affected.++#### Types created by this RFC++These types will be added with the initial creation of the `tf.types` namespace.++* Core tensor types++  * `DType`+  * `Shape`+  * `Tensor` - generic dense tensor++    * `Symbol` - the regular graph tensor+    * `Value` - eager tensors++  * `TensorLike` - any type that can be implicitly converted to `Tensor` (see for example https://github.com/tensorflow/addons/blob/master/tensorflow_addons/utils/types.py)++  * `Variable`++#### Potential types for subsequent implementation++These types are raised for discussion by this RFC, but are not part of the original implementation, unless they are strictly required for consistency (to be determined during the initial submission).++Many of these are expected to be required when breaking the cyclic dependencies that currently exist between submodules. However, it is hoped that opening them up for discussion early can help create a more coherent type system.++* Container types++  * `Composite` - low-level static structure (opaque to GraphDef/IR)+  * `Module` - builder for structures of `Variables` (invisible to GraphDef/IR)+  * `Optional` - basic programming construct

Is this an alias to typing.Optional or a separate type? If the latter, could you give an example when one might need a tf.types.Optional and not typing.Optional?

mdanatg

comment created time in 2 days

push eventselectel/pyte

Mickaël Schoentgen

commit sha cf3f63fd8b69e071ce459e3452b8f63decfde797

Fix ResourceWarning: unclosed file in setup.py Signed-off-by: Mickaël Schoentgen <contact@tiger-222.fr>

view details

push time in 6 days

PR merged selectel/pyte

Fix ResourceWarning: unclosed file in setup.py

Signed-off-by: Mickaël Schoentgen contact@tiger-222.fr

+2 -1

2 comments

1 changed file

BoboTiG

pr closed time in 6 days

pull request commentselectel/pyte

Fix ResourceWarning: unclosed file in setup.py

Thanks, and sorry for the delay!

BoboTiG

comment created time in 6 days

push eventselectel/pyte

Jason Green

commit sha 4672869d175cea2f80d124f6153fdcc62b53692b

make AIXTERM colors return "bright" and unset "bold" so we can decide what to do with the color

view details

Jason Green

commit sha d1d0ba0a731261cd949474ce23b282a8457593cc

updated test for AIXTERM no longer setting bold

view details

push time in 6 days

PR merged selectel/pyte

make AIXTERM colors return "bright" and unset "bold" so we can decide…

Ran into an issue writing a terminal emulator for a screen on a Raspberry Pi. There was no way to detect bright colours if the bold attribute was already set. This change allows for the code doing the formatting to decide how to interpret fg, bg, bold combos when AIXTERM is involved.

I updated the CSS for the example, but didn't actually add any bright colours, they're just copies, mostly so the JavaScript wouldn't fail.

+42 -30

1 comment

4 changed files

jason-green-io

pr closed time in 6 days

pull request commenttensorflow/tensorflow

Fix: fixed LSTMBlockCell cuda kernel

Thanks for the fix!

vladbataev

comment created time in 10 days

issue commentdeepmind/sonnet

The test TrainableStateTest.testForCore fails consistently

@loopylangur could you upgrade tree and let us know if this helps?

loopylangur

comment created time in 24 days

issue commentdeepmind/sonnet

The test TrainableStateTest.testForCore fails consistently

Hi @loopylangur, the failure you're seeing is a result of switching Sonnet 2 to deepmind/tree which did not (until recently, see deepmind/tree@66ace75ebec22c1bacb4a57446b0b5a4db254104) work well with wrapt.ObjectProxy objects used by tf.AutoTrackable.

I will do a bugfix release of tree in the coming days which should fix the issue.

loopylangur

comment created time in 25 days

issue openeddavidhalter/jedi

_compatibility.find_module behavior is not aligned with importlib.find_spec

I've been trying to use Jedi with a VFS (see discussion #1419), and discovered that the behavior of _compatibility.find_module when importing a submodule is different from importlib.find_spec.

Reproducer (using Python 3.7):

import importlib
import sys

import jedi


class Finder:

    def find_spec(self, fullname, path, target=None):
        print(fullname, path, target)
        return None


sys.meta_path.insert(0, Finder())


importlib.util.find_spec("foo.bar")
# ==> foo None None
# ==> foo.bar ['/tmp/foo'] None



jedi.Script(
    "import foo.bar; foo.bar.boo",
    environment=jedi.api.environment.InterpreterEnvironment()
).completions()
# ==> foo None None
# ==> bar ['/tmp/foo'] None

I think the difference is due to the way find_module_py34 and find_module_py33 use the full_name argument. During import of foo these the values of these arguments are equal, but when we get to foo.bar, string becomes just "bar" whereas full_name keeps "foo.bar". And for some reason find_module_py* functions prefer string over full_name:

https://github.com/davidhalter/jedi/blob/bec87f7ff82b0731713c6520a14c213341b4cecf/jedi/_compatibility.py#L79 https://github.com/davidhalter/jedi/blob/bec87f7ff82b0731713c6520a14c213341b4cecf/jedi/_compatibility.py#L89 https://github.com/davidhalter/jedi/blob/bec87f7ff82b0731713c6520a14c213341b4cecf/jedi/_compatibility.py#L112

Does this look like a bug in find_module_py* functions or is the bug somewhere upstream, i.e. in the caller of find_module? Happy to submit a PR with a fix regardless.

created time in a month

fork superbobry/jedi

Awesome autocompletion and static analysis library for python.

http://jedi.readthedocs.io

fork in a month

pull request commentapache/spark

[SPARK-22674][PYTHON] Removed the namedtuple pickling patch

@HyukjinKwon I think you might still want to merge this eventually. Closing the PR will only make the issue harder to discover.

superbobry

comment created time in 2 months

issue commentsuperbobry/pareto

BCa bootstrap is wrong

Thanks for the bug report! I'm not actively looking after pareto these days and will happily accept a PR fixing that.

edwintorok

comment created time in 2 months

issue commenttensorflow/tensorflow

Using tf.contrib.framework.nest functions in TF2.0

Feel free to close the issue and open any new ones on the tree issue tracker.

aravic

comment created time in 3 months

issue commenttensorflow/tensorflow

Using tf.contrib.framework.nest functions in TF2.0

Hi @aravic, we have recently released a subset of tf.contrib.framework.nest functions as a separate library. It does not have exactly the same API as nest but it is fairly close, and most of the functions are there.

aravic

comment created time in 3 months

created tagdeepmind/tree

tag0.1.1

tree is a library for working with nested data structures

created time in 4 months

release deepmind/tree

0.1.1

released time in 4 months

issue commenttensorflow/tensorflow

Share memory between numpy and tensorflow doesn't work

NumPy arrays are mutable whereas tensors aren't, therefore .numpy() has to copy to avoid breaking the runtime's invariants. In theory, a NumPy array could be flagged as non-writable (via a.flags["WRITABLE"]), but unfortunately a lot of internal (and I assume external) users already rely on the mutability of .numpy().

Two caveats:

Tensor supports the buffer interface, so you could get a readonly NumPy array via

>>> t = tf.constant([42])
>>> a = np.asarray(memoryview(t))
>>> a
array([42], dtype=int32)

Passing a CPU tensor directly to any NumPy APIs is zero-copy, because NumPy uses buffer interface behind the scenes (but NumPy will allocate a new array for the result):

>>> np.square(t)
array([1764], dtype=int32)
VoVAllen

comment created time in 4 months

pull request commenttensorflow/tensorflow

MKL DNN: Fix issues with hadoopFileSystem load error message

Thanks for the heads up @penpornk!

ashraf-bhuiyan

comment created time in 4 months

issue commentselectel/pyte

screem.display not correct when feed data has many '\x1b[C' chars

Could you provide a reproducer please? Are you driving the Screen manually or from a pty?

JJJJJJJJames

comment created time in 4 months

Pull request review commenttensorflow/tfjs

Only call register_prelu_op with open-source TensorFlow 1.X

 def prelu_fn(*args):   with graph.as_default():     prelu_fn(tf.constant(1.0), tf.constant(1.0)) -register_prelu_op()++if tf.version.GIT_VERSION != 'unknown' and tf.__version__ <= '2.1':

Nope, changed to <. Thanks for spotting that!

superbobry

comment created time in 5 months

push eventsuperbobry/tfjs

Sergei Lebedev

commit sha 2555196de6e544d5048716f93320c2f1a6f848b5

Only call register_prelu_op with open-source TensorFlow 1.X and 2.0 register_prelu_op is a workaround for the MetaGraph validity check which requires all GraphDef ops to be registered. This check has been lifted in tensorflow/tensorflow@8b287e23.

view details

push time in 5 months

push eventsuperbobry/tfjs

Sergei Lebedev

commit sha 87a07dae60931e6a10ced15d5e6d5d6cb230f6fd

Only call register_prelu_op with open-source TensorFlow 1.X and 2.0 register_prelu_op is a workaround for the MetaGraph validity check which requires all GraphDef ops to be registered. This check has been lifted in tensorflow/tensorflow@8b287e23.

view details

push time in 5 months

push eventsuperbobry/tfjs

Sergei Lebedev

commit sha 3235e972022103ee8f3cddcee494a98f6a5a9bc0

Only call register_prelu_op with open-source TensorFlow 1.X register_prelu_op is a workaround for the MetaGraph validity check which requires all GraphDef ops to be registered. This check has been lifted in tensorflow/tensorflow@8b287e23.

view details

push time in 5 months

PR opened tensorflow/tfjs

Only call register_prelu_op with open-source TensorFlow 1.X

register_prelu_op is a workaround for the MetaGraph validity check which requires all GraphDef ops to be registered. This check has been lifted in tensorflow/tensorflow@8b287e23.

+10 -3

0 comment

1 changed file

pr created time in 5 months

create barnchsuperbobry/tfjs

branch : optionally-disable-prelu-registration

created branch time in 5 months

fork superbobry/tfjs

A WebGL accelerated JavaScript library for training and deploying ML models.

https://js.tensorflow.org

fork in 5 months

more