profile
viewpoint
Sergei Lebedev superbobry Software Engineer at @deepmind London, UK

google/jax 9742

Composable transformations of Python+NumPy programs: differentiate, vectorize, JIT to GPU/TPU, and more

deepmind/sonnet 8520

TensorFlow-based neural network library

pytries/marisa-trie 773

Static memory-efficient Trie-like structures for Python (2.x and 3.x) based on marisa-trie C++ library.

deepmind/dm-haiku 740

JAX-based neural network library

selectel/pyte 441

Simple VTXXX-compatible linux terminal emulator

pytries/datrie 437

Fast, efficiently stored Trie for Python. Uses libdatrie.

slacy/minimongo 316

A lightweight, Pythonic, Object Oriented Interface to MongoDB.

selectel/yawndb 128

YAWNDB is an in-memory circular array database written in Erlang.

deepmind/tree 96

tree is a library for working with nested data structures

selectel/tempo 69

NIF-based date and time parsing and formatting for Erlang.

delete branch superbobry/google-modes

delete branch : clif-mode

delete time in 2 days

PR opened codemirror/google-modes

[clif] Added indentation support

As with the grammar itself, the majority of the logic has been adopted from the Python mode.

Note that both modes correctly (but annoyingly) ignore comments when computing indentation, e.g. here

class A:
  x: int

# Foo bar. <CURSOR>

The suggested indentation level will be 2, because according to the spec, comments do not affect indentation. Yet ~any user will expect the suggested indentation level to match that of the comment.

+529 -938

0 comment

5 changed files

pr created time in 3 days

push eventsuperbobry/google-modes

Sergei Lebedev

commit sha 8d0a73b70243f7a82e25e57dd507e272af12ba2f

[clif] Added indentation support As with the grammar itself, the majority of the logic has been adopted from the Python mode.

view details

push time in 3 days

push eventsuperbobry/google-modes

Sergei Lebedev

commit sha 3b87d7b206c465348b1f30bf698aee87c9df9473

[py] Fixed parsing of relative from-imports

view details

Marijn Haverbeke

commit sha 802cd98b3d36529ac2f711f0022c888bb6a8de0c

[c and c++ modes] Allow concatenated strings Closes #313

view details

Sergei Lebedev

commit sha 840258fe1504df7bf8c624a77cb5d9a3daece3bc

Added initial version of the CLIF grammar The grammar is modeled after the pyparsing definition and has to be relaxed to be practical for syntax highlighting. See https://google.github.io/clif/clif/python/primer.html.

view details

Sergei Lebedev

commit sha 4af3267d3313d7a28ce3321505056d030032691d

[clif] Relaxed the grammar and added tests

view details

Marijn Haverbeke

commit sha 8b68488e4e0b8dbf8852e0802886db77b8f8c992

Load stylesheets for addons used in demo Issue #314

view details

Marijn Haverbeke

commit sha 117e84f76828e5ce657617f1aa84603ccae035c8

[go mode] Support [...] slice syntax Closes #315

view details

Marijn Haverbeke

commit sha 9160a280440170e7304e0ae327191d6d78b6e010

[cpp mode] Allow declNames to be qualified operator names Closes #316

view details

push time in 3 days

pull request commentcodemirror/google-modes

Added a mode for CLIF

Awesome, thank you so much!

superbobry

comment created time in 4 days

pull request commentcodemirror/google-modes

Added a mode for CLIF

Sure, the problem is that it's already included and enabled in index.html, yet there're no gutter icons nor markup in the DOM for neither of the languages.

https://github.com/codemirror/google-modes/blob/4af3267d3313d7a28ce3321505056d030032691d/index.html#L36-L36

https://github.com/codemirror/google-modes/blob/4af3267d3313d7a28ce3321505056d030032691d/index.html#L940-L941

superbobry

comment created time in 4 days

pull request commentcodemirror/google-modes

Added a mode for CLIF

@marijnh Thanks for merging this! Do you have any pointers on how I could enable fold icons in the gutter locally?

superbobry

comment created time in 4 days

pull request commentcodemirror/google-modes

Added a mode for CLIF

I'd be keen to add indentation-based code folding, but the Demo HTML doesn't seem to show the fold arrow in the gutter (see screenshot).

image

superbobry

comment created time in 4 days

PR opened codemirror/google-modes

Added a mode for CLIF

CLIF is a tool for exposing C++ code to various languages via auto-generated wrappers. This PR adds a mode for PyCLIF, a Python dialect implemented in google/clif.

The implementation is based on the pyparsing grammar.

Note that the mode does not currently support folding.

+1288 -2

0 comment

7 changed files

pr created time in 4 days

create barnchsuperbobry/google-modes

branch : clif-mode

created branch time in 4 days

delete branch superbobry/typeshed

delete branch : fix-asyncio.base_futures-contextvars

delete time in 15 days

pull request commentpython/typeshed

Made contextvars import in asyncio.base_futures conditional

Ooops, sorry, done.

superbobry

comment created time in 15 days

push eventsuperbobry/typeshed

Sergei Lebedev

commit sha 0f1597dec5817947dfa5906cbeee69c986e6ee56

Made contextvars import in asyncio.base_futures conditional contextvars are only available in 3.7+.

view details

push time in 15 days

push eventsuperbobry/typeshed

Sergei Lebedev

commit sha 2c6f92013f0b6bf05515d00e13a70ca285072e5c

Made contextvars import in asyncio.base_futures conditional contextvars are only available in 3.7+.

view details

push time in 15 days

PR opened python/typeshed

Made contextvars import in asyncio.base_futures conditional

contextvars are only available in 3.7+.

+7 -2

0 comment

1 changed file

pr created time in 15 days

create barnchsuperbobry/typeshed

branch : fix-asyncio.base_futures-contextvars

created branch time in 15 days

fork superbobry/typeshed

Collection of library stubs for Python, with static types

fork in 15 days

pull request commentcodemirror/google-modes

[py] Fixed parsing of relative from-imports

I'm not quite sure why the previous grammar didn't work, so if you have any insights, that'd be really helpful.

superbobry

comment created time in 23 days

create barnchsuperbobry/google-modes

branch : fix-relative-from-import

created branch time in 23 days

issue commentdeepmind/dm-haiku

Haiku LSTM vs Torch LSTM

Hi Pranav,

Yes, hk.LSTM is a cell or a core in Haiku lingo, i.e. a single-step computation which expects a tensor of shape [B, D] or [D] where B is the batch dimension. In order to apply a core to the whole sequence (and get PyTorch's LSTM) you need to use one of the unroll functions, whether static_* or dynamic_* doesn't really matter.

Could you post the performance differences you've observed here and include the code (if possible)?

Disclaimer: I would expect Haiku to be slower than PyTorch at least on GPU, because PyTorch uses a fused CuDNN-RNN kernel for both forward and backward computations, whereas Haiku implements LSTM only in terms of XLA primitives.

pranavsubramani

comment created time in 24 days

issue commenttensorflow/tensorflow

tf.gfile.GFile deadlocks on HDFS access after a fork()

I suspect the only fix is to avoid initializing HDFS prior to forking.

superbobry

comment created time in a month

PR opened codemirror/google-modes

[py] Allowed trailing comma and newlines in a parenthesized from-import

Note that Python mode currently has multiple special-cased contexts for parenthesized "things" where newlines are ignored. According to https://docs.python.org/3/reference/lexical_analysis.html#implicit-line-joining whitespace and comments are always ignored inside parens/brackets/braces. Perhaps, we could clean this up in a followup.

+12 -5

0 comment

3 changed files

pr created time in a month

push eventsuperbobry/google-modes

Sergei Lebedev

commit sha cf04e27dddd420225a838f0db655fa114ce34f8b

[python mode] Added support for f-strings Expressions in f-string replacements are now highlighted as such. Fixes #301.

view details

Sergei Lebedev

commit sha 65941bc035f931b57066aecf03a13e9a4f4cf1a8

Added a few more tests including a case with := Strictly speaking f"{x := 42:42d}" is an invalid f-string, but it doesn't mean we shouldn't highlight it properly.

view details

Sergei Lebedev

commit sha 91289120d452265a26140e9e091791562ce79f3d

Added the = specifier

view details

Sergei Lebedev

commit sha b963a00e5831ef0166321b352ed4e9ab0c133f71

[py] Allowed implicit joining of normal and f-strings Closes #304.

view details

Sergei Lebedev

commit sha 9bbd088fc10648c0bf52073e054ffa36f9e964e5

[py] Fixed parsing of extended slices Prior to this commit the following failed to parse and as a result affected auto-indentation for x in xs[i::j]: # <ENTER> Interestingly, the new tests for slices pass with and without the grammar change because IIUC the variables are correctly tokenized regardless of the surrounding context.

view details

Sergei Lebedev

commit sha 5650065ce7b0c40bdf0b98d7068514b31692a60c

[py] Fixed parsing of type annotations in variable declarations Note that after this commit type annotations are no longer recognized in arbitrary target context, e.g. for x: int in ...: ... is invalid (as it should be).

view details

Sergei Lebedev

commit sha 0c562e8e7f0a3b735d452638ae8a0a314ee6a664

[py] Allowed arbitrary target in with statement See added test for an example which was previously illegal.

view details

Marijn Haverbeke

commit sha 233582bb814f57959cd536b2abf2ff9465f48908

[cpp mode] Allow base declarations for enums Issue #307

view details

Marijn Haverbeke

commit sha 970284f78973f5bff0ec11d8b2d602f0618c8041

Fix bad interaction between ... and type lookahead Issue #307

view details

Marijn Haverbeke

commit sha ffd874ccf9e0047f43231ed3418650989a0c652e

Support line-broken constructor names Closes #308

view details

Marijn Haverbeke

commit sha a941e14efbc30fb9d3aa46f5efa7c5579f5b4759

Allow overloading of == operator Closes #309

view details

Marijn Haverbeke

commit sha afe9e8ff2d0b1efd2362f028abfe0ae0319f630e

[ts mode] Add support for prefix type assertions Closes #310

view details

Sergei Lebedev

commit sha c55ac15f63b464e5390ab445807e65bd93d4ae55

[py] Allowed trailing comma and newlines in a parenthesized from-import Note that Python mode currently has multiple special-cased contexts for parenthesized "things" where newlines are ignored. According to https://docs.python.org/3/reference/lexical_analysis.html#implicit-line-joining whitespace and comments are always ignored inside parens/brackets/braces. Perhaps, we could clean this up in a followup.

view details

push time in a month

issue closedselectel/pyte

Need example like webterm but working only locally

For CudaText plugin I want to use Pyte to run Bash process and catch its output into CudaText panel-editor. CudaText plugin has input field which text I must put to Bash.

I cannot make example to test. (No XP with asyncio and Pyte). webterm example is using HTTP and asyncio http, but I need an example which works in pure Linux terminal. so I can run in terminal - "python3 nnnnn.py" and

  • it runs Bash and Bash output is shown in some window (eg original terminal)
  • I somehow enter commands to running bash, eg "ls"

closed time in a month

Alexey-T

issue commentselectel/pyte

Need example like webterm but working only locally

I'm sorry, I'm not able to provide examples on demand. Feel free to ask a more specific question on StackOverflow.

Alexey-T

comment created time in a month

issue commentdeepmind/sonnet

How to install to Jetson

Interesting. I think the wheels for Sonnet 2 are platform-agnostic because Sonnet 2 does not have any native code. So you only really need to build tree.

n-kats

comment created time in a month

issue closeddeepmind/sonnet

How to install to Jetson

The cpu architecture of Nvidia Jetson is aarm64. So, I failed to install sonnet to Jetson. I read the following message:

unable to execute 'bazel': No such file or directory
error: command 'bazel' failed with exit status 1 

I installed bazel. I downloaded it from the release page of bazel, moved it to /bin/bazel, and made it executable by chmod. (I used https://github.com/bazelbuild/bazel/releases/download/3.4.1/bazel-3.4.1-linux-arm64) After then, I succeeded to install sonnet. Is this way correct?

I think someone also will try install sonnet to Jetson. So, the following may be useful:

  • write how to install to Jetson(using bazel)
  • add wheels for aarm64(dm-tree for aarm64 is missing in pypi).

Thank you.

closed time in a month

n-kats

issue commentdeepmind/sonnet

How to install to Jetson

Thanks for writing this up!

Neither Sonnet 1 nor tree have wheels for aarm64, and I'm afraid this will remain the case for a while. So, the only way of installing Sonnet 1 on Jetson is indeed by downloading Bazel (as you did) and building both tree and Sonnet 1 from source. Note that you'll likely have to build TensorFlow from source as well for the same reason.

n-kats

comment created time in a month

issue closedsuperbobry/pareto

Fatal error: exception Invalid_argument "KolmogorovSmirnov.two_sample: ties are not allowed"

Can't you provide a more robust KS-test implementation?

closed time in 2 months

UnixJunkie

issue commentsuperbobry/pareto

Fatal error: exception Invalid_argument "KolmogorovSmirnov.two_sample: ties are not allowed"

KS is indeed nonparameteric, but IIRC it is intended to be used with continuous random variables which is why ties are not allowed.

UnixJunkie

comment created time in 2 months

issue commentsuperbobry/pareto

Fatal error: exception Invalid_argument "KolmogorovSmirnov.two_sample: ties are not allowed"

Thanks for reporting this. I haven't used KS test for a while, but I vaguely recall that it cannot be applied to samples with ties, therefore the error.

UnixJunkie

comment created time in 2 months

issue commentselectel/pyte

Write some command to bash and get its result

Hey Alexey, webterm.py does that. It could be a bit hard to comprehend so if you are still unsure after giving it a go, please followup on StackOverflow. The issue tracker is not the best place to ask these kind of questions.

Alexey-T

comment created time in 2 months

push eventselectel/pyte

Sergei Lebedev

commit sha e295a589077f055b7d13a213c3aeb3dca88451bb

Remove b prefix from the tutorial Closes #132

view details

push time in 2 months

issue closedselectel/pyte

TypeError: cannot use a string pattern on a bytes-like object, when running Tutorial

Hi, I just installed pyte on python 3.7.5, windows 10, hoping to make my own terminal GUI using pygame but I ran across this error when running the tutorial.

C:\Users\ahmed>python
Python 3.7.5 (tags/v3.7.5:5c02a39a0b, Oct 15 2019, 00:11:34) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from __future__ import unicode_literals
>>> import pyte
>>> screen = pyte.Screen(80, 24)
>>> stream = pyte.Stream(screen)
>>> stream.feed(b"Hello World!")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\ahmed\AppData\Local\Programs\Python\Python37\lib\site-packages\pyte\streams.py", line 189, in feed
    match = match_text(data, offset)
TypeError: cannot use a string pattern on a bytes-like object

Thank you, Ahmed Khalf.

closed time in 2 months

ahmedkhalf

issue closedselectel/pyte

Docs are blurry here

set_margins(top=None, bottom=None)[source] Select top and bottom margins for the scrolling region.

Parameters: top (int) – the smallest line number that is scrolled. bottom (int) – the biggest line number that is scrolled.

===== Very blurry text. I don't get by text, what is the margin? Maybe the number1 is the minimal scroll position (for scroll cmds) and number2 is maximal?

Need new text, pls

closed time in 2 months

Alexey-T

issue commentselectel/pyte

Docs are blurry here

Hey Alexey, this method implements the DECSTBM CSI sequence, see e.g. here and here. I'd welcome a PR with better wording ;)

Alexey-T

comment created time in 2 months

PR closed tensorflow/tensorflow

Reviewers
Prefer Tensor._shape_tuple over Tensor._shape_as_list cla: yes size:S stat:awaiting response

This PR removes the Tensor._shape_as_list() method. Tensor._shape_tuple() provides the same functionality and its use is more prevelent throughout the codebase. Since it is a private method this removal should not be a breaking change.

This change also improves performance of Tensor._shape_tuple() and ResourceVariable._shape_tuple() by about 20-25% as it avoids one list creation.

+19 -19

2 comments

4 changed files

lgeiger

pr closed time in 3 months

Pull request review commenttensorflow/tensorflow

Prefer Tensor._shape_tuple over Tensor._shape_as_list

 def __iter__(self):     return _TensorIterator(self, shape[0])    def _shape_as_list(self):-    if self.shape.ndims is not None:-      return [dim.value for dim in self.shape.dims]-    else:+    shape = self._shape_tuple()+    if shape is None:       return None+    return list(shape)    def _shape_tuple(self):-    shape = self._shape_as_list()-    if shape is None:+    try:+      return tuple(dim.value for dim in self.shape.dims)

I agree that pybind11 changes are best left out of this PR. However, I don't think this PR alone is worth merging.

Reassigning to @jaingaurav for a second opinion.

lgeiger

comment created time in 3 months

pull request commentdeepmind/sonnet

Carefully move some internal methods of recurrent.py to a dedicated utils file

Thanks for the PR, Sergey!

I understand your concerns regarding the size of recurrent.py. Big modules could be tricky to work with, but I'm not sure that splitting recurrent.py into multiple smaller modules would significantly improve readability/maintainability. Having internal functions defined closer to the usage sites makes refactoring easier; _ names clearly indicate that these are not designed for use outside of Sonnet. So, all in all, I'd rather not merge this. I hope it's okay. That said, please do not hesitate to tackle #161, we'd welcome a fix for that!

I’ve noticed quite a few TODOs in recurrent.py. Maybe you have some refactoring plans or ideas for this file which should be considered in the current or next PRs?

I don't think we have any refactoring planned atm. Most TODOs are there to provide context or serve as a reminder, but none are critical to implement/fix.

SergeyVolodko

comment created time in 3 months

pull request commentapache/spark

[SPARK-22674][PYTHON] Removed the namedtuple pickling patch

It's a bit difficult to debug as well.

I know :) It's a shame the PR was not merged in time for 3.0.

superbobry

comment created time in 3 months

Pull request review commenttensorflow/tensorflow

Prefer Tensor._shape_tuple over Tensor._shape_as_list

 def __iter__(self):     return _TensorIterator(self, shape[0])    def _shape_as_list(self):-    if self.shape.ndims is not None:-      return [dim.value for dim in self.shape.dims]-    else:+    shape = self._shape_tuple()+    if shape is None:       return None+    return list(shape)    def _shape_tuple(self):-    shape = self._shape_as_list()-    if shape is None:+    try:+      return tuple(dim.value for dim in self.shape.dims)

I suspect that such optimization will be negligible in a realistic benchmark. A more noticeable optimization (maybe) is to make Python TensorShape a thin pybind11 wrapper around tensorflow::TensorShape.

This is slightly complicated by the fact that TensorShape behavior is configurable, but overall, it should be doable.

lgeiger

comment created time in 3 months

pull request commentapache/spark

[SPARK-22674][PYTHON] Removed the namedtuple pickling patch

I suspect it might be too late know that 3.X is out, but perhaps @HyukjinKwon could comment?

superbobry

comment created time in 3 months

more