profile
viewpoint
Shahin Azad shahinism Shahinism Amsterdam, Netherlands https://shahinism.com Data engineer at @DataChefHQ by day; Avid code writer by night.

AppforallHQ/appbuy 3

Send Gifts from Appstore

shahinism/linuxjourney 2

Courses and lesson content for linuxjourney.com

AppforallHQ/appleid_frontent 1

Registration Interface For Appleid Generator

shahinism/archer 1

Arch Linux Provisioning with Ansible

AppforallHQ/a3mess 0

SMS queue in node that interacts with magfa messaging webservice.

AppforallHQ/companion 0

Appforall Companion Project to help users Backup and Update their Applications

AppforallHQ/downloader 0

An automatic download manger with ability to download from shared file hostings

AppforallHQ/earlypage 0

An earlypage application in node

push eventshahinism/emacs.d

Shahin

commit sha 6050de695c7ea6768d66438a49c0dafed4ee062b

refactor(*): move package configuration to org file.

view details

push time in 12 days

startedabo-abo/hydra

started time in 12 days

startedembeddedkafka/embedded-kafka-schema-registry

started time in 12 days

delete branch shahinism/emacs.d

delete branch : refactor/old-modular-config

delete time in 13 days

delete branch shahinism/emacs.d

delete branch : refactor/cleanup

delete time in 13 days

create barnchshahinism/emacs.d

branch : development

created branch time in 13 days

startedkunalb/poet

started time in 15 days

startedseagle0128/.emacs.d

started time in 15 days

starteddelta-io/delta

started time in 17 days

startedapache/spark

started time in 17 days

startedbigH/git-fuzzy

started time in 25 days

startedLisaDziuba/Marketing-for-Engineers

started time in a month

create barnchshahinism/onioff

branch : refactor/main

created branch time in a month

delete branch shahinism/onioff

delete branch : refactor/main

delete time in a month

PR closed k4m4/onioff

Reviewers
Support Python 3 and some additional future/refactors enhancement

Hi, In these set of commits, I refactored the codebase and added some features to it. Currently implemented features are:

  • [x] Python 3 support (And compatible with Python 2.7)
  • [x] Bootstrap a Tor instance directly and use it instead of system-wide version (Enables us to add features on customizing Tor itself).
  • [x] Timeout support
  • [x] Packaging (No requirements.txt and installable through pip)
  • [x] Install onioff script in the user's PATH
  • [x] Formatted output instead of text files (JSON, TSV)
  • [x] Progress bar
  • [x] Specific messages for other inactive states (Timeouts, Inaccessible server, SSL Error)

Yet still I plan to develop the following as soon as I have some free time:

  • [ ] Unit tests
  • [ ] Tox test runner to confirm complete support on all supported Python versions
  • [ ] Make SSL verification optional
  • [ ] Custom Tor configuration
  • [ ] Update Readme
+352 -289

9 comments

9 changed files

shahinism

pr closed time in a month

PR opened poteto/hiring-without-whiteboards

Add DataChef.

<!-- Thank you for contributing!

Pull requests that do not adhere to the format will be rejected. Please ensure you complete the following checkboxes.

Please also:

  • Add one company at a time.
  • Insert in alphabetical order
  • Do not sort other listings -->

Add/Update/Remove <CompanyName>

  • [x] I have read the contributing guidelines
  • [x] I agree to the Code of Conduct
  • [x] I have followed the format prescribed in the contributing guidelines
  • [ ] (OPTIONAL) In your pull request message, add additional context on the interview process if necessary

<!-- Please give additional context about the interview process if necessary. -->

+1 -0

0 comment

1 changed file

pr created time in a month

push eventDataChefHQ/hiring-without-whiteboards

Shahin

commit sha 60c77a6a4229c77b641d2727d2b63d4127a13b02

Add DataChef.

view details

push time in a month

issue commentgreat-expectations/great_expectations

NameError: name 'WithinGroup' is not defined

@shahinism Thank you for reporting this! Which database were you connecting to?

I use it with Spark, however as described above, I just invoked --version command and it failed.

shahinism

comment created time in 2 months

issue openedgreat-expectations/great_expectations

NameError: name 'WithinGroup' is not defined

Describe the bug After installing the new released version from Pypi (version 0.10.11), invoking great expectation will raise following exception:

$ great_expectations --version
Traceback (most recent call last):
  File "/home/hadoop/.local/bin/great_expectations", line 7, in <module>
    from great_expectations.cli import main
  File "/home/hadoop/.local/lib/python3.6/site-packages/great_expectations/__init__.py", line 7, in <module>
    from great_expectations.data_context import DataContext
  File "/home/hadoop/.local/lib/python3.6/site-packages/great_expectations/data_context/__init__.py", line 3, in <module>
    from .data_context import BaseDataContext, DataContext, ExplorerDataContext
  File "/home/hadoop/.local/lib/python3.6/site-packages/great_expectations/data_context/data_context.py", line 20, in <module>
    from great_expectations.core.usage_statistics.usage_statistics import (
  File "/home/hadoop/.local/lib/python3.6/site-packages/great_expectations/core/usage_statistics/usage_statistics.py", line 20, in <module>
    from great_expectations.core.usage_statistics.anonymizers.data_docs_site_anonymizer import (
  File "/home/hadoop/.local/lib/python3.6/site-packages/great_expectations/core/usage_statistics/anonymizers/data_docs_site_anonymizer.py", line 2, in <module>
    from great_expectations.core.usage_statistics.anonymizers.site_builder_anonymizer import (
  File "/home/hadoop/.local/lib/python3.6/site-packages/great_expectations/core/usage_statistics/anonymizers/site_builder_anonymizer.py", line 2, in <module>
    from great_expectations.render.renderer.site_builder import (
  File "/home/hadoop/.local/lib/python3.6/site-packages/great_expectations/render/renderer/__init__.py", line 7, in <module>
    from .other_section_renderer import ProfilingResultsOverviewSectionRenderer
  File "/home/hadoop/.local/lib/python3.6/site-packages/great_expectations/render/renderer/other_section_renderer.py", line 4, in <module>
    from great_expectations.profile.basic_dataset_profiler import BasicDatasetProfiler
  File "/home/hadoop/.local/lib/python3.6/site-packages/great_expectations/profile/__init__.py", line 1, in <module>
    from .basic_dataset_profiler import BasicDatasetProfiler
  File "/home/hadoop/.local/lib/python3.6/site-packages/great_expectations/profile/basic_dataset_profiler.py", line 3, in <module>
    from great_expectations.profile.base import (
  File "/home/hadoop/.local/lib/python3.6/site-packages/great_expectations/profile/base.py", line 8, in <module>
    from ..dataset import Dataset
  File "/home/hadoop/.local/lib/python3.6/site-packages/great_expectations/dataset/__init__.py", line 3, in <module>
    from .dataset import Dataset
  File "/home/hadoop/.local/lib/python3.6/site-packages/great_expectations/dataset/dataset.py", line 13, in <module>
    from great_expectations.dataset.util import (
  File "/home/hadoop/.local/lib/python3.6/site-packages/great_expectations/dataset/util.py", line 578, in <module>
    selects: List[WithinGroup], sql_engine_dialect: DefaultDialect
NameError: name 'WithinGroup' is not define

To Reproduce Steps to reproduce the behaviour:

  1. pip install great-expectations
  2. great_expectations --version

Expected behaviour To not raise the exception.

Environment (please complete the following information):

  • OS: Linux
  • GE Version: Ubuntu 19.10

created time in 2 months

startedghosh/microtip

started time in 2 months

startedelixir-lsp/elixir-ls

started time in 2 months

startedJohnSnowLabs/spark-nlp

started time in 2 months

delete branch shahinism/great_expectations

delete branch : bug/unhashable_obj

delete time in 2 months

push eventshahinism/docker-stealth-openvpn

Shahin

commit sha 8e2a03fb53231634538174332b67528600fc3c31

Set proper IP address for openvpn client to run behind stunnel.

view details

push time in 2 months

startedrootkovska/tweetbrief

started time in 2 months

push eventshahinism/docker-stunnel

fossabot

commit sha 3278a54b4a8203d35361944a6d43e833319c4adb

Add license scan report and status Signed-off-by: fossabot <badges@fossa.io>

view details

Shahin Azad

commit sha 8b36a8772f33a183f17cd84f316521c35d368a12

Merge branch 'master' into master

view details

Shahin Azad

commit sha 6c1365fff2f1183adccd0b888a4d0a5373da0a40

Merge pull request #1 from fossabot/master Add license scan report and status

view details

push time in 2 months

PR merged shahinism/docker-stunnel

Add license scan report and status

Your FOSSA integration was successful! Attached in this PR is a badge and license report to track scan status in your README.

Below are docs for integrating FOSSA license checks into your CI:

+3 -0

1 comment

1 changed file

fossabot

pr closed time in 2 months

push eventfossabot/docker-stunnel

Shahin

commit sha adb8ecebe4386c2dbae2dc410258c1ff0d832e53

Fix license policy

view details

Shahin

commit sha 3f6cd5e9f04ee4ddc94dbc3cf4c136cb31f39111

Add license badge.

view details

Shahin Azad

commit sha 8b36a8772f33a183f17cd84f316521c35d368a12

Merge branch 'master' into master

view details

push time in 2 months

push eventshahinism/great_expectations

Shahin

commit sha 838c5106bfdf27e28d97c7d20f518c3fd07b7f91

Fix hash_dataframe function name ambiguity. This function will only work with pandas dataframes.

view details

push time in 2 months

push eventshahinism/great_expectations

Shahin

commit sha 91628162a9e7c0d04a27edece7e5fd8103a4bd3a

Fix tests.

view details

push time in 2 months

PR opened great-expectations/great_expectations

Use pickle to generate hash for dataframes with unhashable objects.

These changes are regarding to fix #1309 issue. According to pandas-dev/pandas#18771 it's not possible to generate hash when the data frame contains a mutable object, so as a fallback solution I used pickle for such data frames.

+28 -4

0 comment

3 changed files

pr created time in 2 months

create barnchshahinism/great_expectations

branch : bug/unhashable_obj

created branch time in 2 months

push eventshahinism/docker-stunnel

Shahin

commit sha 3f6cd5e9f04ee4ddc94dbc3cf4c136cb31f39111

Add license badge.

view details

push time in 2 months

push eventshahinism/docker-stealth-openvpn

Shahin

commit sha 7c0b05ff13c0b679ace757528e7ae443e2e429e3

Push verbosity.

view details

Shahin

commit sha 7a434fd67da1f235cc43280b6a1db77025b68960

Add todos.

view details

push time in 2 months

push eventshahinism/docker-stealth-openvpn

Shahin

commit sha ce77d3261bd14c12dddf66ea35ff996669d623a9

Fix config path.

view details

push time in 2 months

push eventshahinism/docker-stealth-openvpn

Shahin

commit sha 0f9f99b283c7eedd54b977ac8999c1528befc38d

Make it possible to just download a client.

view details

Shahin

commit sha baf2e3c1c18915778f6fe6e97dd29c0f30737dd0

Fix config path.

view details

push time in 2 months

push eventshahinism/docker-stealth-openvpn

Shahin

commit sha e4e892cc98ce186bb60473ff47740a86b72d6427

Push route config to clients.

view details

Shahin

commit sha e7c3df0b54d53d717f1efe5695b0cc2a40a50c5b

Fix license policy.

view details

push time in 2 months

push eventshahinism/docker-stunnel

Shahin

commit sha adb8ecebe4386c2dbae2dc410258c1ff0d832e53

Fix license policy

view details

push time in 2 months

push eventshahinism/docker-stunnel

Shahin

commit sha b8ad7c107fc317c083dc637e3fba293945457673

Fix license policy

view details

push time in 2 months

startedtiangolo/typer

started time in 2 months

startedfastai/nbdev

started time in 2 months

startedmahmoud/glom

started time in 3 months

issue commentgreat-expectations/great_expectations

unhashable type: 'dict' on files with nested map/object

As a hotfix test, I just commented out that if condition and the suite creation was successful :thinking:. However I'm not sure if that would cause issues in other places.

shahinism

comment created time in 3 months

issue openedgreat-expectations/great_expectations

unhashable type: 'dict' on files with nested map/object

Problem

If you try to load an object with nested dict (JSON object, or map/Struct in Parquet), on Pandas computation engine, will raise this error:

Traceback (most recent call last):
  File "/home/shahin/Work/Experiment/ge_app/.direnv/python-3.7.2/bin/great_expectations", line 8, in <module>
    sys.exit(main())
  File "/home/shahin/Work/Experiment/ge_app/.direnv/python-3.7.2/lib/python3.7/site-packages/great_expectations/cli/cli.py", line 66, in main
    cli()
  File "/home/shahin/Work/Experiment/ge_app/.direnv/python-3.7.2/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/shahin/Work/Experiment/ge_app/.direnv/python-3.7.2/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/shahin/Work/Experiment/ge_app/.direnv/python-3.7.2/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/shahin/Work/Experiment/ge_app/.direnv/python-3.7.2/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/shahin/Work/Experiment/ge_app/.direnv/python-3.7.2/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/shahin/Work/Experiment/ge_app/.direnv/python-3.7.2/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/shahin/Work/Experiment/ge_app/.direnv/python-3.7.2/lib/python3.7/site-packages/great_expectations/cli/suite.py", line 318, in suite_new
    raise e
  File "/home/shahin/Work/Experiment/ge_app/.direnv/python-3.7.2/lib/python3.7/site-packages/great_expectations/cli/suite.py", line 275, in suite_new
    open_docs=view,
  File "/home/shahin/Work/Experiment/ge_app/.direnv/python-3.7.2/lib/python3.7/site-packages/great_expectations/cli/datasource.py", line 906, in create_expectation_suite
    additional_batch_kwargs=additional_batch_kwargs)
  File "/home/shahin/Work/Experiment/ge_app/.direnv/python-3.7.2/lib/python3.7/site-packages/great_expectations/cli/datasource.py", line 833, in get_batch_kwargs
    batch_kwargs_generator_name=batch_kwargs_generator_name,
  File "/home/shahin/Work/Experiment/ge_app/.direnv/python-3.7.2/lib/python3.7/site-packages/great_expectations/cli/datasource.py", line 1107, in _get_batch_kwargs_from_generator_or_from_file_path
    batch = datasource.get_batch(batch_kwargs=batch_kwargs)
  File "/home/shahin/Work/Experiment/ge_app/.direnv/python-3.7.2/lib/python3.7/site-packages/great_expectations/datasource/pandas_datasource.py", line 205, in get_batch
    df, index=True).values).hexdigest()
  File "/home/shahin/Work/Experiment/ge_app/.direnv/python-3.7.2/lib/python3.7/site-packages/pandas/core/util/hashing.py", line 137, in hash_pandas_object
    h = _combine_hash_arrays(hashes, num_items)
  File "/home/shahin/Work/Experiment/ge_app/.direnv/python-3.7.2/lib/python3.7/site-packages/pandas/core/util/hashing.py", line 40, in _combine_hash_arrays
    first = next(arrays)
  File "/home/shahin/Work/Experiment/ge_app/.direnv/python-3.7.2/lib/python3.7/site-packages/pandas/core/util/hashing.py", line 136, in <genexpr>
    hashes = (x for x in _hashes)
  File "/home/shahin/Work/Experiment/ge_app/.direnv/python-3.7.2/lib/python3.7/site-packages/pandas/core/util/hashing.py", line 119, in <genexpr>
    hashes = (hash_array(series.values) for _, series in obj.items())
  File "/home/shahin/Work/Experiment/ge_app/.direnv/python-3.7.2/lib/python3.7/site-packages/pandas/core/util/hashing.py", line 306, in hash_array
    codes, categories = factorize(vals, sort=False)
  File "/home/shahin/Work/Experiment/ge_app/.direnv/python-3.7.2/lib/python3.7/site-packages/pandas/core/algorithms.py", line 636, in factorize
    values, na_sentinel=na_sentinel, size_hint=size_hint, na_value=na_value
  File "/home/shahin/Work/Experiment/ge_app/.direnv/python-3.7.2/lib/python3.7/site-packages/pandas/core/algorithms.py", line 484, in _factorize_array
    uniques, codes = table.factorize(values, na_sentinel=na_sentinel, na_value=na_value)
  File "pandas/_libs/hashtable_class_helper.pxi", line 1815, in pandas._libs.hashtable.PyObjectHashTable.factorize
  File "pandas/_libs/hashtable_class_helper.pxi", line 1731, in pandas._libs.hashtable.PyObjectHashTable._unique
TypeError: unhashable type: 'dict'

Of course, if you use a non-compressed file, it'll show a more polite version of the same error:

Cannot load file.
  - Please check the file and try again or select a different data file.
  - Error: unhashable type: 'dict'

How to reproduce

Create a JSON file with the following content and try to create a suite out of it:

[
    {"key": {"type": 2}}
]

Resolution

As it's visible in the exception error, it's happening on this call to pandas.util.hash_pandas_object. And that happens when the data frame size is below a fixed threshold.

created time in 3 months

startedjgraph/drawio

started time in 3 months

push eventshahinism/docker-stealth-openvpn

Shahin

commit sha 6bed8a48f5be3bccbef7c048bb60534ae1113b10

Extract colored message printers to common library.

view details

Shahin

commit sha 9e6c72b66d6f3d25edac08d75fdce15679c22c97

Use info colors.

view details

Shahin

commit sha 579343438732721950e3a7abef750943dd2e4f3a

Ignore stunnel.conf

view details

push time in 3 months

startedgoogle-research/text-to-text-transfer-transformer

started time in 3 months

push eventshahinism/docker-stealth-openvpn

Shahin

commit sha c9665fee4af6e0c8c32b4a5fe4949aa34e9ec48f

Generate stunnel config on initialization.

view details

push time in 3 months

push eventshahinism/docker-stealth-openvpn

Shahin

commit sha 8827202428da17c1f9308c496ae185d7969fb93a

Add start script to make file.

view details

push time in 3 months

push eventshahinism/docker-stealth-openvpn

Shahin

commit sha db3ecb7d95787ffd9c5e0e1b044d744f16b53d56

Set -e on bash scripts.

view details

push time in 3 months

push eventshahinism/docker-stealth-openvpn

Shahin

commit sha 47a77aadb3825a54a4365e3be81bd288d9e9e878

Fix client directory builder.

view details

push time in 3 months

push eventshahinism/docker-stealth-openvpn

Shahin

commit sha 96b4043683c350b1f69c0b4dd9a937e4f3fae92c

make scripts executable

view details

push time in 3 months

push eventshahinism/docker-stealth-openvpn

Shahin

commit sha 0b2757d6fcddeb24e38395186d6292a7a87728bd

Add gitignores.

view details

push time in 3 months

push eventshahinism/docker-stealth-openvpn

Shahin

commit sha 5decda16f56526481b69eeea29118b8513a9b2bb

Add main logic.

view details

push time in 3 months

create barnchshahinism/docker-stealth-openvpn

branch : master

created branch time in 3 months

created repositoryshahinism/docker-stealth-openvpn

created time in 3 months

push eventshahinism/docker-stunnel

Shahin

commit sha 1abdf514b049290964d006752a432f3cd3ef961b

Fix license link.

view details

push time in 3 months

push eventshahinism/docker-stunnel

Shahin

commit sha d3d157c07388eebcbb0df02fe29b99497fe44665

Break if `/etc/stunnel` doesn't exist.

view details

push time in 3 months

push eventshahinism/docker-stunnel

Shahin

commit sha fc84fc95e63d33c64d04c723139197994f16b503

Add build badges.

view details

Shahin

commit sha a6bb666684899226bf2c6ad5223a2265fc102aa8

Update Readme.

view details

push time in 3 months

push eventshahinism/docker-stunnel

Shahin

commit sha 00e81291c003aba4f7b253ae71ba2565fd73a437

Initialize the project.

view details

push time in 3 months

create barnchshahinism/docker-stunnel

branch : master

created branch time in 3 months

created repositoryshahinism/docker-stunnel

Stunnel in a docker container.

created time in 3 months

issue commentkylemanna/docker-openvpn

Client connected but no internet

I was able to resolve this issue by adding the following line to the client config:

route SERVER_PUBLIC_IP 255.255.255.255 net_gateway

Probably this should get pushed by the server.

Madh93

comment created time in 3 months

startedmeilisearch/MeiliSearch

started time in 3 months

startedmbrt/gmailctl

started time in 3 months

more