profile
viewpoint

akki/py3spark-docker-boilerplate 6

Minimal installation for working with Python3 & Spark via Docker

akki/stream 1

To play songs somewhere far away and learn a bit of PHP :)

akki/2019.djangocon.eu 0

🚲 DjangoCon Europe 2019 website

akki/airflow 0

Apache Airflow

akki/arangodb 0

ArangoDB is a native multi-model database with flexible data models for documents, graphs, and key-values. Build high performance applications using a convenient SQL-like query language or JavaScript extensions.

akki/azure-storage-python 0

Microsoft Azure Storage Library for Python

akki/community 0

Kubernetes community content

akki/cpython 0

The Python programming language

push eventakki/Validation-Using-React

Akshesh Doshi

commit sha 61a0da4b5ef08394f3b67590ee9ab72c3ac9cde8

Link CodeSandbox in README

view details

push time in 4 days

issue commentiovisor/bcc

bcc.BPF() throws exception on CentOS 7.6 in Python

Ahh, I just found this:

root@centos-ebpf$ cat /etc/*release
CentOS Linux release 7.8.2003 (Core)
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

CentOS Linux release 7.8.2003 (Core)
CentOS Linux release 7.8.2003 (Core)

Seems like the centos:7.6.1810 Docker image is actually building CentOS 7.8 ? 🤔 And the installation guide mentions that it is only for 7.6.

Can this be the reason? 😐

akki

comment created time in 5 days

issue commentiovisor/bcc

bcc.BPF() throws exception on CentOS 7.6 in Python

Hi @ethercflow

Thanks for the advise.

I tried it out but I get

No package kernel-devel-4.19.76-linuxkit available.

Then I installed kernel-devel with yum install kernel-devel (note that I already had kernel-debug-devel installed) but I still get the same error from my Python code.

Can you think of anything that I might be doing wrong? (or if there is any other details about my environment which might help in investigating this issue, I would be happy to share). I am running all this in a Docker container built upon centos:7.6.1810 image on a Mac laptop.

akki

comment created time in 5 days

issue commentiovisor/bcc

bcc.BPF() throws exception on CentOS 7.6 in Python

Appreciate your time & help.

The steps mentioned in https://github.com/iovisor/bcc/blob/master/INSTALL.md#centos---source do install the kernel-debug-devel package which I think is what the "or installing the kernel development package for your running kernel version" part of the error is referring to.

I'll try to investigate further. Maybe someone else knows the fix to this problem so it might be helpful like to keep this issue open.

akki

comment created time in 6 days

issue commentiovisor/bcc

bcc.BPF() throws exception on CentOS 7.6 in Python

@willfindlay Thank you for the quick response. I did try installing them (but I am not 100% sure if I am doing the right thing).

This is what I tried and the output:

[root@centos-ebpf]$ yum install kernel-headers
.
.
.
<successful installation output>

But I saw that it is trying to look for the directory /lib/modules/4.19.76-linuxkit/build, so I also tried:

[root@centos-ebpf]$ yum install kernel-headers-$(uname -r)
Loaded plugins: fastestmirror, ovl
Loading mirror speeds from cached hostfile
 * base: mirrors.thzhost.com
 * centos-sclo-rh: mirrors.thzhost.com
 * centos-sclo-sclo: mirrors.thzhost.com
 * epel: my.mirrors.thegigabit.com
 * extras: mirrors.thzhost.com
 * updates: mirrors.thzhost.com
No package kernel-headers-4.19.76-linuxkit available.
Error: Nothing to do

I get the same error on running that Python script, after doing these steps.

akki

comment created time in 6 days

issue openediovisor/bcc

bcc.BPF() throws exception on CentOS 7.6 in Python

Hi,

I am receiving the following exception when using bcc.BPF() method in Python:

Unable to find kernel headers. Try rebuilding kernel with CONFIG_IKHEADERS=m (module) or installing the kernel development package for your running kernel version.
chdir(/lib/modules/4.19.76-linuxkit/build): No such file or directory
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "/usr/lib/python2.7/site-packages/bcc/__init__.py", line 359, in __init__
    raise Exception("Failed to compile BPF module %s" % (src_file or "<text>"))
Exception: Failed to compile BPF module <text>

These are the steps I followed to hit this error:

akshesh$ docker run -itd --name centos-ebpf centos:7.6.1810
akshesh$ docker exec -it centos-ebpf bash

root@centos-ebpf$ # From https://github.com/iovisor/bcc/blob/master/INSTALL.md#centos---source
root@centos-ebpf$ yum install -y epel-release
root@centos-ebpf$ yum update -y
root@centos-ebpf$ yum groupinstall -y "Development tools"
root@centos-ebpf$ yum install -y elfutils-libelf-devel cmake3 git bison flex ncurses-devel

root@centos-ebpf$ git clone https://github.com/iovisor/bcc.git
root@centos-ebpf$ mkdir bcc/build; cd bcc/build
root@centos-ebpf$ cmake3 ..
root@centos-ebpf$ make
root@centos-ebpf$ sudo make install

Then I ran the following code:

[root@681d46b746ef build]# python
Python 2.7.5 (default, Apr  2 2020, 13:16:51)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from bcc import BPF
>>> from threading import Thread
>>> from socket import socket
>>> import ctypes as ct
>>> import time
>>> from bcc import BPF
>>>
>>> prog = '''
... #include <linux/sched.h>
... /* This is a BCC macro for that creates a BPF data-structure that we can communicate with in user-mode */
... BPF_PERF_OUTPUT(events);
... // The following is the data-structure we'll pass to our user-land program
... struct data_t {
...     char process_name[TASK_COMM_LEN];    // Process name
...     u32 pid;                             // Process ID
...     int socket_fd;                       // Bound Socket FD
... };
... /* This is our BPF routine, it contains two arguments:
... - A pt_regs* struct, which contains the BPF VM registers
at gets the process name that invoked the bind operation
    bpf_get_current_comm(&data.process_name, sizeof(data.process_name));

    // Gets the pid via the bpf helper (pid is the upper 32 bits)
    data.pid = (u32) (bpf_get_current_pid_tgid() >> 32);
    data.socket_fd = sockfd;
    // Copies the data to the BPF structure, it is now available to user-mode
    events.perf_submit(ctx, &data, sizeof(data));
    return 0;
}
'''
# Compiles the BPF program via LLVM
b = BPF(text=prog)... - A socket fd - this will actually be transformed by bcc to a local variable that is set by the registers, see note below
... */
... int on_bind(struct pt_regs* ctx, int sockfd) {
...     struct data_t data = {};
...     // A bpf helper that gets the process name that invoked the bind operation
...     bpf_get_current_comm(&data.process_name, sizeof(data.process_name));
...
...     // Gets the pid via the bpf helper (pid is the upper 32 bits)
...     data.pid = (u32) (bpf_get_current_pid_tgid() >> 32);
...     data.socket_fd = sockfd;
...     // Copies the data to the BPF structure, it is now available to user-mode
...     events.perf_submit(ctx, &data, sizeof(data));
...     return 0;
... }
... '''
>>> # Compiles the BPF program via LLVM
... b = BPF(text=prog)
Unable to find kernel headers. Try rebuilding kernel with CONFIG_IKHEADERS=m (module) or installing the kernel development package for your running kernel version.
chdir(/lib/modules/4.19.76-linuxkit/build): No such file or directory
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "/usr/lib/python2.7/site-packages/bcc/__init__.py", line 359, in __init__
    raise Exception("Failed to compile BPF module %s" % (src_file or "<text>"))
Exception: Failed to compile BPF module <text>

Some details about my OS/environment:

root@centos-ebpf$ cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

root@centos-ebpf$ uname -a
Linux 681d46b746ef 4.19.76-linuxkit #1 SMP Tue May 26 11:42:35 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Can anyone please confirm that I have hit a bug or am I doing something wrong? I would appreciate if you can point me in the correct direction to fix this (either make a PR or fix my environment).

created time in 6 days

issue commentapache/airflow

Delay until Docker Swarm recognizes finished service

@CodingJonas Just curious if you asked upstream why docker-py behaves that way? It might be worth asking them to know if they can fix it.

CodingJonas

comment created time in 20 days

Pull request review commentapache/airflow

Fix DockerOperator xcom

 def _run_image(self):                 tty=self.tty,             ) -            lines = self.cli.attach(container=self.container['Id'],

So I decided to just keep as a list to make the code less complex. And do not repeat the request as the original code.

I do read this in the description. I am guessing the original code makes the repeated request because of the tiny memory footprint of this API.

nullhack

comment created time in 2 months

pull request commentapache/airflow

Fix DockerOperator xcom

@mik-laj Thanks for drawing my attention to this. @nullhack I have commented some questions/suggestions for these changes, hope they make sense to you and are useful.

nullhack

comment created time in 2 months

Pull request review commentapache/airflow

Fix DockerOperator xcom

 def _run_image(self):                 tty=self.tty,             ) -            lines = self.cli.attach(container=self.container['Id'],-                                    stdout=True,-                                    stderr=True,-                                    stream=True)+            self.cli.start(self.container["Id"]) -            self.cli.start(self.container['Id'])+            result = self.cli.wait(self.container["Id"])++            lines = [+                templ.decode("utf-8").strip() if hasattr(templ, "decode") else str(templ)+                for templ in self.cli.logs(+                    self.container["Id"], stream=True, stdout=True, stderr=False+                )+            ]              line = ''             for line in lines:-                line = line.strip()-                if hasattr(line, 'decode'):-                    line = line.decode('utf-8')                 self.log.info(line) -            result = self.cli.wait(self.container['Id'])-            if result['StatusCode'] != 0:-                raise AirflowException('docker container failed: ' + repr(result))+            warnings = [+                tempw.decode("utf-8").strip() if hasattr(tempw, "decode") else str(tempw)+                for tempw in self.cli.logs(+                    self.container["Id"], stream=True, stdout=False, stderr=True+                )+            ]++            for warning in warnings:+                self.log.warning(warning)++            if result["StatusCode"] != 0:+                raise AirflowException("docker container failed: " + repr(result)) -            # duplicated conditional logic because of expensive operation             ret = None             if self.do_xcom_push:-                ret = self.cli.logs(container=self.container['Id']) \-                    if self.xcom_all else line.encode('utf-8')+                ret = "\n".join(lines) if self.xcom_all else line +            self.cli.stop(self.container['Id'])

I have a couple of questions here:

  1. Why does Airflow explicitly need to stop the Docker container?
  2. If I understand correctly the .wait() API (which we are already calling above) ensures that the container is stopped by blocking until the container gets stopped so why would the container be running at this point of time (hence why the need for this call?).

If there is some reason that justifies this call, I would suggest adding some (warn?) logging here telling the user that the container was forcefully stopped by Airflow, so that it is easy for them to debug.

nullhack

comment created time in 2 months

Pull request review commentapache/airflow

Fix DockerOperator xcom

 def _run_image(self):                 tty=self.tty,             ) -            lines = self.cli.attach(container=self.container['Id'],-                                    stdout=True,-                                    stderr=True,-                                    stream=True)+            self.cli.start(self.container["Id"]) -            self.cli.start(self.container['Id'])+            result = self.cli.wait(self.container["Id"])

Please correct me if I am wrong but If we .wait() here, I understand that Airflow won't show up the application logs until the Docker container stops?

For long-running applications (which typically run for 1 or more hours), I think it is very crucial to see their logs in near-real-time but with this, I think we won't see any logs until the container process completes? Sorry if I am missing something here.

nullhack

comment created time in 2 months

Pull request review commentapache/airflow

Fix DockerOperator xcom

 def _run_image(self):                 tty=self.tty,             ) -            lines = self.cli.attach(container=self.container['Id'],-                                    stdout=True,-                                    stderr=True,-                                    stream=True)+            self.cli.start(self.container["Id"]) -            self.cli.start(self.container['Id'])+            result = self.cli.wait(self.container["Id"])++            lines = [+                templ.decode("utf-8").strip() if hasattr(templ, "decode") else str(templ)+                for templ in self.cli.logs(+                    self.container["Id"], stream=True, stdout=True, stderr=False+                )+            ]

So this will load the full logs of the application into memory, right? So if the application log file is 1 GB in size Airflow will start using 1GB extra RAM?

nullhack

comment created time in 2 months

Pull request review commentapache/airflow

Fix DockerOperator xcom

 def _run_image(self):                 tty=self.tty,             ) -            lines = self.cli.attach(container=self.container['Id'],

I think the self.cli.attach method with stream=True returns a generator which allows you to read the logs in a "streaming" fashion without using significant memory.

nullhack

comment created time in 2 months

PR closed apache/helix

At-most-one-online-partition system with substantial draining time (to go offline)

http://mail-archives.apache.org/mod_mbox/helix-user/202006.mbox/browser

+96 -46

2 comments

2 changed files

akki

pr closed time in 2 months

pull request commentapache/helix

At-most-one-online-partition system with substantial draining time (to go offline)

Hi @jiajunwang

Thank you for bringing my attention to this. I actually wanted to make a PR to my own fork akki/helix to showcase this code to someone else but I think GitHub (because of default) created the PR to apache/helix.

I'm closing this as it was created by mistake and I apologise for this confusion.

akki

comment created time in 2 months

issue closedapache/helix

Issue with sending email regarding a problem with setting up a state machine with Helix

Hi

I am facing some issues while trying to achieve a desired state machine using Helix in my application. I have tried to post it to the Helix Users mailing list (by sending an email to user@helix.apache.org but it is not showing up at http://mail-archives.apache.org/mod_mbox/helix-user/. Is there something I am doing wrong?

Or would you suggest me to put my question here (on GitHub)?

closed time in 2 months

akki

issue commentapache/helix

Issue with sending email regarding a problem with setting up a state machine with Helix

The conversation has continued here - http://mail-archives.apache.org/mod_mbox/helix-user/202006.mbox/browser

akki

comment created time in 2 months

PR opened apache/helix

At-most-one-online-partition system with substantial draining time (to go offline)

http://mail-archives.apache.org/mod_mbox/helix-user/202006.mbox/browser

+96 -46

0 comment

2 changed files

pr created time in 2 months

create barnchakki/helix

branch : quickstart-poc

created branch time in 2 months

fork akki/helix

Mirror of Apache Helix

fork in 2 months

issue commentapache/helix

Issue with sending email regarding a problem with setting up a state machine with Helix

Hi, @pkuwm Thanks for your reply. I actually tried subscribing to the mailing list & sending my mail and this time it worked.

Copy-pasting it here as well, in case this is the preferred medium:

Hi Helix community

Nice to e-meet you guys. I am pretty new to this project and it is my first time writing to this mailing list - I apologize in advance for any mistakes.

I am trying to implement a system's state model requirement here but am not able to achieve it. Hoping anyone here could point me in the right direction.

GOAL

My system is a typical multi-node + multi-resource system with the following properties:

  1. Any partition should have one & only one online partition at any given point of time.
  2. The ONLINE -> OFFLINE transition is not instantaneous (typically takes minutes).
  3. Offline partitions have no special role - they can be dropped as soon as they become offline.

If it helps in understanding better, my application is a tool which copies data from Kafka to Hadoop. And having two ONLINE partitions at the same time means I am duplicating this data in Hadoop.

WHAT I HAVE TRIED

I was able to successfully modify the Quickstart script to imitate my use-case so I believe Helix can handle this scenario. But when I do it in my application I see that Helix fires the ONLINE -> OFFLINE & OFFLINE -> ONLINE transitions (to the corresponding 2 nodes) almost simultaneously. I want Helix to signal "ONLINE -> OFFLINE", then wait until the partition goes offline and only then fire the "OFFLINE -> ONLINE" transition to the new upcoming node. I have implemented my @Transition(from = "ONLINE", to = "OFFLINE") function in such a way that it waits for the partition to go offline (using latch.await()) and only then returns (I have confirmed this from application logs).

My application is different from my Quickstart trial app in the following ways (or at least, these are the ones known to me, I am building upon someone else's project so there might be code that I am not aware of):

  1. The rebalancing algo is not AUTO - I am using my own custom logic to distribute partitions among nodes
  2. I have enabled nodes to auto-join i.e. props.put(ZKHelixManager.ALLOW_PARTICIPANT_AUTO_JOIN, String.valueOf(true)); Is it possible for me to achieve this system with these settings enabled?

DEBUG LOGS / CODE

If it helps, this is what I see in Zookeeper after adding a 2nd node to my cluster which had 1 node with 1 resource with 6 partitions - https://gist.github.com/akki/1d80c97463198275b3abe39350688bda#file-zookeeper-output-txt As you can see, there are a few partitions which have 2 ONLINE replicas at the same time (after a while the draining replica goes away but in that duration, my data gets duplicated, which is the problem I want to overcome). I cannot understand how this is possible when I have set up these bounds in my model definition.

I would really appreciate if anyone here could give me any clues that what I might be doing wrong (or what I am trying to achieve is even possible or not with Helix).

Thank you so much for building such a wonderful tool and having this mailing list to help us out.

Regards Akshesh Doshi

akki

comment created time in 2 months

issue openedapache/helix

Issue with sending email regarding a problem with setting up a state machine with Helix

Hi

I am facing some issues while trying to achieve a desired state machine using Helix in my application. I have tried to post it to the Helix Users mailing list (by sending an email to user@helix.apache.org but it is not showing up at http://mail-archives.apache.org/mod_mbox/helix-user/. Is there something I am doing wrong?

Or would you suggest me to put my question here (on GitHub)?

created time in 2 months

more