profile
viewpoint
Greg Poirier grepory @sensu San Francisco, CA

grepory/birdin 1

Twitter bot for me

dan-compton/assignment-exchange 0

online tutoring service for computer science

grepory/amazon-ecs-cli 0

A custom Amazon ECS CLI that eases up the cluster setup process, enables users to run their applications locally or on ECS using the same Docker Compose file format and familiar Compose commands.

grepory/aptly 0

aptly - Debian repository management tool

grepory/aws-sdk-go 0

AWS SDK for the Go programming language.

grepory/aws-tools 0

Random AWS scripts

grepory/awscan 0

AWS resource scanner.

grepory/awscli 0

Python awscli wrapper container

pull request commentStackStorm-Exchange/stackstorm-aws

Add SQSServiceSensor for non-polling SQS sensor

My desire to work on this is non-existent at this point. Our internal fork is working fine for us, and I’ve moved on to another project.

On Fri, Feb 7, 2020 at 7:02 PM JP Bourget notifications@github.com wrote:

@grepory https://github.com/grepory Hey - I want to get this merged, so I am reviewing this PR and I have a few questions.

  1. What happens if someone adds a large amount of queues? Say 30? 100? 500?
  2. Any system requirements recommendations? I can see this sensor getting people into trouble without care consideration before turning it on.
  3. What if you named the sensor SQSContinuousSensor or something like that to better differentiate?

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/StackStorm-Exchange/stackstorm-aws/pull/91?email_source=notifications&email_token=AACXAXXSDKJ2HEX6SVDJQ3DRBYOCVA5CNFSM4J4ULUSKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELFH4GY#issuecomment-583695899, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACXAXRNGMVDTC23NVDQ5BLRBYOCVANCNFSM4J4ULUSA .

-- greg poirier Old dancers never die, they just leap from barre to barre.

grepory

comment created time in 15 days

Pull request review commentStackStorm-Exchange/stackstorm-aws

Add SQSServiceSensor for non-polling SQS sensor

+"""+This is generic SQS Sensor using boto3 api to fetch messages from sqs queue.+After receiving a message it's content is passed as payload to a trigger 'aws.sqs_new_message'+This sensor can be configured either by using config.yaml within a pack or by creating+following values in datastore:+    - aws.input_queues (list queues as comma separated string: first_queue,second_queue)+    - aws.aws_access_key_id+    - aws.aws_secret_access_key+    - aws.region+    - aws.max_number_of_messages (must be between 1 - 10)+For configuration in config.yaml with config like this+    setup:+      aws_access_key_id:+      aws_access_key_id:+      region:+    sqs_sensor:+      input_queues:+        - first_queue+        - second_queue+    sqs_other:+        max_number_of_messages: 1+If any value exist in datastore it will be taken instead of any value in config.yaml+"""++import six+import json+from boto3.session import Session+from botocore.exceptions import ClientError+from botocore.exceptions import NoRegionError+from botocore.exceptions import NoCredentialsError+from botocore.exceptions import EndpointConnectionError++from st2reactor.sensor.base import Sensor+++class AWSSQSServiceSensor(Sensor):+    def __init__(self, sensor_service, config=None):+        super(AWSSQSServiceSensor, self).__init__(sensor_service=sensor_service, config=config)++    def setup(self):+        self._logger = self._sensor_service.get_logger(name=self.__class__.__name__)++        self.session = None+        self.sqs_res = None++    def run(self):+        # setting SQS ServiceResource object from the parameter of datastore or configuration file+        self._may_setup_sqs()++        while True:+            for queue in self.input_queues:+                msgs = self._receive_messages(queue=self._get_queue_by_name(queue),+                                            num_messages=self.max_number_of_messages)+                for msg in msgs:+                    if msg:+                        payload = {"queue": queue, "body": json.loads(msg.body)}+                        self._sensor_service.dispatch(trigger="aws.sqs_new_message",+                                                      payload=payload)+                        msg.delete()++    def cleanup(self):+        pass++    def add_trigger(self, trigger):+        # This method is called when trigger is created+        pass++    def update_trigger(self, trigger):+        # This method is called when trigger is updated+        pass++    def remove_trigger(self, trigger):+        pass++    def _get_config_entry(self, key, prefix=None):+        ''' Get configuration values either from Datastore or config file. '''+        config = self.config+        if prefix:+            config = self._config.get(prefix, {})++        value = self._sensor_service.get_value('aws.%s' % (key), local=False)+        if not value:+            value = config.get(key, None)++        if not value and config.get('setup', None):+            value = config['setup'].get(key, None)++        return value++    def _may_setup_sqs(self):+        queues = self._get_config_entry(key='input_queues', prefix='sqs_sensor')++        # XXX: This is a hack as from datastore we can only receive a string while+        # from config.yaml we can receive a list+        if isinstance(queues, six.string_types):+            self.input_queues = [x.strip() for x in queues.split(',')]+        elif isinstance(queues, list):+            self.input_queues = queues+        else:+            self.input_queues = []++        self.aws_access_key = self._get_config_entry('aws_access_key_id')+        self.aws_secret_key = self._get_config_entry('aws_secret_access_key')+        self.aws_region = self._get_config_entry('region')++        self.max_number_of_messages = self._get_config_entry('max_number_of_messages',+                                                             prefix='sqs_other')++        # checker configuration is update, or not+        def _is_same_credentials():+            c = self.session.get_credentials()+            return c is not None and \+                c.access_key == self.aws_access_key and \+                c.secret_key == self.aws_secret_key and \+                self.session.region_name == self.aws_region++        if self.session is None or not _is_same_credentials():+            self._setup_sqs()++    def _setup_sqs(self):+        ''' Setup Boto3 structures '''+        self._logger.debug('Setting up SQS resources')+        self.session = Session(aws_access_key_id=self.aws_access_key,+                               aws_secret_access_key=self.aws_secret_key,+                               region_name=self.aws_region)++        try:+            self.sqs_res = self.session.resource('sqs')+        except NoRegionError:+            self._logger.warning("The specified region '%s' is invalid", self.aws_region)++    def _get_queue_by_name(self, queueName):+        ''' Fetch QUEUE by it's name create new one if queue doesn't exist '''+        try:+            return self.sqs_res.get_queue_by_name(QueueName=queueName)+        except ClientError as e:+            if e.response['Error']['Code'] == 'AWS.SimpleQueueService.NonExistentQueue':+                self._logger.warning("SQS Queue: %s doesn't exist, creating it.", queueName)+                return self.sqs_res.create_queue(QueueName=queueName)+            elif e.response['Error']['Code'] == 'InvalidClientTokenId':+                self._logger.warning("Cloudn't operate sqs because of invalid credential config")

I don't know why this was logged and not raised. It shouldn't be possible to get here if you have invalid credentials. We tested that scenario, and it raises an exception. I can't recall if it did so when getting the resource or creating the session.

grepory

comment created time in 2 months

Pull request review commentStackStorm-Exchange/stackstorm-aws

Add SQSServiceSensor for non-polling SQS sensor

+"""+This is generic SQS Sensor using boto3 api to fetch messages from sqs queue.+After receiving a message it's content is passed as payload to a trigger 'aws.sqs_new_message'+This sensor can be configured either by using config.yaml within a pack or by creating+following values in datastore:+    - aws.input_queues (list queues as comma separated string: first_queue,second_queue)+    - aws.aws_access_key_id+    - aws.aws_secret_access_key+    - aws.region+    - aws.max_number_of_messages (must be between 1 - 10)+For configuration in config.yaml with config like this+    setup:+      aws_access_key_id:+      aws_access_key_id:+      region:+    sqs_sensor:+      input_queues:+        - first_queue+        - second_queue+    sqs_other:+        max_number_of_messages: 1+If any value exist in datastore it will be taken instead of any value in config.yaml+"""++import six+import json+from boto3.session import Session+from botocore.exceptions import ClientError+from botocore.exceptions import NoRegionError+from botocore.exceptions import NoCredentialsError+from botocore.exceptions import EndpointConnectionError++from st2reactor.sensor.base import Sensor+++class AWSSQSServiceSensor(Sensor):+    def __init__(self, sensor_service, config=None):+        super(AWSSQSServiceSensor, self).__init__(sensor_service=sensor_service, config=config)++    def setup(self):+        self._logger = self._sensor_service.get_logger(name=self.__class__.__name__)++        self.session = None+        self.sqs_res = None++    def run(self):+        # setting SQS ServiceResource object from the parameter of datastore or configuration file+        self._may_setup_sqs()++        while True:+            for queue in self.input_queues:+                msgs = self._receive_messages(queue=self._get_queue_by_name(queue),+                                            num_messages=self.max_number_of_messages)+                for msg in msgs:+                    if msg:+                        payload = {"queue": queue, "body": json.loads(msg.body)}+                        self._sensor_service.dispatch(trigger="aws.sqs_new_message",+                                                      payload=payload)+                        msg.delete()

If this throws an exception, it's not because SQS couldn't delete the message due to some unrecoverable error. It will be because of a misconfiguration on the client side or something like that. I don't think it is necessary to try/catch this.

grepory

comment created time in 2 months

Pull request review commentStackStorm-Exchange/stackstorm-aws

Add SQSServiceSensor for non-polling SQS sensor

+"""+This is generic SQS Sensor using boto3 api to fetch messages from sqs queue.+After receiving a message it's content is passed as payload to a trigger 'aws.sqs_new_message'+This sensor can be configured either by using config.yaml within a pack or by creating+following values in datastore:+    - aws.input_queues (list queues as comma separated string: first_queue,second_queue)+    - aws.aws_access_key_id+    - aws.aws_secret_access_key+    - aws.region+    - aws.max_number_of_messages (must be between 1 - 10)+For configuration in config.yaml with config like this+    setup:+      aws_access_key_id:+      aws_access_key_id:+      region:+    sqs_sensor:+      input_queues:+        - first_queue+        - second_queue+    sqs_other:+        max_number_of_messages: 1+If any value exist in datastore it will be taken instead of any value in config.yaml+"""++import six+import json+from boto3.session import Session+from botocore.exceptions import ClientError+from botocore.exceptions import NoRegionError+from botocore.exceptions import NoCredentialsError+from botocore.exceptions import EndpointConnectionError++from st2reactor.sensor.base import Sensor+++class AWSSQSServiceSensor(Sensor):+    def __init__(self, sensor_service, config=None):+        super(AWSSQSServiceSensor, self).__init__(sensor_service=sensor_service, config=config)++    def setup(self):+        self._logger = self._sensor_service.get_logger(name=self.__class__.__name__)++        self.session = None+        self.sqs_res = None++    def run(self):+        # setting SQS ServiceResource object from the parameter of datastore or configuration file+        self._may_setup_sqs()++        while True:

boto3 receive_messages accepts a WaitTimeSeconds argument, which _receive_messages defaults to 2 seconds. That's what's keeping the loop from spinning too fast.

grepory

comment created time in 2 months

push eventgrepory/stackstorm-aws

Greg Poirier

commit sha 50d0d3197e0d450358cdd24b10f7b2085e655c89

Add SQSServiceSensor for non-polling SQS sensor This adds a SQS Sensor with its own polling loop so that we can consume messages from one or more SQS queues as quickly as possible without relying on StackStorm to trigger a poll interval.

view details

push time in 2 months

push eventgrepory/stackstorm-aws

Greg Poirier

commit sha 06516751fa37a75af7dc4cd45daa35e22b8f6a9d

At what point does the linter stop trolling me?

view details

push time in 2 months

push eventgrepory/stackstorm-aws

Greg Poirier

commit sha ef47e6792daa6c02d8c9b130c4cb29637899ec95

Appease the linter

view details

push time in 2 months

push eventgrepory/stackstorm-aws

Greg Poirier

commit sha 7c3dcdeec4f7fab471f858d727af9e9ffa863945

Import Sensor instead of Polling Sensor

view details

push time in 2 months

PR opened StackStorm-Exchange/stackstorm-aws

Add SQSServiceSensor for non-polling SQS sensor

This adds a SQS Sensor with its own polling loop so that we can consume messages from one or more SQS queues as quickly as possible without relying on StackStorm to trigger a poll interval.

Closes #90 cc @Kami

The SQSServiceSensor class name is meh, but I lack in creativity. This is what we're using right now to process anywhere between 30-100 messages per second from a single queue.

+169 -0

0 comment

2 changed files

pr created time in 2 months

create barnchgrepory/stackstorm-aws

branch : grepory/non-polling-sqs-sensor

created branch time in 2 months

fork grepory/stackstorm-aws

st2 content pack containing Amazon Web Services integrations.

https://exchange.stackstorm.org/

fork in 2 months

issue commentStackStorm-Exchange/stackstorm-aws

Change to Sensor from PollingSensor for SQS Sensor

That makes sense to me. Thanks!

grepory

comment created time in 2 months

issue openedStackStorm-Exchange/stackstorm-aws

Change to Sensor from PollingSensor for SQS Sensor

Right now, the PollingSensor caps at 10 messages per second due to the API limit for maximum messages in a transaction and the minimum polling rate of 1 Hz.

We have modified the SQS sensor locally to be a continuously running Sensor that does its own polling loop instead of a scheduled poll as we have to handle 30+ messages per second. Would this be a welcomed upstream contribution?

created time in 2 months

issue closedfluent/fluent-bit

Fluent Bit fails to start syslog input in Docker with: [plugins/in_syslog/syslog_server.c:120 errno=12] Cannot allocate memory

Bug Report

Describe the bug Fluent Bit v1.3.0 running in Docker for Mac via docker-compose.

Upon startup, I get the following error:

fluent-bit_1    | [2019/10/25 20:39:31] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
fluent-bit_1    | [2019/10/25 20:39:31] [error] [plugins/in_syslog/syslog_server.c:120 errno=12] Cannot allocate memory
fluent-bit_1    | [2019/10/25 20:39:31] [error] Failed initialize input syslog.0

Full startup logs here:

fluent-bit_1    | Fluent Bit v1.3.0
fluent-bit_1    | Copyright (C) Treasure Data
fluent-bit_1    |
fluent-bit_1    | [2019/10/25 20:39:31] [debug] [storage] [cio stream] new stream registered: syslog.0
fluent-bit_1    | [2019/10/25 20:39:31] [ info] [storage] initializing...
fluent-bit_1    | [2019/10/25 20:39:31] [ info] [storage] in-memory
fluent-bit_1    | [2019/10/25 20:39:31] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
fluent-bit_1    | [2019/10/25 20:39:31] [ info] [engine] started (pid=1)
fluent-bit_1    | [2019/10/25 20:39:31] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
fluent-bit_1    | [2019/10/25 20:39:31] [error] [plugins/in_syslog/syslog_server.c:120 errno=12] Cannot allocate memory
fluent-bit_1    | [2019/10/25 20:39:31] [error] Failed initialize input syslog.0
fluent-bit_1    | [2019/10/25 20:39:31] [ info] [out_kafka] brokers='kafka' topics='batten-logs'
fluent-bit_1    | [2019/10/25 20:39:31] [ info] [sp] stream processor started
fluent-bit_1    | [engine] caught signal (SIGTERM)
fluent-bit_1    | [2019/10/25 21:36:09] [ warn] [engine] service will stop in 5 seconds
fluent-bit_1    | [2019/10/25 21:36:13] [ info] [engine] service stopped

To Reproduce

  • Rubular link if applicable:
  • Example log message if applicable:
{"log":"YOUR LOG MESSAGE HERE","stream":"stdout","time":"2018-06-11T14:37:30.681701731Z"}
  • Steps to reproduce the problem:

docker run --volume pwd/configs/fluent-bit:/fluent-bit/etc -p 5140:5140 fluent/fluent-bit:1.3

Where configs/fluent-bit has two files:

fluent-bit.conf:

[SERVICE]
    Flush        5
    Daemon       Off
    Log_Level    debug
    Parsers_File parsers.conf
    HTTP_Server  Off
    HTTP_Listen  0.0.0.0
    HTTP_Port    2020

[INPUT]
    Name                syslog
    mode                unix_udp
    Path                /tmp/syslog_in
    Unix_Perm           0600
    Parser              eos
    Buffer_Chunk_Size   32 # KB

[OUTPUT]
    Name        kafka
    Format      json
    Message_Key message
    Brokers     kafka
    Topics      batten-logs
    Match *

parsers.conf:

[PARSER]
    Name        eos
    Format      regex
    Regex       ^(?<time>\w+ \d+ \d+:\d+:\d+)\s+(?<host>[^ ]+)\s+(?<process>.*):\s+(?<error>%[^ ]+):?\s+(?<message>.*)
    Time_Key    time
    Time_Format %b %-d %H:%M:%S

This happens regardless of the availability of the kafka broker.

Expected behavior

fluent-bit should start the udp syslog listener.

Your Environment

  • Version used: v1.2.0, v1.3.0
  • Configuration: Shown above
  • Environment name and version (e.g. Kubernetes? What version?):
➜  st2-docker git:(master) ✗ docker version                                                                                                              
Client: Docker Engine - Community
 Version:           19.03.4
 API version:       1.40
 Go version:        go1.12.10
 Git commit:        9013bf5
 Built:             Thu Oct 17 23:44:48 2019
 OS/Arch:           darwin/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.4
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.10
  Git commit:       9013bf5
  Built:            Thu Oct 17 23:50:38 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.2.10
  GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
 runc:
  Version:          1.0.0-rc8+dev
  GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683
➜  st2-docker git:(master) ✗ uname -a                                                                                                                     
Darwin C02WG0MNHV2T 18.7.0 Darwin Kernel Version 18.7.0: Tue Aug 20 16:57:14 PDT 2019; root:xnu-4903.271.2~2/RELEASE_X86_64 x86_64
  • Filters and plugins: input=syslog, filter=regex, output=kafka

closed time in 4 months

grepory

issue openedfluent/fluent-bit

Fluent Bit fails to start syslog input in Docker with: [plugins/in_syslog/syslog_server.c:120 errno=12] Cannot allocate memory

Bug Report

Describe the bug Fluent Bit v1.3.0 running in Docker for Mac via docker-compose.

Upon startup, I get the following error:

fluent-bit_1    | [2019/10/25 20:39:31] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
fluent-bit_1    | [2019/10/25 20:39:31] [error] [plugins/in_syslog/syslog_server.c:120 errno=12] Cannot allocate memory
fluent-bit_1    | [2019/10/25 20:39:31] [error] Failed initialize input syslog.0

Full startup logs here:

fluent-bit_1    | Fluent Bit v1.3.0
fluent-bit_1    | Copyright (C) Treasure Data
fluent-bit_1    |
fluent-bit_1    | [2019/10/25 20:39:31] [debug] [storage] [cio stream] new stream registered: syslog.0
fluent-bit_1    | [2019/10/25 20:39:31] [ info] [storage] initializing...
fluent-bit_1    | [2019/10/25 20:39:31] [ info] [storage] in-memory
fluent-bit_1    | [2019/10/25 20:39:31] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
fluent-bit_1    | [2019/10/25 20:39:31] [ info] [engine] started (pid=1)
fluent-bit_1    | [2019/10/25 20:39:31] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
fluent-bit_1    | [2019/10/25 20:39:31] [error] [plugins/in_syslog/syslog_server.c:120 errno=12] Cannot allocate memory
fluent-bit_1    | [2019/10/25 20:39:31] [error] Failed initialize input syslog.0
fluent-bit_1    | [2019/10/25 20:39:31] [ info] [out_kafka] brokers='kafka' topics='batten-logs'
fluent-bit_1    | [2019/10/25 20:39:31] [ info] [sp] stream processor started
fluent-bit_1    | [engine] caught signal (SIGTERM)
fluent-bit_1    | [2019/10/25 21:36:09] [ warn] [engine] service will stop in 5 seconds
fluent-bit_1    | [2019/10/25 21:36:13] [ info] [engine] service stopped

To Reproduce

  • Rubular link if applicable:
  • Example log message if applicable:
{"log":"YOUR LOG MESSAGE HERE","stream":"stdout","time":"2018-06-11T14:37:30.681701731Z"}
  • Steps to reproduce the problem:

docker run --volume pwd/configs/fluent-bit:/fluent-bit/etc -p 5140:5140 fluent/fluent-bit:1.3

Where configs/fluent-bit has two files:

fluent-bit.conf:

[SERVICE]
    Flush        5
    Daemon       Off
    Log_Level    debug
    Parsers_File parsers.conf
    HTTP_Server  Off
    HTTP_Listen  0.0.0.0
    HTTP_Port    2020

[INPUT]
    Name                syslog
    mode                unix_udp
    Path                /tmp/syslog_in
    Unix_Perm           0600
    Parser              eos
    Buffer_Chunk_Size   32 # KB

[OUTPUT]
    Name        kafka
    Format      json
    Message_Key message
    Brokers     kafka
    Topics      batten-logs
    Match *

parsers.conf:

[PARSER]
    Name        eos
    Format      regex
    Regex       ^(?<time>\w+ \d+ \d+:\d+:\d+)\s+(?<host>[^ ]+)\s+(?<process>.*):\s+(?<error>%[^ ]+):?\s+(?<message>.*)
    Time_Key    time
    Time_Format %b %-d %H:%M:%S

This happens regardless of the availability of the kafka broker.

Expected behavior

fluent-bit should start the udp syslog listener.

Your Environment

  • Version used: v1.2.0, v1.3.0
  • Configuration: Shown above
  • Environment name and version (e.g. Kubernetes? What version?):
➜  st2-docker git:(master) ✗ docker version                                                                                                              
Client: Docker Engine - Community
 Version:           19.03.4
 API version:       1.40
 Go version:        go1.12.10
 Git commit:        9013bf5
 Built:             Thu Oct 17 23:44:48 2019
 OS/Arch:           darwin/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.4
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.10
  Git commit:       9013bf5
  Built:            Thu Oct 17 23:50:38 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.2.10
  GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
 runc:
  Version:          1.0.0-rc8+dev
  GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683
➜  st2-docker git:(master) ✗ uname -a                                                                                                                     
Darwin C02WG0MNHV2T 18.7.0 Darwin Kernel Version 18.7.0: Tue Aug 20 16:57:14 PDT 2019; root:xnu-4903.271.2~2/RELEASE_X86_64 x86_64
  • Filters and plugins: input=syslog, filter=regex, output=kafka

created time in 4 months

issue openednapalm-automation/napalm-logs

kafka-python missing from requirements.txt

The official and locally built napalm-logs Docker container I made don't support the kafka transport. Upon inspection, I realized that kafka-python is missing from requirements.txt--so it will never make it into the container. I've fixed this in my fork, but wanted to know if that was deliberately left out or if there's a way to get it into the Docker container other than this that I'm not aware of.

created time in 4 months

fork grepory/napalm-logs

Cross-vendor normalisation for network syslog messages, following the OpenConfig and IETF YANG models

https://napalm-logs.com

fork in 4 months

more