profile
viewpoint
Greg Poirier grepory @sensu San Francisco, CA

issue closedfluent/fluent-bit

Fluent Bit fails to start syslog input in Docker with: [plugins/in_syslog/syslog_server.c:120 errno=12] Cannot allocate memory

Bug Report

Describe the bug Fluent Bit v1.3.0 running in Docker for Mac via docker-compose.

Upon startup, I get the following error:

fluent-bit_1    | [2019/10/25 20:39:31] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
fluent-bit_1    | [2019/10/25 20:39:31] [error] [plugins/in_syslog/syslog_server.c:120 errno=12] Cannot allocate memory
fluent-bit_1    | [2019/10/25 20:39:31] [error] Failed initialize input syslog.0

Full startup logs here:

fluent-bit_1    | Fluent Bit v1.3.0
fluent-bit_1    | Copyright (C) Treasure Data
fluent-bit_1    |
fluent-bit_1    | [2019/10/25 20:39:31] [debug] [storage] [cio stream] new stream registered: syslog.0
fluent-bit_1    | [2019/10/25 20:39:31] [ info] [storage] initializing...
fluent-bit_1    | [2019/10/25 20:39:31] [ info] [storage] in-memory
fluent-bit_1    | [2019/10/25 20:39:31] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
fluent-bit_1    | [2019/10/25 20:39:31] [ info] [engine] started (pid=1)
fluent-bit_1    | [2019/10/25 20:39:31] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
fluent-bit_1    | [2019/10/25 20:39:31] [error] [plugins/in_syslog/syslog_server.c:120 errno=12] Cannot allocate memory
fluent-bit_1    | [2019/10/25 20:39:31] [error] Failed initialize input syslog.0
fluent-bit_1    | [2019/10/25 20:39:31] [ info] [out_kafka] brokers='kafka' topics='batten-logs'
fluent-bit_1    | [2019/10/25 20:39:31] [ info] [sp] stream processor started
fluent-bit_1    | [engine] caught signal (SIGTERM)
fluent-bit_1    | [2019/10/25 21:36:09] [ warn] [engine] service will stop in 5 seconds
fluent-bit_1    | [2019/10/25 21:36:13] [ info] [engine] service stopped

To Reproduce

  • Rubular link if applicable:
  • Example log message if applicable:
{"log":"YOUR LOG MESSAGE HERE","stream":"stdout","time":"2018-06-11T14:37:30.681701731Z"}
  • Steps to reproduce the problem:

docker run --volume pwd/configs/fluent-bit:/fluent-bit/etc -p 5140:5140 fluent/fluent-bit:1.3

Where configs/fluent-bit has two files:

fluent-bit.conf:

[SERVICE]
    Flush        5
    Daemon       Off
    Log_Level    debug
    Parsers_File parsers.conf
    HTTP_Server  Off
    HTTP_Listen  0.0.0.0
    HTTP_Port    2020

[INPUT]
    Name                syslog
    mode                unix_udp
    Path                /tmp/syslog_in
    Unix_Perm           0600
    Parser              eos
    Buffer_Chunk_Size   32 # KB

[OUTPUT]
    Name        kafka
    Format      json
    Message_Key message
    Brokers     kafka
    Topics      batten-logs
    Match *

parsers.conf:

[PARSER]
    Name        eos
    Format      regex
    Regex       ^(?<time>\w+ \d+ \d+:\d+:\d+)\s+(?<host>[^ ]+)\s+(?<process>.*):\s+(?<error>%[^ ]+):?\s+(?<message>.*)
    Time_Key    time
    Time_Format %b %-d %H:%M:%S

This happens regardless of the availability of the kafka broker.

Expected behavior

fluent-bit should start the udp syslog listener.

Your Environment

  • Version used: v1.2.0, v1.3.0
  • Configuration: Shown above
  • Environment name and version (e.g. Kubernetes? What version?):
➜  st2-docker git:(master) ✗ docker version                                                                                                              
Client: Docker Engine - Community
 Version:           19.03.4
 API version:       1.40
 Go version:        go1.12.10
 Git commit:        9013bf5
 Built:             Thu Oct 17 23:44:48 2019
 OS/Arch:           darwin/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.4
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.10
  Git commit:       9013bf5
  Built:            Thu Oct 17 23:50:38 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.2.10
  GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
 runc:
  Version:          1.0.0-rc8+dev
  GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683
➜  st2-docker git:(master) ✗ uname -a                                                                                                                     
Darwin C02WG0MNHV2T 18.7.0 Darwin Kernel Version 18.7.0: Tue Aug 20 16:57:14 PDT 2019; root:xnu-4903.271.2~2/RELEASE_X86_64 x86_64
  • Filters and plugins: input=syslog, filter=regex, output=kafka

closed time in 19 days

grepory

issue openedfluent/fluent-bit

Fluent Bit fails to start syslog input in Docker with: [plugins/in_syslog/syslog_server.c:120 errno=12] Cannot allocate memory

Bug Report

Describe the bug Fluent Bit v1.3.0 running in Docker for Mac via docker-compose.

Upon startup, I get the following error:

fluent-bit_1    | [2019/10/25 20:39:31] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
fluent-bit_1    | [2019/10/25 20:39:31] [error] [plugins/in_syslog/syslog_server.c:120 errno=12] Cannot allocate memory
fluent-bit_1    | [2019/10/25 20:39:31] [error] Failed initialize input syslog.0

Full startup logs here:

fluent-bit_1    | Fluent Bit v1.3.0
fluent-bit_1    | Copyright (C) Treasure Data
fluent-bit_1    |
fluent-bit_1    | [2019/10/25 20:39:31] [debug] [storage] [cio stream] new stream registered: syslog.0
fluent-bit_1    | [2019/10/25 20:39:31] [ info] [storage] initializing...
fluent-bit_1    | [2019/10/25 20:39:31] [ info] [storage] in-memory
fluent-bit_1    | [2019/10/25 20:39:31] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
fluent-bit_1    | [2019/10/25 20:39:31] [ info] [engine] started (pid=1)
fluent-bit_1    | [2019/10/25 20:39:31] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
fluent-bit_1    | [2019/10/25 20:39:31] [error] [plugins/in_syslog/syslog_server.c:120 errno=12] Cannot allocate memory
fluent-bit_1    | [2019/10/25 20:39:31] [error] Failed initialize input syslog.0
fluent-bit_1    | [2019/10/25 20:39:31] [ info] [out_kafka] brokers='kafka' topics='batten-logs'
fluent-bit_1    | [2019/10/25 20:39:31] [ info] [sp] stream processor started
fluent-bit_1    | [engine] caught signal (SIGTERM)
fluent-bit_1    | [2019/10/25 21:36:09] [ warn] [engine] service will stop in 5 seconds
fluent-bit_1    | [2019/10/25 21:36:13] [ info] [engine] service stopped

To Reproduce

  • Rubular link if applicable:
  • Example log message if applicable:
{"log":"YOUR LOG MESSAGE HERE","stream":"stdout","time":"2018-06-11T14:37:30.681701731Z"}
  • Steps to reproduce the problem:

docker run --volume pwd/configs/fluent-bit:/fluent-bit/etc -p 5140:5140 fluent/fluent-bit:1.3

Where configs/fluent-bit has two files:

fluent-bit.conf:

[SERVICE]
    Flush        5
    Daemon       Off
    Log_Level    debug
    Parsers_File parsers.conf
    HTTP_Server  Off
    HTTP_Listen  0.0.0.0
    HTTP_Port    2020

[INPUT]
    Name                syslog
    mode                unix_udp
    Path                /tmp/syslog_in
    Unix_Perm           0600
    Parser              eos
    Buffer_Chunk_Size   32 # KB

[OUTPUT]
    Name        kafka
    Format      json
    Message_Key message
    Brokers     kafka
    Topics      batten-logs
    Match *

parsers.conf:

[PARSER]
    Name        eos
    Format      regex
    Regex       ^(?<time>\w+ \d+ \d+:\d+:\d+)\s+(?<host>[^ ]+)\s+(?<process>.*):\s+(?<error>%[^ ]+):?\s+(?<message>.*)
    Time_Key    time
    Time_Format %b %-d %H:%M:%S

This happens regardless of the availability of the kafka broker.

Expected behavior

fluent-bit should start the udp syslog listener.

Your Environment

  • Version used: v1.2.0, v1.3.0
  • Configuration: Shown above
  • Environment name and version (e.g. Kubernetes? What version?):
➜  st2-docker git:(master) ✗ docker version                                                                                                              
Client: Docker Engine - Community
 Version:           19.03.4
 API version:       1.40
 Go version:        go1.12.10
 Git commit:        9013bf5
 Built:             Thu Oct 17 23:44:48 2019
 OS/Arch:           darwin/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.4
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.10
  Git commit:       9013bf5
  Built:            Thu Oct 17 23:50:38 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.2.10
  GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
 runc:
  Version:          1.0.0-rc8+dev
  GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683
➜  st2-docker git:(master) ✗ uname -a                                                                                                                     
Darwin C02WG0MNHV2T 18.7.0 Darwin Kernel Version 18.7.0: Tue Aug 20 16:57:14 PDT 2019; root:xnu-4903.271.2~2/RELEASE_X86_64 x86_64
  • Filters and plugins: input=syslog, filter=regex, output=kafka

created time in 22 days

issue openednapalm-automation/napalm-logs

kafka-python missing from requirements.txt

The official and locally built napalm-logs Docker container I made don't support the kafka transport. Upon inspection, I realized that kafka-python is missing from requirements.txt--so it will never make it into the container. I've fixed this in my fork, but wanted to know if that was deliberately left out or if there's a way to get it into the Docker container other than this that I'm not aware of.

created time in 25 days

fork grepory/napalm-logs

Cross-vendor normalisation for network syslog messages, following the OpenConfig and IETF YANG models

https://napalm-logs.com

fork in a month

more