Ask questionsLoki + Kubernetes labels question
Hey! I've been trying to setup Vector on my cluster, but am struggling with the config as it seems various resources are outdated, or unclear as to how to approach Loki and Kubernetes.
Loki config:
data_dir = "/var/lib/vector"
[sources.kubernetes]
type = "kubernetes"
[sinks.loki]
type = "loki"
inputs = ["kubernetes"]
endpoint = "http://loki:3100"
encoding.codec = "json"
The output of running this is:
➜ ~ kubectl -n vector logs vector-4d5np vector
May 29 18:27:28.838 INFO vector: Log level "info" is enabled.
May 29 18:27:28.838 INFO vector: Loading configs. path=["/etc/vector/vector.toml"]
May 29 18:27:28.841 ERROR vector: Configuration error: "/etc/vector/vector.toml": missing field `labels` for key `sinks.loki`
What sort of labels should I be putting the config?
How do I extract k8s labels and ship them to Loki?
See Loki's fluent-bit
config for what sorts of labels (I assume) should be added.
Answer
questions
rrichardson
I got the deployment working using the latest from the k8s-impl
branch from https://github.com/timberio/vector/pull/2653
However, the process crashes immediately with a "capacity overflow" in the Loki sink configuration.
Here is my config:
managed.toml: |
# Data dir is location controlled at the `DaemonSet`.
data_dir = "/vector-data-dir"
# Ingest logs from Kubernetes.
[sources.kubernetes_logs]
type = "kubernetes_logs"
[sinks.out]
# General
type = "loki"
inputs = ["kubernetes_logs"]
endpoint = "http://loki.loki.svc.cluster.local:3100"
healthcheck = true
encoding.codec = "json"
[sinks.out.labels]
k8s_pod_name = "{{ kubernetes.pod_name }}"
k8s_pod_uid = "{{ kubernetes.pod_uid }}"
here is the log with backtrace:
Jul 09 21:14:07.635 INFO vector: Vector is starting. version="0.10.0" git_version="v0.9.0-421-g33de5ef" released="Thu, 09 Jul 2020 20:33:25 +0000" arch="x86_64"
Jul 09 21:14:07.635 INFO vector::sources::kubernetes_logs: obtained Kubernetes Node name to collect logs for (self) self_node_name="ip-10-210-70-174.ec2.internal"
thread 'main' panicked at 'capacity overflow', src/liballoc/raw_vec.rs:622:5
stack backtrace:
0: backtrace::backtrace::libunwind::trace
at ./cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.46/src/backtrace/libunwind.rs:86
1: backtrace::backtrace::trace_unsynchronized
at ./cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.46/src/backtrace/mod.rs:66
2: std::sys_common::backtrace::_print_fmt
at src/libstd/sys_common/backtrace.rs:78
3: <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt
at src/libstd/sys_common/backtrace.rs:59
4: core::fmt::write
at src/libcore/fmt/mod.rs:1069
5: std::io::Write::write_fmt
at src/libstd/io/mod.rs:1504
6: std::sys_common::backtrace::_print
at src/libstd/sys_common/backtrace.rs:62
7: std::sys_common::backtrace::print
at src/libstd/sys_common/backtrace.rs:49
8: std::panicking::default_hook::{{closure}}
at src/libstd/panicking.rs:198
9: std::panicking::default_hook
at src/libstd/panicking.rs:218
10: std::panicking::rust_panic_with_hook
at src/libstd/panicking.rs:511
11: rust_begin_unwind
at src/libstd/panicking.rs:419
12: core::panicking::panic_fmt
at src/libcore/panicking.rs:111
13: core::panicking::panic
at src/libcore/panicking.rs:54
14: alloc::raw_vec::capacity_overflow
at src/liballoc/raw_vec.rs:622
15: alloc::raw_vec::RawVec<T,A>::allocate_in::{{closure}}
16: <vector::sinks::loki::LokiConfig as vector::topology::config::SinkConfig>::build
17: vector::topology::builder::build_pieces
18: vector::topology::builder::check_build
19: vector::topology::validate
20: vector::main::{{closure}}
21: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
22: <futures_util::compat::compat03as01::Compat<Fut> as futures::future::Future>::poll
23: futures::task_impl::std::set
24: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
25: tokio::runtime::enter::Enter::block_on
26: tokio::runtime::context::enter
27: tokio::runtime::Runtime::block_on
28: tokio_compat::runtime::threadpool::Runtime::block_on
29: vector::main
30: std::rt::lang_start::{{closure}}
31: std::rt::lang_start_internal::{{closure}}
at src/libstd/rt.rs:52
32: std::panicking::try::do_call
at src/libstd/panicking.rs:331
33: std::panicking::try
at src/libstd/panicking.rs:274
34: std::panic::catch_unwind
at src/libstd/panic.rs:394
35: std::rt::lang_start_internal
at src/libstd/rt.rs:51
36: main
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
Related questions
No questions were found.