profile
viewpoint
coffee coder c0ffeec0der Code as long as there is coffee

c0ffeec0der/scylla-monitoring 0

Simple monitoring of Scylla with Grafana

c0ffeec0der/scylla-operator 0

Kubernetes operator for Scylla

c0ffeec0der/tanka 0

Flexible, reusable and concise configuration for Kubernetes

c0ffeec0der/terraform-google-bastion-host 0

This module will generate a bastion host vm compatible with OS Login and IAP Tunneling that can be used to access internal VMs.

c0ffeec0der/terraform-google-kubernetes-engine 0

A Terraform module for configuring GKE clusters.

pull request commentgrafana/jsonnet-libs

Memcached: reduce cpu request

Thank you @jtlisi

I could do it with:

memcached_chunks+: { memory_limit_mb: 1024, memcached_container+:: $.util.resourcesRequests('10m', $.util.bytesToK8sQuantity(self.memory_request_bytes)) + $.util.resourcesLimits('20m',$.util.bytesToK8sQuantity(self.memory_limits_bytes)), },

c0ffeec0der

comment created time in 13 days

issue closedgrafana/loki

Feature request: Configurable resource request and limit on all loki components

Hello,

I would like to request feature when deploying with tanka, resource request and limit is configurable.


Is your feature request related to a problem? Please describe. Loki components resource request and limit is too large light to moderate usage

Describe the solution you'd like Configurable resource request and limit

Describe alternatives you've considered Using single instance

Additional context

closed time in 13 days

c0ffeec0der

issue commentgrafana/loki

Feature request: Configurable resource request and limit on all loki components

Solution from conversations

// adjust resource gateway_container+:: $.util.resourcesRequests('10m', '10Mi') + $.util.resourcesLimits('20m','20Mi'),

// adjust replica gateway_deployment+: $.apps.v1.deployment.mixin.spec.withReplicas(1),

// adjust resource for memcached memcached_chunks+: { memory_limit_mb: 1024, memcached_container+:: $.util.resourcesRequests('10m', $.util.bytesToK8sQuantity(self.memory_request_bytes)) + $.util.resourcesLimits('20m',$.util.bytesToK8sQuantity(self.memory_limits_bytes)), },

c0ffeec0der

comment created time in 13 days

PR closed grafana/loki

Reviewers
Ksonnet: Configurable resource request and limit on all loki components size/L

Changes:

Loki.libsonnet (changed)

  • Update import for consul from jsonnet-libs/consul to local consul.libsonnet

config.libsonnet (changed)

  • Add consul_replicas
  • ingester :
    • Update pvc size from 5 to 10Gi (same as ingester.libsonnet)
    • Add replicas, resource request and limit (same as ingester.libsonnet)
  • querier : add replicas, resource request and limit (same as querier.libsonnet)
  • query-frontend : add resource request and limit (same as query-frontend.libsonnet)
  • table-manager : add replicas, resource request and limit (same as table-manager.libsonnet)
  • gateway : add replicas, resource request and limit (same as table-manager.libsonnet)
  • distributor : add replicas, resource request and limit (same as table-manager.libsonnet)
  • storage_section: Reformat section

boltdb_shipper.libsonnet

  • Rename stateful parameter

consul.libsonnet (new)

  • Add explicit resource requirement (same as jsonnet-libs jsonnet-libs/consul)

ingester.libsonnet (changed)

  • Refer replicas, resource request and limit, pvc size to config.libsonnet

querier.libsonnet (changed)

  • Refer replicas, resource request and limit to config.libsonnet

query-frontend.libsonnet (changed)

  • Refer resource request and limit to config.libsonnet

table-manager.libsonnet (changed)

  • Refer replicas, resource request and limit to config.libsonnet

gateway.libsonnet (changed)

  • Refer replicas, resource request and limit to config.libsonnet

distributor.libsonnet (changed)

  • Refer replicas, resource request and limit to config.libsonnet

memcached.libsonnet (changed)

  • Expose max_item_size and memory_limit_mb to config.libsonnet

<!-- Thanks for sending a pull request! Before submitting:

  1. Read our CONTRIBUTING.md guide
  2. Name your PR as <Feature Area>: Describe your change
  3. Rebase your PR if it gets out of sync with master
  4. If changing the Helm chart, please ensure the chart version is increased per semantic versioning (https://semver.org) -->

What this PR does / why we need it: Provides configurable resource request and limit in config. Beneficial for user who would want to use microservice but start with modest resource

Which issue(s) this PR fixes: #2687

Special notes for your reviewer: For memcached_chunks, it looks like tanka have its own calc by container arguments? I am putting the values the same as generated manifest. Suggest correction if the value is incorrect

Checklist

  • [ ] Documentation added
  • [ ] Tests updated
+212 -108

9 comments

12 changed files

c0ffeec0der

pr closed time in 13 days

pull request commentgrafana/loki

Ksonnet: Configurable resource request and limit on all loki components

Hello @owen-d

Thank you! This is a relief! I was figuring out how to do it in main.jsonnet for quite a time before changing config.libsonnet

I am posting the solution here for other people who want to adjust the resource

// adjust resource gateway_container+:: $.util.resourcesRequests('10m', '10Mi') + $.util.resourcesLimits('20m','20Mi'),

// adjust replica gateway_deployment+: $.apps.v1.deployment.mixin.spec.withReplicas(1),

// adjust resource for memcached memcached_chunks+: { memory_limit_mb: 1024, memcached_container+:: $.util.resourcesRequests('10m', $.util.bytesToK8sQuantity(self.memory_request_bytes)) + $.util.resourcesLimits('20m',$.util.bytesToK8sQuantity(self.memory_limits_bytes)), },

Closing the PR :)

I am posting the solution here for other people who want to adjust the resource

// adjust resource gateway_container+:: $.util.resourcesRequests('10m', '10Mi') + $.util.resourcesLimits('20m','20Mi'),

// adjust replica gateway_deployment+: $.apps.v1.deployment.mixin.spec.withReplicas(1),

// adjust resource for memcached memcached_chunks+: { memory_limit_mb: 1024, memcached_container+:: $.util.resourcesRequests('10m', $.util.bytesToK8sQuantity(self.memory_request_bytes)) + $.util.resourcesLimits('20m',$.util.bytesToK8sQuantity(self.memory_limits_bytes)), },

Closing the PR :)

c0ffeec0der

comment created time in 13 days

pull request commentgrafana/loki

Ksonnet: Configurable resource request and limit on all loki components

Hey, I don't think I was clear enough earlier, but I was suggesting we create the $._config.components object to store this data. It doesn't as of yet exist and that's why you were seeing errors there.

I had created $._config.components object but it could not be referenced to from config.libsonnet or other files. There is a limitation in tanka that during runtime it could only call parse reference to '$._config'. I know it's odd, but I ran to runtime error when using $._config.components and tried to reference it.

After talking about this issue with @slim-bean, I'm questioning if adding these parameters is actually helpful. Jsonnet, unlike helm, can easily be arbitrarily overridden

Yes, I agree jsonnet can be overriden. But when installing loki with tanka, I could not change the resource request from environments/loki/main.jsonnet, that is why I go through the troubles exposing them all in config.libsonnet. If it's not a trouble, could you please show me how to override the resource request from environments/loki/main.jsonnet?

c0ffeec0der

comment created time in 14 days

pull request commentgrafana/loki

Ksonnet: Configurable resource request and limit on all loki components

Hello @owen-d, I just saw there was a file conflict. It's corrected now. Please review and let me know your thoughts. Thank you

c0ffeec0der

comment created time in 14 days

push eventc0ffeec0der/loki

Philipp Gassmann

commit sha 5cb32668c7c9eeb19abb39b8c0e2a294daf69b99

docker plugin documentation update (#2689) * docker plugin documentation update add --force to plugin disable link installation and configuration of the plugin * grant all permissions on upgrade * change wording

view details

Stefan Grühn

commit sha fc00227d5c25497f5ad599f1ad6537d286ccfe46

Update metrics.md (#2691) Fix wrong parameter name (#2688)

view details

Chad H

commit sha c53679181384b8aa899ec6f500a2b1b2e28f0491

Fix link to code of conduct (#2686)

view details

BaiLian

commit sha 0ac1ce4e5a502245f10279e336d0db52d8590672

Querier APIs respond JSON Content-Type (#2665)

view details

Ed Welch

commit sha 2162a0c1b1844b20aaecf2a2859dbb0c58c30936

Do not mark the position if the file is removed, doing so only results in an error because you can't get the file size of a removed file. (#2658)

view details

Owen Diehl

commit sha d8e571e3134a6bd7de05742084a5d892a8a7e39c

updates alerting docs with new cortex tool loki linting support (#2697)

view details

RS Krishna

commit sha bffd9a5c98a2025560d8b3d18a23046bb45d1484

[Promtail] enables configuring psp in helm chart (#2659) * makes pod security policy configurable * bump promtail chart version * bump loki-stack chart version

view details

Jiajin Zheng

commit sha 1b2bae46ba4b1f6e75f152aabef84cf7394df2db

fluent-bit plugin support tls (#2568)

view details

Cyril Tovena

commit sha 6500f821951b737c980df832d9d78a34220430d6

Improve lz4 compression (#2614) * Improve lz4 compression. - Move to v4. - Remove not required checksuming. - Default will now write 4M blocks, it's backward compatible for reads. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * vendor update Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>

view details

Owen Diehl

commit sha 499e4efc24d03b0e0029d62010a76a107bf135ab

shows cortextool lint command for loki in alerting docs (#2705)

view details

San Nguyen

commit sha c00c7ed252dc360f92fc9ba8e490f6225e60d3bd

docs: use repetitive numbering (#2699)

view details

Cyril Tovena

commit sha 3640429532303fefc447e993141ab981bf040665

Bypass sharding middleware when a query can't be sharded. (#2709) Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>

view details

Benjamin Krämer

commit sha 9e6afea5f6353212f403768e2de5174a2364fdea

Fix timestamp parser for short year format (#2708)

view details

Ed Welch

commit sha d3bf21e7742882d6696e865ddde0f8b8a081bc96

Promtail: (and also fluent-bit) change the max batch size to 1MB (#2710) * change the max batch size to 1MB for all the defaults including helm and fluent-bit, attempt to centralize this config a little where possible. * fix test

view details

Ed Welch

commit sha 8ea6c381cb39f5aae2592a1948e59060f9956009

Promtail: Fix deadlock on tailer shutdown. (#2717) * to avoid a deadlock on shutdown, leave the goroutine running which reads the Lines channel from the tailer and it will exit when the tailer closes the channel. * remove the flag I added a few weeks ago and instead when updating the position information of the file gracefully fail if the file no longer exists. * update the vendoring of the tail library, more re-work on how tailing shutdown and errors are handled. * fix mistake * updating tail lib to merged commit hash * go mod tidy * update tail library with additional mutex lock * break up the tailer goroutine into two separate goroutines to make it a little more clear and remove some special case logic. * don't forget to close your channels! * make the stop method in the tailer only execute at most one time. * update hp-tail again with improvements to the mutex locking in closeFile * replace the more complicated channel based single execution code with a sync.Once

view details

Calle Pettersson

commit sha 426bac7f5b283d03b95ef022c8015bc966433194

Add counter promtail_batch_retries_total (#2723) Signed-off-by: Calle Pettersson <calle@cape.nu>

view details

Matt Young

commit sha 5973f044bf01c40c71d152504cde3f71a75d9aad

nit: fix formatting for ksonnet/loki (#2719)

view details

huikang

commit sha b9e8ddbb1bb0af7b48bf42a65dae8e12cf212b7c

Doc: fixing parameter name in configuration (#2716) - http_path_prefix should be http_prefix Signed-off-by: Hui Kang <kangh@us.ibm.com>

view details

Madhav Jivrajani

commit sha 356e2765ff54fe0d0e945e7013ab81855f7f45a2

[DOCS] change format of unordered lists in tech docs (#2724)

view details

Davor Kapsa

commit sha fffcfa6e814fd3e2128cb67125bcabcdabc2df31

fix some misspells (#2725)

view details

push time in 14 days

push eventc0ffeec0der/loki

c0ffeec0der

commit sha eec0334e828fd151938b5f2276163a9d9f36c812

Clear error symlink

view details

push time in 14 days

PR closed grafana/loki

Promtail: Helm and Ksonnet: Clear error symlink size/XS

<!-- Thanks for sending a pull request! Before submitting:

  1. Read our CONTRIBUTING.md guide
  2. Name your PR as <Feature Area>: Describe your change
  3. Rebase your PR if it gets out of sync with master
  4. If changing the Helm chart, please ensure the chart version is increased per semantic versioning (https://semver.org) -->

What this PR does / why we need it: Clear promtail log from reporting error for non-existent symlink. Beneficial for new deployment to avoid unecessary logs due to k8s issue

Which issue(s) this PR fixes: Fixes #2513

Special notes for your reviewer: Should updating values increase helm version?

Checklist

  • [ ] Documentation added
  • [ ] Tests updated
+5 -0

3 comments

2 changed files

c0ffeec0der

pr closed time in 14 days

pull request commentgrafana/loki

Promtail: Helm and Ksonnet: Clear error symlink

Hello @slim-bean , ok no worries. I'll close the PR :)

Note:

  • For edge user who have similiar issue, please refer to the file changes section
c0ffeec0der

comment created time in 14 days

pull request commentgrafana/jsonnet-libs

Memcached: reduce cpu request

Hello,

Yes I am aware of it. The issue is that this library is being used by loki

https://github.com/grafana/loki/blob/master/production/ksonnet/loki/memcached.libsonnet

and it seems that changing loki library to expose direct memcached resource change is not preffered

https://github.com/grafana/loki/pull/2695#discussion_r496865690

I am proposing to the loki contributor to expose available memcached args. However, memcached cpu request is not available in args.

My first solution is to reduce the memcached cpu request directly as in this PR, this is simpler as it will not affect other libary using it.

If this solution is not preffered, then can we expose the minimum cpu in args? So that user of loki library can adjust memcached cpu request?

What are your thoughts?

c0ffeec0der

comment created time in 14 days

pull request commentgrafana/loki

Promtail: Helm and ksonnet: Clear promtail log from reporting error for non-existent symlink

@slim-bean Hello, would you be able to comment on this? Thank you

c0ffeec0der

comment created time in 15 days

pull request commentgrafana/jsonnet-libs

Memcached: reduce cpu request

Hello, anyone to comment? Thanks

c0ffeec0der

comment created time in 15 days

pull request commentgrafana/loki

Ksonnet: Configurable resource request and limit on all loki components

@owen-d Hello, any updates, comments?

c0ffeec0der

comment created time in 15 days

PR opened grafana/loki

Promtail: Helm and Ksonnet: Clear error symlink

<!-- Thanks for sending a pull request! Before submitting:

  1. Read our CONTRIBUTING.md guide
  2. Name your PR as <Feature Area>: Describe your change
  3. Rebase your PR if it gets out of sync with master
  4. If changing the Helm chart, please ensure the chart version is increased per semantic versioning (https://semver.org) -->

What this PR does / why we need it: Clear promtail log from reporting error for non-existent symlink

Which issue(s) this PR fixes: Fixes #2513

Special notes for your reviewer: Should updating values increase helm version?

Checklist

  • [ ] Documentation added
  • [ ] Tests updated
+5 -0

0 comment

2 changed files

pr created time in 20 days

create barnchc0ffeec0der/loki

branch : fix/error_symlink3

created branch time in 20 days

delete branch c0ffeec0der/loki

delete branch : fix/error_symlink2

delete time in 20 days

create barnchc0ffeec0der/loki

branch : fix/error_symlink2

created branch time in 20 days

delete branch c0ffeec0der/loki

delete branch : fix/error_symlink2

delete time in 20 days

create barnchc0ffeec0der/loki

branch : fix/error_symlink2

created branch time in 20 days

pull request commentgrafana/loki

Promtail: Helm and ksonnet: Clear promtail log from reporting error for non-existent symlink

The files changed in this branch is incorrect. will create a new PR

c0ffeec0der

comment created time in 20 days

delete branch c0ffeec0der/loki

delete branch : fix/error_symlink

delete time in 20 days

PR closed grafana/loki

Promtail: Helm and ksonnet: Clear promtail log from reporting error for non-existent symlink size/L

<!-- Thanks for sending a pull request! Before submitting:

  1. Read our CONTRIBUTING.md guide
  2. Name your PR as <Feature Area>: Describe your change
  3. Rebase your PR if it gets out of sync with master
  4. If changing the Helm chart, please ensure the chart version is increased per semantic versioning (https://semver.org) -->

What this PR does / why we need it: Clear promtail log from reporting error for non-existent symlink

Which issue(s) this PR fixes: Fixes #2513

Special notes for your reviewer: Should updating values increase helm version?

Checklist

  • [ ] Documentation added
  • [ ] Tests updated
+212 -103

1 comment

14 changed files

c0ffeec0der

pr closed time in 20 days

PR opened grafana/loki

Promtail: Helm and ksonnet: Clear promtail log from reporting error for non-existent symlink

<!-- Thanks for sending a pull request! Before submitting:

  1. Read our CONTRIBUTING.md guide
  2. Name your PR as <Feature Area>: Describe your change
  3. Rebase your PR if it gets out of sync with master
  4. If changing the Helm chart, please ensure the chart version is increased per semantic versioning (https://semver.org) -->

What this PR does / why we need it: Clear promtail log from reporting error for non-existent symlink

Which issue(s) this PR fixes: Fixes #2513

Special notes for your reviewer: Should updating values increase helm version?

Checklist

  • [ ] Documentation added
  • [ ] Tests updated
+212 -103

0 comment

14 changed files

pr created time in 20 days

create barnchc0ffeec0der/loki

branch : fix/error_symlink

created branch time in 20 days

PR opened grafana/jsonnet-libs

Memcached: reduce cpu request

Hello,

I would like to request to reduce cpu request from 500m to 100m.

500m is to much as starting point for small servers. 100m is also sufficient to keep alive most programs

Thank you

+1 -1

0 comment

1 changed file

pr created time in 20 days

create barnchc0ffeec0der/jsonnet-libs

branch : feat/memcached_cpu_request

created branch time in 20 days

fork c0ffeec0der/jsonnet-libs

Grafana Labs' Jsonnet libraries

fork in 20 days

pull request commentgrafana/loki

Ksonnet: Configurable resource request and limit on all loki components

There is unfortunate limitation in tanka that it could refer to <$._config.components>. Failed on runtime.. It seems refferign to nested variable is not supported yet. (possible different parser engine with https://github.com/google/jsonnet) so I have to put in $._config

c0ffeec0der

comment created time in 20 days

Pull request review commentgrafana/loki

Ksonnet: Configurable resource request and limit on all loki components

+local consul = import "consul/consul.libsonnet";++consul {

Sorry, I forgot to expose them to config.libsonnet

c0ffeec0der

comment created time in 20 days

PullRequestReviewEvent

Pull request review commentgrafana/loki

Ksonnet: Configurable resource request and limit on all loki components

 memcached {     name: 'memcached',     max_item_size: '2m',     memory_limit_mb: 4096,+

Ok, I am passing max_item_size and memory_limit_mb args to config.libsonnet

c0ffeec0der

comment created time in 20 days

PullRequestReviewEvent

push eventc0ffeec0der/loki

c0ffeec0der

commit sha 34aad55d01ee51d2c712bdb112020570936e1a37

Refer to main commit message for changes

view details

push time in 20 days

PR opened grafana/loki

Ksonnet: Configurable resource request and limit on all loki components

Changes:

Loki.libsonnet (changed)

  • Update import for consul from jsonnet-libs/consul to local consul.libsonnet

config.libsonnet (changed)

  • Add consul_replicas
  • ingester :
    • Update pvc size from 5 to 10Gi (same as ingester.libsonnet)
    • Add replicas, resource request and limit (same as ingester.libsonnet)
  • querier : add replicas, resource request and limit (same as querier.libsonnet)
  • query-frontend : add resource request and limit (same as query-frontend.libsonnet)
  • table-manager : add replicas, resource request and limit (same as table-manager.libsonnet)
  • gateway : add replicas, resource request and limit (same as table-manager.libsonnet)
  • distributor : add replicas, resource request and limit (same as table-manager.libsonnet)

consul.libsonnet (new)

  • Add explicit resource requirement (same as jsonnet-libs jsonnet-libs/consul)

ingester.libsonnet (changed)

  • Refer replicas, resource request and limit, pvc size to config.libsonnet

querier.libsonnet (changed)

  • Refer replicas, resource request and limit to config.libsonnet

query-frontend.libsonnet (changed)

  • Refer resource request and limit to config.libsonnet

table-manager.libsonnet (changed)

  • Refer replicas, resource request and limit to config.libsonnet

gateway.libsonnet (changed)

  • Refer replicas, resource request and limit to config.libsonnet

distributor.libsonnet (changed)

  • Refer replicas, resource request and limit to config.libsonnet

memcached.libsonnet (changed)

  • Add explicit resource requirement (based on generated loki manifests)

<!-- Thanks for sending a pull request! Before submitting:

  1. Read our CONTRIBUTING.md guide
  2. Name your PR as <Feature Area>: Describe your change
  3. Rebase your PR if it gets out of sync with master
  4. If changing the Helm chart, please ensure the chart version is increased per semantic versioning (https://semver.org) -->

What this PR does / why we need it: Provides configurable resource request and limit in config. Beneficial for user who would want to use microservice but start with modest resource

Which issue(s) this PR fixes: #2687

Special notes for your reviewer: For memcached_chunks, it looks like tanka have its own calc by container arguments? I am putting the values the same as generated manifest. Suggest correction if the value is incorrect

Checklist

  • [ ] Documentation added
  • [ ] Tests updated
+160 -22

0 comment

10 changed files

pr created time in 21 days

create barnchc0ffeec0der/loki

branch : feature/configurable_resource

created branch time in 21 days

IssuesEvent

fork c0ffeec0der/loki

Like Prometheus, but for logs.

https://grafana.com/loki

fork in 21 days

issue closedgrafana/loki

Feature request: Configurable resource request and limit (cpu, memory, pvc size) on all loki components (i.e distributor, ingester)

Hello,

I would like to request feature when deploying with tanka, resource request and limit is configurable.

I am happy to contribute.

If you approve this request, I will start with the code

Thank you


Is your feature request related to a problem? Please describe. Loki components resource request and limit is too large light to moderate usage

Describe the solution you'd like Configurable resource request and limit

Describe alternatives you've considered Using single instance

Additional context

closed time in 22 days

c0ffeec0der

issue openedgrafana/loki

Feature request: Configurable resource request and limit (cpu, memory, pvc size) on all loki components (i.e distributor, ingester)

Hello,

I would like to request feature when deploying with tanka, resource request and limit is configurable.

I am happy to contribute.

If you approve this request, I will start with the code

Thank you


Is your feature request related to a problem? Please describe. Loki components resource request and limit is too large light to moderate usage

Describe the solution you'd like Configurable resource request and limit

Describe alternatives you've considered Using single instance

Additional context

created time in 22 days

issue commentgrafana/loki

Bug: Promtail log is reporting error for non-existent symlink

Hello

After some reading on the docs,

It seems like to add 'drop' function would be to add below code in pipeline_stages

  - drop:          
      expression: ".*(error).*(filetarget.go).*(no such file).*"

for existing scraping job:

  1. kubernetes-pods-name
  2. kubernetes-pods-app
  3. kubernetes-pods-direct-controllers
  4. kubernetes-pods-indirect-controller
  5. kubernetes-pods-static

The reason being is that these jobs are created so that the subsequent jobs covered the relabeling process so that it does not create duplicate entry. (i.e job 1 detect by __meta_kubernetes_pod_label_name, job 2 skip if __meta_kubernetes_pod_label_app has value and detect by __meta_kubernetes_pod_label_app)

If you agree on this, we could modify file

https://github.com/grafana/loki/blob/master/production/helm/promtail/values.yaml

for section pipelineStages. Let me know and I will create a PR

c0ffeec0der

comment created time in 2 months

pull request commentgrafana/tanka

Fix issue #329: Windows indefinite loop

Hello,

Is there still anything required for this PR to be merged?

Thank you

c0ffeec0der

comment created time in 2 months

issue commentgrafana/loki

Bug: Promtail log is reporting error for non-existent symlink

Hello,

Thank you for the explanation. I think I understand now why you would like to leave the symlink as is. It seems promtail is copying all the actual log files and symlinks from the containers; keep it as original as is, and then iterate one by one.

I am still playing with the drop function in version 1.6.0, so far no luck.. I'll try reading the docs and update when I found the solution.

Thank you

c0ffeec0der

comment created time in 2 months

issue commentgrafana/loki

Bug: Promtail log is reporting error for non-existent symlink

Hello,

Yes you are right. I did comment on #429. However, this happen after a clean reset of the kubernetes. where you did not experience based on comment https://github.com/grafana/loki/issues/429#issuecomment-502453125

#429 assumed the root cause is from docker daemon not cleaning up symlink, https://github.com/grafana/loki/issues/429#issuecomment-520859169.

I created this issue because it still appears after a clean reset. Sorry if I didn't mention this in the first place. And it occurs only on some pods (in this case kube-apiserver, kube-controller, kube-scheduler). The other pods are doing ok.

Is this the still the same issue as #429 ?

It's just so confusing because a fresh install, immediately creates error log.

I was also thinking about dropping while scrapping the log as workaround (assuming the concept with prometheus because I haven't gone into detail with promtail). It would clean the log before entering loki and chunk store. However, promtail own log file would still suffer

Question : Why not upon detection of "err no such file or directory", delete the file in process instead of showing message. Is there another scenario where the detection is needed?

c0ffeec0der

comment created time in 2 months

issue openedgrafana/loki

Promtail log is reporting error for non-existent symlink

Describe the bug Promtail log is reporting error for non-existent symlink

Sample error log

level=error ts=2020-08-15T11:01:19.658356919Z caller=filetarget.go:283 msg="failed to tail file, stat failed" error="stat /var/log/pods/kube-system_kube-apiserver-docker-desktop_2966bdea771eade42b079889c75cf17e/kube-apiserver/2.log: no such file or directory" filename=/var/log/pods/kube-system_kube-apiserver-docker-desktop_2966bdea771eade42b079889c75cf17e/kube-apiserver/2.log

File 2.log is pointing to

lrwxrwxrwx 1 root root 165 Aug 15 09:07 2.log -> /var/lib/docker/containers/52b0d469a5ba5e6f1124d67c0b91308151f34fc9df5183c80b24c712bb52de32/52b0d469a5ba5e 6f1124d67c0b91308151f34fc9df5183c80b24c712bb52de32-json.log

When perform "cat" to target file json.log

cat: /var/lib/docker/containers/52b0d469a5ba5e6f1124d67c0b91308151f34fc9df5183c80b24c712bb52de32/52b0d469a5ba5e6f1124d67c0b91308151f34fc9df5183c80b24c712bb 52de32-json.log: No such file or directory

The pod kube-system_kube-apiserver-docker-desktop is up and running. In grafana explorer, promtail return logs for the pod. They are in file 0.log

To Reproduce Steps to reproduce the behavior:

  1. Started Loki (1.5)
  2. Started Promtail (1.5)
  3. Query: {pod="loki-promtail-[input your pod id]"}|="error"

Expected behavior no error as this is fresh run

Environment:

  • Infrastructure: Docker desktop with kubernetes 1.16.5
  • Deployment tool: helm

Screenshots, Promtail config, or terminal output image

created time in 2 months

issue commentgrafana/loki

promtail logs no such file or directory

Issue still exists:

Log from the pod (deployed using helm)

level=error ts=2020-08-15T04:05:58.53556454Z caller=filetarget.go:283 msg="failed to tail file, stat failed" error="stat /var/log/pods/kube-system_kube-sch eduler-docker-desktop_28dd1b1230fbe15350eb1b896ae9493d/kube-scheduler/1.log: no such file or directory" filename=/var/log/pods/kube-system_kube-scheduler-d ocker-desktop_28dd1b1230fbe15350eb1b896ae9493d/kube-scheduler/1.log level=error ts=2020-08-15T04:05:58.51975884Z caller=filetarget.go:283 msg="failed to tail file, stat failed" error="stat /var/log/pods/kube-system_kube-con troller-manager-docker-desktop_9076958db0c2cc26f76def745bfc1928/kube-controller-manager/2.log: no such file or directory" filename=/var/log/pods/kube-syste m_kube-controller-manager-docker-desktop_9076958db0c2cc26f76def745bfc1928/kube-controller-manager/2.log level=error ts=2020-08-15T04:05:58.53574224Z caller=filetarget.go:283 msg="failed to tail file, stat failed" error="stat /var/log/pods/kube-system_kube-con troller-manager-docker-desktop_9076958db0c2cc26f76def745bfc1928/kube-controller-manager/1.log: no such file or directory" filename=/var/log/pods/kube-syste m_kube-controller-manager-docker-desktop_9076958db0c2cc26f76def745bfc1928/kube-controller-manager/1.log

matti

comment created time in 2 months

pull request commentgrafana/tanka

Fix issue #329: Windows indefinite loop

Ah yes, good one! Code updated with filepath.VolumeName

c0ffeec0der

comment created time in 2 months

push eventc0ffeec0der/tanka

c0ffeec0der

commit sha adadd5bb329ea12153f9b17d64a2ab95716e314a

Fix issue #329 Problem: Indefinite loop in function findRoot Root cause: Root path only consider linux "/" Solution: Detect os at runtime and decide root path. Windows "C:\\", others "/"

view details

push time in 2 months

push eventc0ffeec0der/tanka

c0ffeec0der

commit sha a9356936c8e777c82b40e003cb5cc030582d3b96

Fix issue #329 Problem: Indefinite loop in function findRoot Root cause: Root path only consider linux "/" Solution: Detect os at runtime and decide root path. Windows "C:\\", others "/"

view details

c0ffeec0der

commit sha ee2a20088887c6c405d19ea44ccabc27f0db9c2b

Fix issue #329 Problem: Indefinite loop in function findRoot Root cause: Root path only consider linux "/" Solution: Detect os at runtime and decide root path. Windows "C:\\", others "/"

view details

c0ffeec0der

commit sha 8300f911f4a1ea7ff84ffcfb043e46c5ffed0b25

Fix issue #329 Problem: Indefinite loop in function findRoot Root cause: Root path only consider linux "/" Solution: Detect os at runtime and decide root path. Windows "C:\\", others "/"

view details

push time in 2 months

PR opened grafana/tanka

Fix issue #329: Windows indefinite loop

Problem: Indefinite loop in function findRoot

Root cause: Root path only consider linux "/"

Solution: Detect os at runtime and decide root path. Windows "C:\", others "/"

+11 -2

0 comment

1 changed file

pr created time in 2 months

create barnchc0ffeec0der/tanka

branch : fix/windowsNeverEndingLoop

created branch time in 2 months

fork c0ffeec0der/tanka

Flexible, reusable and concise configuration for Kubernetes

https://tanka.dev

fork in 2 months

issue commentgrafana/tanka

Windows: Tanka not running on windows

The environment is windows 10 version 2004

I was following the instructions on https://grafana.com/docs/loki/latest/installation/tanka/

When reaching step :

  • Run "tk show environments/loki" to see the manifests that will be deployed to the cluster.

The process keep running and does not end

c0ffeec0der

comment created time in 2 months

issue openedgrafana/tanka

Windows: Tanka not running on windows

When running "tk show environments/loki"

The process would keep running and not end

created time in 2 months

PR opened scylladb/scylla-operator

Propose resource request update for scylla manager agent sidecar

Propose resource request update for scylla manager agent sidecar starting at 5 percent

Rationale: If the DB is pinned to max 1 CPU per pod, with scylla manager agent sidecar also at 1 CPU is quite a lot

Could we start at: cpu 50m, memory 10M; and max out to cpu 1, memory 200M? This would be beneficial for developer testing and k8s will scale it vertically for production

Description of your changes: Update resource.go for scylla manager agent

Which issue is resolved by this Pull Request: Resolves # None

Checklist:

  • [ ] Documentation has been updated, if necessary.
  • [ ] Image has been built (make docker-build) on the last commit.
+2 -2

0 comment

1 changed file

pr created time in 3 months

create barnchc0ffeec0der/scylla-operator

branch : scylla-manager-agent_resource

created branch time in 3 months

fork c0ffeec0der/scylla-operator

Kubernetes operator for Scylla

fork in 3 months

fork c0ffeec0der/kustomize

Customization of kubernetes YAML configurations

fork in 3 months

more