profile
viewpoint
If you are wondering where the data of this site comes from, please visit https://api.github.com/users/sandeepsukhani/events. GitMemory does not store any data, but only uses NGINX to cache data for a period of time. The idea behind GitMemory is simply to give users a better reading experience.
Sandeep Sukhani sandeepsukhani @grafana Pune, Maharashtra, India Go & Python Developer

sandeepsukhani/devp2p-example 3

Go program to connect two machines using devp2p, a p2p protocol by Ethereum

grafana/bigtable-backup 1

Helps with creating and restoring Bigtable backups

sandeepsukhani/Colloqmote 0

Mobile assisted user interface for desktop navigation and gaming

sandeepsukhani/Competitive-programming 0

Some of my solutions to competitive programming problems

sandeepsukhani/cortex 0

A multitenant, horizontally scalable Prometheus as a Service

sandeepsukhani/dcrd 0

Decred daemon in Go (golang).

sandeepsukhani/find-max-number 0

A simple gRPC project in Go to find max number from stream of numbers

sandeepsukhani/go-github 0

Go library for accessing the GitHub API

sandeepsukhani/go-graphql-tools 0

Generator GraphQL schema from Go-structs

push eventsandeepsukhani/loki

Sandeep Sukhani

commit sha b8f928cb614f220e112a780c01b2aed0658dba21

lint

view details

push time in a day

PR opened grafana/loki

add logfmt promtail stage to be able to extract data from logfmt formatted log

<!-- Thanks for sending a pull request! Before submitting:

  1. Read our CONTRIBUTING.md guide
  2. Name your PR as <Feature Area>: Describe your change. a. Do not end the title with punctuation. It will be added in the changelog. b. Start with an imperative verb. Example: Fix the latency between System A and System B. c. Use sentence case, not title case. d. Use a complete phrase or sentence. The PR title will appear in a changelog, so help other people understand what your change will be.
  3. Rebase your PR if it gets out of sync with master
  4. If changing the Helm chart, please ensure the chart version is increased per semantic versioning (https://semver.org) -->

What this PR does / why we need it: It would be good to have a logfmt stage, the same as json stage, for extracting data from a log line. It works the same as json stage where the user can specify the mapping for extracted data to define which kvs to extract and set a different key than the one in logs for the extracted data.

Special notes for your reviewer: I would open a flow up PR for adding documentation.

Checklist

  • [x] Tests updated
+466 -0

0 comment

3 changed files

pr created time in a day

create barnchsandeepsukhani/loki

branch : logfmt-promtail-stage

created branch time in a day

PR closed grafana/loki

Docs: Fix numbering size/XS
+3 -3

3 comments

1 changed file

khaledismaeel

pr closed time in 2 days

pull request commentgrafana/loki

Docs: Fix numbering

Thanks @khaledismaeel for the PR! I am afraid we need to close this in favour of #4320.

khaledismaeel

comment created time in 2 days

push eventgrafana/loki

lingenavd

commit sha d388472475ce410dfb0f6da3086a337111ce139b

Update _index.md (#4324)

view details

push time in 2 days

PR merged grafana/loki

Docs: Update _index.md to add value boltdb-shipper for the key store size/XS

see: https://community.grafana.com/t/boltdb-shipper-store-configuration-dependant-on-boltdb-entries/53260

<!-- Thanks for sending a pull request! Before submitting:

  1. Read our CONTRIBUTING.md guide
  2. Name your PR as <Feature Area>: Describe your change. a. Do not end the title with punctuation. It will be added in the changelog. b. Start with an imperative verb. Example: Fix the latency between System A and System B. c. Use sentence case, not title case. d. Use a complete phrase or sentence. The PR title will appear in a changelog, so help other people understand what your change will be.
  3. Rebase your PR if it gets out of sync with master
  4. If changing the Helm chart, please ensure the chart version is increased per semantic versioning (https://semver.org) -->

What this PR does / why we need it: see: https://community.grafana.com/t/boltdb-shipper-store-configuration-dependant-on-boltdb-entries/53260

Which issue(s) this PR fixes: n.a.

Special notes for your reviewer:

Checklist

  • [ ] Documentation added
  • [ ] Tests updated
+1 -1

1 comment

1 changed file

lingenavd

pr closed time in 2 days

PullRequestReviewEvent

pull request commentgrafana/loki

Chore: Copy logging and tenant code from Cortex

Thank you so much @aknuds1 for all the efforts you have been putting into these PRs.

The PR just copies cortex/pkg/util/log to pkg/logutil (named this way to avoid clashing with go-kit/kit/log) I think keeping it under loki/pkg/util/log would have been better for 2 reasons:

  1. Avoid updating usages in the code which would have resulted in a smaller PR.
  2. Avoid having yet another util package at the top.

What do you think, should we go with that? I don't mind doing that change if you want.

aknuds1

comment created time in 2 days

push eventgrafana/loki

Karen Miller

commit sha be4cebc516a7f9039b9e81676377ce7a69cbeb21

Docs: improve spelling, grammar, and formatting. (#4320) - section titled "Get logs into Loki"

view details

push time in 2 days

PR merged grafana/loki

Docs: improve spelling, grammar, and formatting. size/L
  • section titled "Get logs into Loki"

This PR replaces (and does more needed corrections) than https://github.com/grafana/loki/pull/4314. Thanks @khaledismaeel for the original PR.

+61 -61

0 comment

1 changed file

KMiller-Grafana

pr closed time in 2 days

PullRequestReviewEvent

PR closed grafana/loki

Reviewers
reduce number of list calls on shared object store when using boltdb-shipper keepalive size/L

<!-- Thanks for sending a pull request! Before submitting:

  1. Read our CONTRIBUTING.md guide
  2. Name your PR as <Feature Area>: Describe your change
  3. Rebase your PR if it gets out of sync with master
  4. If changing the Helm chart, please ensure the chart version is increased per semantic versioning (https://semver.org) -->

What this PR does / why we need it: This PR contains the same code from PR #3283 but was reverted by PR #3396 due to a bug in the code causing multiple goroutines to attempt to open the same boltdb files. The bug was in caching layer which was not cleared before being rebuilt which means subsequent list calls would add the same filenames again to the cache if they still exist. This PR just fixes that bug and adds a test case for it.

Checklist

  • [x] Tests updated
+223 -5

3 comments

4 changed files

sandeepsukhani

pr closed time in a month

pull request commentgrafana/loki

reduce number of list calls on shared object store when using boltdb-shipper

Going to close this in favour of another PR that I am going to work on for per user index. The code has anyways diverged due to PR #4128

sandeepsukhani

comment created time in a month

push eventgrafana/loki

Oliver van Porten

commit sha 3d571f8bce19933afaf8f786f53051d521c3119e

Update docker.md (#4184) The path to promtail-docker-config.yaml was not correct. Updated to the correct path.

view details

push time in a month

PR merged grafana/loki

Reviewers
Update docker.md size/XS

The path to promtail-docker-config.yaml was not correct. Updated to the correct path.

<!-- Thanks for sending a pull request! Before submitting:

  1. Read our CONTRIBUTING.md guide
  2. Name your PR as <Feature Area>: Describe your change. a. Do not end the title with punctuation. It will be added in the changelog. b. Start with an imperative verb. Example: Fix the latency between System A and System B. c. Use sentence case, not title case. d. Use a complete phrase or sentence. The PR title will appear in a changelog, so help other people understand what your change will be.
  3. Rebase your PR if it gets out of sync with master
  4. If changing the Helm chart, please ensure the chart version is increased per semantic versioning (https://semver.org) -->

What this PR does / why we need it:

Which issue(s) this PR fixes: Fixes #<issue number>

Special notes for your reviewer:

Checklist

  • [ ] Documentation added
  • [ ] Tests updated
+2 -2

1 comment

1 changed file

mcdeck

pr closed time in a month

PullRequestReviewEvent

push eventgrafana/loki

Sandeep Sukhani

commit sha 29b07d190de2edab4fc03333e3e224d824b5a677

Retention speedup (#4172) * drop the multi table chunk entries from index when the whole table is out of retention * speed up cleaning of deleted series from index * mark partially deleted chunk for deletion when all its tables are processed

view details

push time in a month

PR merged grafana/loki

Reviewers
Retention speedup size/XL

<!-- Thanks for sending a pull request! Before submitting:

  1. Read our CONTRIBUTING.md guide
  2. Name your PR as <Feature Area>: Describe your change. a. Do not end the title with punctuation. It will be added in the changelog. b. Start with an imperative verb. Example: Fix the latency between System A and System B. c. Use sentence case, not title case. d. Use a complete phrase or sentence. The PR title will appear in a changelog, so help other people understand what your change will be.
  3. Rebase your PR if it gets out of sync with master
  4. If changing the Helm chart, please ensure the chart version is increased per semantic versioning (https://semver.org) -->

What this PR does / why we need it: It contains the following changes to speed up the mark phase of retention:

  1. We delete a chunk only when the whole chunk is out of retention. If a chunk spans multiple tables, we will retain its entry in all those tables until the whole chunk expires, even if the table is not expected to have it based on retention config. This PR changes the approach to drop the chunk index entry early by checking the interval's end time for which a table is expected to have chunks indexed.

  2. We clean up the series by iterating over the index entries to find which ones should be deleted. Because of the index format, we need to go through all the label entries(including labels for unrelated series) for the user. This PR changes the code to build the index keys using GetCacheKeysAndLabelWriteEntries, which can be used to delete the index entries expected to be deleted efficiently.

  3. When a chunk is deleted partially, we mark it for deletion. If a chunk spans multiple tables, we will mark it for deletion when the first table that indexes it is processed. If, for some reason, the compactor goes down or delays applying retention to the next tables, the compactor would fail to find the source chunk(which is needed for re-building a new chunk), which would fail the retention. This PR changes the code to mark source chunk for deletion when the table being processed is the last change that indexes it.

Checklist

  • [x] Tests updated
+478 -158

1 comment

12 changed files

sandeepsukhani

pr closed time in a month

PullRequestReviewEvent

push eventsandeepsukhani/loki

Owen Diehl

commit sha 6ac96478a7704a0d004888b9ca3e878c7d909d47

safely close nonOverlapping iterators (#4160)

view details

Sandeep Sukhani

commit sha 8ef3d5fee9653f8e542bd01cfb50cf05005b00e4

add a storage client for boltdb-shipper which would do all the object key management for storage operations (#4128) * add a storage client for boltdb-shipper which would do all the object key management for storage operations * changes suggested from PR review

view details

Owen Diehl

commit sha de5d2628f6c77f322f8232c7820707ec5d86176f

corrects query scheduler image (#4180)

view details

Jack Baldry

commit sha f38126473cd9e53fa5324ff0660a0142b15c45f9

Jsonnet: Add Grafana Enterprise Logs library (#4165) * feat: Add enterprise-logs Jsonnet library Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * fixup: Use correct image for GEL Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * fix: Separate k8s-libsonnet dependency Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * chore: Simplify Makefile Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * fix: Remove extra comma in jsonnetfile.json Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * fix: Remove unused prerequisite of test evaluation Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * doc: Remove unnecessary comments Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * fix: Remove storageClassName from PVCs Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * feat: Ensure all GEL Pods run as non-root user Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * feat: Configure memberlist for a more stable ring Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * style: Simplify gossip-ring Service ports configuration Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * chore: Update dependency lock Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * chore: Add convenience test target Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * fix: Disable tokengen Job by default Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * doc: Add TODO for utility functions Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * chore: Add tooling for developer testing Signed-off-by: Jack Baldry <jack.baldry@grafana.com>

view details

Sandeep Sukhani

commit sha 6b775ac31caaea6fe4bc71af3cd8ff47886275fd

allow setting tenant id for querying logs from loki (#4185)

view details

Owen Diehl

commit sha 944bbd16f569b0994b575ac10deaa869754c8eb0

fine tune grpc configs jsonnet (#4182)

view details

Karen Miller

commit sha 7ee89ed1d9dd57c2194bfe16dfcdf1cc2a237bd6

Docs: correct path to Promtail configuration file (#4175)

view details

Owen Diehl

commit sha eb5643be2679283c59b73022efffa420b6fd100f

initiate grpc health check always (#4181)

view details

Owen Diehl

commit sha 654b543e4e05027f13f9822629ecce063b5f0b15

updates unordered writes config docs (#4131) * updates unordered writes config docs * removes experimental flag * Apply suggestions from code review Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>

view details

Owen Diehl

commit sha 8f8f12713db4664afef5b29c08e9e2b1d2452a4f

add unordered writes to local config (#4149)

view details

Owen Diehl

commit sha 3f6f2fc717803da8436d30ee6bce97818a13ef56

remove unused code and reformat (#4155)

view details

Owen Diehl

commit sha f17b189a3f84a41ac06aefc1e7d5af8862912a08

Promote/ruler api (#4132) * unmark the ruler api as experimental * deprecation notice for experimental prefix * removes vendor changes, adds -ruler.enable-api (without experimental prefix) on our ruler.Config

view details

Sandeep Sukhani

commit sha 9634a6056d5a2849aad9024a7e0ccdc98feb9a48

drop the multi table chunk entries from index when the whole table is out of retention

view details

Sandeep Sukhani

commit sha 9b7da0e644051cde82524181525640438ef19bb1

speed up cleaning of deleted series from index

view details

Sandeep Sukhani

commit sha 91fd42c5767babe0013bd619a93cd86ea8bde051

mark partially deleted chunk for deletion when all its tables are processed

view details

push time in a month

pull request commentgrafana/loki

Retention speedup

Code wise this looks good to me. Unfortunately, I don't know enough to judge the logic.

Thanks @jeschkies for the review! I am going to merge this and let this go through our internal release cycle. I am also testing this in parallel with a dev cluster.

sandeepsukhani

comment created time in a month

PR opened grafana/loki

allow setting tenant id for querying logs from loki

<!-- Thanks for sending a pull request! Before submitting:

  1. Read our CONTRIBUTING.md guide
  2. Name your PR as <Feature Area>: Describe your change. a. Do not end the title with punctuation. It will be added in the changelog. b. Start with an imperative verb. Example: Fix the latency between System A and System B. c. Use sentence case, not title case. d. Use a complete phrase or sentence. The PR title will appear in a changelog, so help other people understand what your change will be.
  3. Rebase your PR if it gets out of sync with master
  4. If changing the Helm chart, please ensure the chart version is increased per semantic versioning (https://semver.org) -->

What this PR does / why we need it: Add support for specifying tenant id for canaries which would be used for querying loki with X-Scope-OrgID header.

+19 -5

0 comment

2 changed files

pr created time in a month

create barnchsandeepsukhani/loki

branch : loki-canary-tenant-id

created branch time in a month

Pull request review commentgrafana/loki

Docs: improve LogQL section

+---+title: Query examples+weight: 40+---++# Query examples++Some useful query examples here.++## Log Query examples++### Examples that filter on IP address ++- Return log lines that are not within a range of IPv4 addresses:++    ```logql+    {job_name="myapp"} != ip("192.168.4.5-192.168.4.20")+    ```+++- This example matches log lines with all IPv4 subnet values `192.168.4.5/16` except IP address `192.168.4.2`:++    ```logql+    {job_name="myapp"}+		| logfmt+		| addr = ip("192.168.4.5/16")+		| addr != ip("192.168.4.2")+    ```++### Examples that aid in security evaluation++- Extract the user and IP address of failed logins from Linux `/var/log/secure`++    ```logql+    {job="security"} +        |~ "Invalid user.*"+        |regexp "(^(?P<user>\\S+ {1,2}){8})"+        |regexp "(^(?P<ip>\\S+ {1,2}){10})"+        | line_format "IP = {{.ip}}\tUSER = {{.user}}"+    ```+   +- Get successful logins from Linux `/var/log/secure`++    ```logql+    {job="security"}+        != "grafana_com"+        |~ "session opened"

There is no regex pattern so we can just do contains filter

        |= "session opened"
KMiller-Grafana

comment created time in a month

PullRequestReviewEvent

Pull request review commentgrafana/loki

Docs: improve LogQL section

+---+title: Query examples+weight: 40+---++# Query examples++Some useful query examples here.++## Log Query examples++### Examples that filter on IP address ++- Return log lines that are not within a range of IPv4 addresses:++    ```logql+    {job_name="myapp"} != ip("192.168.4.5-192.168.4.20")+    ```+++- This example matches log lines with all IPv4 subnet values `192.168.4.5/16` except IP address `192.168.4.2`:++    ```logql+    {job_name="myapp"}+		| logfmt+		| addr = ip("192.168.4.5/16")+		| addr != ip("192.168.4.2")+    ```++### Examples that aid in security evaluation++- Extract the user and IP address of failed logins from Linux `/var/log/secure`++    ```logql+    {job="security"} +        |~ "Invalid user.*"+        |regexp "(^(?P<user>\\S+ {1,2}){8})"+        |regexp "(^(?P<ip>\\S+ {1,2}){10})"+        | line_format "IP = {{.ip}}\tUSER = {{.user}}"+    ```+   +- Get successful logins from Linux `/var/log/secure`++    ```logql+    {job="security"}+        != "grafana_com"+        |~ "session opened"+        != "sudo: "+        |regexp "(^(?P<user>\\S+ {1,2}){11})"

Same as above

KMiller-Grafana

comment created time in a month

Pull request review commentgrafana/loki

Docs: improve LogQL section

+---+title: Query examples+weight: 40+---++# Query examples++Some useful query examples here.++## Log Query examples++### Examples that filter on IP address ++- Return log lines that are not within a range of IPv4 addresses:++    ```logql+    {job_name="myapp"} != ip("192.168.4.5-192.168.4.20")+    ```+++- This example matches log lines with all IPv4 subnet values `192.168.4.5/16` except IP address `192.168.4.2`:++    ```logql+    {job_name="myapp"}+		| logfmt+		| addr = ip("192.168.4.5/16")+		| addr != ip("192.168.4.2")+    ```++### Examples that aid in security evaluation++- Extract the user and IP address of failed logins from Linux `/var/log/secure`++    ```logql+    {job="security"} +        |~ "Invalid user.*"+        |regexp "(^(?P<user>\\S+ {1,2}){8})"+        |regexp "(^(?P<ip>\\S+ {1,2}){10})"

Just a nit: I like putting a space after the pipe. You have added it for some of them. I think we should add them here too?

        | regexp "(^(?P<user>\\S+ {1,2}){8})"
        | regexp "(^(?P<ip>\\S+ {1,2}){10})"
KMiller-Grafana

comment created time in a month

PullRequestReviewEvent

push eventgrafana/loki

Sandeep Sukhani

commit sha 8ef3d5fee9653f8e542bd01cfb50cf05005b00e4

add a storage client for boltdb-shipper which would do all the object key management for storage operations (#4128) * add a storage client for boltdb-shipper which would do all the object key management for storage operations * changes suggested from PR review

view details

push time in a month