profile
viewpoint
If you are wondering where the data of this site comes from, please visit https://api.github.com/users/owen-d/events. GitMemory does not store any data, but only uses NGINX to cache data for a period of time. The idea behind GitMemory is simply to give users a better reading experience.
Owen Diehl owen-d Washington, DC @grafana/loki maintainer. SWE @grafana.

push eventgrafana/loki

Owen Diehl

commit sha fc0f98b802405147b392c0256a89f086d06748e8

removes deprecated duplicate per stream rate limit fields (#4367)

view details

push time in 3 days

PR merged grafana/loki

removes deprecated duplicate per stream rate limit fields size/M

Followup to https://github.com/grafana/loki/pull/4236

+7 -34

0 comment

1 changed file

owen-d

pr closed time in 3 days

PR opened grafana/loki

removes deprecated duplicate per stream rate limit fields

Followup to https://github.com/grafana/loki/pull/4236

+7 -34

0 comment

1 changed file

pr created time in 3 days

create barnchowen-d/loki

branch : remove/deprecated-limit

created branch time in 3 days

Pull request review commentgrafana/loki

Rewrite lambda-promtail to use subscription filters.

+provider "aws" {+  region = "us-east-2"+}++data "aws_region" "current" {}++resource "aws_iam_role" "iam_for_lambda" {+  name = "iam_for_lambda"++  assume_role_policy = jsonencode({+    "Version": "2012-10-17",+    "Statement": [+      {+        "Action": "sts:AssumeRole",+        "Principal": {+          "Service": "lambda.amazonaws.com"+        },+        "Effect": "Allow",+      }+    ]+  })+}++resource "aws_iam_role_policy" "logs" {+  name = "lambda-logs"+  role = aws_iam_role.iam_for_lambda.name+  policy = jsonencode({+    "Statement" : [+      {+        "Action" : [+          "logs:CreateLogGroup",+          "logs:CreateLogStream",+          "logs:PutLogEvents",+        ],+        "Effect" : "Allow",+        "Resource" : "arn:aws:logs:*:*:*",+      }+    ]+  })+}++resource "aws_lambda_function" "lambda_promtail" {+  image_uri     = "your-image-ecr:latest"

This should be a variable

cstyan

comment created time in 3 days

PullRequestReviewEvent

Pull request review commentgrafana/loki

Rewrite lambda-promtail to use subscription filters.

 Parameters:     Default: 2  Resources:+  LambdaPromtailRole:+    Type: AWS::IAM::Role+    Properties:+      AssumeRolePolicyDocument:+        Version: '2012-10-17'+        Statement:+        - Effect: Allow+          Principal:+            Service:+            - lambda.amazonaws.com+          Action:+          - sts:AssumeRole+      Description: "Lambda Promtail Role"+      Policies:+      - PolicyName: logs+        PolicyDocument:+          Version: '2012-10-17'+          Statement:+          - Effect: Allow+            Action:+            - logs:CreateLogGroup+            - logs:CreateLogStream+            - logs:PutLogEvents+            Resource: arn:aws:logs:*:*:*+      RoleName: iam_for_lambda   LambdaPromtailFunction:-    Type: AWS::Serverless::Function+    Type: AWS::Lambda::Function     Properties:-      CodeUri: lambda-promtail/-      Handler: handler+      Code:+        ImageUri: "your-image-ecr:latest"       MemorySize: 128+      PackageType: Image       Timeout: 60-      EventInvokeConfig:-        MaximumRetryAttempts: 2-      Policies:-        - Statement:-            Effect: Allow-            Action:-              - 'logs:DescribeLogGroups'-              - 'logs:DescribeLogStreams'-              - 'logs:FilterLogEvents'-              - 'logs:GetLogEvents'-            Resource: '*'+      Role: !GetAtt LambdaPromtailRole.Arn       ReservedConcurrentExecutions: !Ref ReservedConcurrency       # # https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-lambda-function-vpcconfig.html       # VpcConfig:-      Runtime: go1.x-      Events:-        CatchAll:-          Type: CloudWatchLogs-          Properties:-            LogGroupName: !Ref LogGroup-            FilterPattern: ''       Environment: # More info about Env Vars: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#environment-object         Variables:-          PROMTAIL_ADDRESS: !Ref PromtailAddress-+          WRITE_ADDRESS: !Ref WriteAddress+  LambdaPromtailVersion:+    Type: AWS::Lambda::Version+    Properties:+      FunctionName: !Ref LambdaPromtailFunction+  LambdaPromtailPermissions:+    Type: AWS::Lambda::Permission+    Properties:+      Action: lambda:InvokeFunction+      FunctionName: !GetAtt LambdaPromtailFunction.Arn+      Principal: !Sub+        - logs.${Region}.amazonaws.com+        - { Region: !Ref "AWS::Region" }+  LambdaPromtailEventInvokeConfig:+    Type: AWS::Lambda::EventInvokeConfig+    Properties:+      FunctionName: !Ref LambdaPromtailFunction+      MaximumRetryAttempts: 2+      Qualifier: !GetAtt LambdaPromtailVersion.Version+  MainLambdaPromtailSubscriptionFilter:+    Type: AWS::Logs::SubscriptionFilter+    Properties:+      DestinationArn: !GetAtt LambdaPromtailFunction.Arn+      FilterPattern: ""+      LogGroupName: "/aws/lambda/some-lamda-log-group"

Again, let's use parameters.

cstyan

comment created time in 3 days

Pull request review commentgrafana/loki

Rewrite lambda-promtail to use subscription filters.

+provider "aws" {+  region = "us-east-2"+}++data "aws_region" "current" {}++resource "aws_iam_role" "iam_for_lambda" {+  name = "iam_for_lambda"++  assume_role_policy = jsonencode({+    "Version": "2012-10-17",+    "Statement": [+      {+        "Action": "sts:AssumeRole",+        "Principal": {+          "Service": "lambda.amazonaws.com"+        },+        "Effect": "Allow",+      }+    ]+  })+}++resource "aws_iam_role_policy" "logs" {+  name = "lambda-logs"+  role = aws_iam_role.iam_for_lambda.name+  policy = jsonencode({+    "Statement" : [+      {+        "Action" : [+          "logs:CreateLogGroup",+          "logs:CreateLogStream",+          "logs:PutLogEvents",+        ],+        "Effect" : "Allow",+        "Resource" : "arn:aws:logs:*:*:*",+      }+    ]+  })+}++resource "aws_lambda_function" "lambda_promtail" {+  image_uri     = "your-image-ecr:latest"+  function_name = "lambda_promtail"+  role          = aws_iam_role.iam_for_lambda.arn++  timeout      = 60+  memory_size  = 128+  package_type = "Image"++  # vpc_config {

Can we expose this in a conditional configuration rather than via comments?

cstyan

comment created time in 3 days

Pull request review commentgrafana/loki

Rewrite lambda-promtail to use subscription filters.

 Parameters:     Default: 2  Resources:+  LambdaPromtailRole:+    Type: AWS::IAM::Role+    Properties:+      AssumeRolePolicyDocument:+        Version: '2012-10-17'+        Statement:+        - Effect: Allow+          Principal:+            Service:+            - lambda.amazonaws.com+          Action:+          - sts:AssumeRole+      Description: "Lambda Promtail Role"+      Policies:+      - PolicyName: logs+        PolicyDocument:+          Version: '2012-10-17'+          Statement:+          - Effect: Allow+            Action:+            - logs:CreateLogGroup+            - logs:CreateLogStream+            - logs:PutLogEvents+            Resource: arn:aws:logs:*:*:*+      RoleName: iam_for_lambda   LambdaPromtailFunction:-    Type: AWS::Serverless::Function+    Type: AWS::Lambda::Function     Properties:-      CodeUri: lambda-promtail/-      Handler: handler+      Code:+        ImageUri: "your-image-ecr:latest"

We should be able to inject this via parameter.

cstyan

comment created time in 3 days

PullRequestReviewEvent

push eventgrafana/loki

Jason Giedymin

commit sha 6477970bc017182190511c6b8cc6e98675a72e44

fix(docs): spelling mistake (#4361) fix(docs): spelling mistake in logstash configuration doc

view details

push time in 4 days

PR merged grafana/loki

Reviewers
fix(docs): spelling mistake size/XS

fix(docs): spelling mistake in logstash configuration doc

<!-- Thanks for sending a pull request! Before submitting:

  1. Read our CONTRIBUTING.md guide
  2. Name your PR as <Feature Area>: Describe your change. a. Do not end the title with punctuation. It will be added in the changelog. b. Start with an imperative verb. Example: Fix the latency between System A and System B. c. Use sentence case, not title case. d. Use a complete phrase or sentence. The PR title will appear in a changelog, so help other people understand what your change will be.
  3. Rebase your PR if it gets out of sync with master
  4. If changing the Helm chart, please ensure the chart version is increased per semantic versioning (https://semver.org) -->

What this PR does / why we need it: Docs

Which issue(s) this PR fixes: Fixes #<issue number>

Special notes for your reviewer:

Checklist

  • [ ] Documentation added
  • [ ] Tests updated
+1 -1

1 comment

1 changed file

JasonGiedymin

pr closed time in 4 days

PullRequestReviewEvent

Pull request review commentgrafana/loki

Ruler: per-tenant WAL

 func engineQueryFunc(engine *logql.Engine, overrides RulesLimits, userID string)  // MultiTenantManagerAdapter will wrap a MultiTenantManager which validates loki rules func MultiTenantManagerAdapter(mgr ruler.MultiTenantManager) ruler.MultiTenantManager {-	return &MultiTenantManager{mgr}+	return &MultiTenantManager{inner: mgr} }  // MultiTenantManager wraps a cortex MultiTenantManager but validates loki rules type MultiTenantManager struct {-	ruler.MultiTenantManager+	inner ruler.MultiTenantManager+}++func (m *MultiTenantManager) SyncRuleGroups(ctx context.Context, ruleGroups map[string]rulespb.RuleGroupList) {+	m.inner.SyncRuleGroups(ctx, ruleGroups)+}++func (m *MultiTenantManager) GetRules(userID string) []*rules.Group {+	return m.inner.GetRules(userID)+}++func (m *MultiTenantManager) Stop() {+	if registry != nil {+		registry.stop()+	}++	m.inner.Stop() }  // ValidateRuleGroup validates a rulegroup func (m *MultiTenantManager) ValidateRuleGroup(grp rulefmt.RuleGroup) []error { 	return ValidateGroups(grp) } -func MemstoreTenantManager(-	cfg Config,-	engine *logql.Engine,-	overrides RulesLimits,-) ruler.ManagerFactory {-	var msMetrics *memstoreMetrics-	var rwMetrics *remoteWriteMetrics+// MetricsPrefix defines the prefix to use for all metrics in this package+const MetricsPrefix = "loki_ruler_wal_" -	return ruler.ManagerFactory(func(+var registry storageRegistry++func MultiTenantRuleManager(cfg Config, engine *logql.Engine, overrides RulesLimits, logger log.Logger, reg prometheus.Registerer) ruler.ManagerFactory {+	reg = prometheus.WrapRegistererWithPrefix(MetricsPrefix, reg)++	registry = newWALRegistry(log.With(logger, "storage", "registry"), reg, cfg, overrides)++	return func( 		ctx context.Context, 		userID string, 		notifier *notifier.Manager, 		logger log.Logger, 		reg prometheus.Registerer, 	) ruler.RulesManager {-		// We'll ignore the passed registerer and use the default registerer to avoid prefix issues and other weirdness.-		// This closure prevents re-registering.-		registerer := prometheus.DefaultRegisterer--		if msMetrics == nil {-			msMetrics = newMemstoreMetrics(registerer)-		}--		if rwMetrics == nil {-			rwMetrics = newRemoteWriteMetrics(registerer)-		}+		registry.configureTenantStorage(userID)  		logger = log.With(logger, "user", userID)-		queryFunc := engineQueryFunc(engine, overrides, userID)-		memStore := NewMemStore(userID, queryFunc, msMetrics, 5*time.Minute, log.With(logger, "subcomponent", "MemStore"))+		queryFunc := engineQueryFunc(engine, overrides, registry, userID)+		memStore := NewMemStore(userID, queryFunc, newMemstoreMetrics(reg), 5*time.Minute, log.With(logger, "subcomponent", "MemStore"))  		mgr := rules.NewManager(&rules.ManagerOptions{-			Appendable:      newAppendable(cfg, overrides, logger, userID, rwMetrics),+			Appendable:      registry, 			Queryable:       memStore,

We're still using the memStore here as queryable, which is fine, but we are now able to read from Prometheus to determine this with less hacks and remove the memstore implementation altogether. This can be in a later PR though. I opened https://github.com/grafana/loki/issues/4362

dannykopping

comment created time in 4 days

Pull request review commentgrafana/loki

Ruler: per-tenant WAL

 func engineQueryFunc(engine *logql.Engine, overrides RulesLimits, userID string)  // MultiTenantManagerAdapter will wrap a MultiTenantManager which validates loki rules func MultiTenantManagerAdapter(mgr ruler.MultiTenantManager) ruler.MultiTenantManager {-	return &MultiTenantManager{mgr}+	return &MultiTenantManager{inner: mgr} }  // MultiTenantManager wraps a cortex MultiTenantManager but validates loki rules type MultiTenantManager struct {-	ruler.MultiTenantManager+	inner ruler.MultiTenantManager+}++func (m *MultiTenantManager) SyncRuleGroups(ctx context.Context, ruleGroups map[string]rulespb.RuleGroupList) {+	m.inner.SyncRuleGroups(ctx, ruleGroups)+}++func (m *MultiTenantManager) GetRules(userID string) []*rules.Group {+	return m.inner.GetRules(userID)+}++func (m *MultiTenantManager) Stop() {+	if registry != nil {+		registry.stop()+	}++	m.inner.Stop() }  // ValidateRuleGroup validates a rulegroup func (m *MultiTenantManager) ValidateRuleGroup(grp rulefmt.RuleGroup) []error { 	return ValidateGroups(grp) } -func MemstoreTenantManager(-	cfg Config,-	engine *logql.Engine,-	overrides RulesLimits,-) ruler.ManagerFactory {-	var msMetrics *memstoreMetrics-	var rwMetrics *remoteWriteMetrics+// MetricsPrefix defines the prefix to use for all metrics in this package+const MetricsPrefix = "loki_ruler_wal_" -	return ruler.ManagerFactory(func(+var registry storageRegistry

I don't love having this globally defined, perhaps we can alter it.

dannykopping

comment created time in 4 days

PullRequestReviewEvent
PullRequestReviewEvent

issue commentgrafana/grafana

Loki: Log context doesn't work with queries that use parsers

This is difficult without us distinguishing between index labels and parsed labels in the API. I don't think this is actionable at the moment, but it's a good datapoint for us to consider as we improve Loki.

re: common labels, how would you determine these?

Dieterbe

comment created time in 4 days

PullRequestReviewEvent

Pull request review commentgrafana/loki

Improve error message for stream rate limit.

 func (s *stream) Push( 		// Check if this this should be rate limited. 		now := time.Now() 		if !s.limiter.AllowN(now, len(entries[i].Line)) {-			failedEntriesWithError = append(failedEntriesWithError, entryWithError{&entries[i], ErrStreamRateLimit})+			failedEntriesWithError = append(failedEntriesWithError, entryWithError{&entries[i], &validation.ErrStreamRateLimit{RateLimit: flagext.ByteSize(s.limiter.lim.Limit()), Labels: s.labelsString, Bytes: flagext.ByteSize(len(entries[i].Line))}})

Can we cache s.limiter.lim.Limit() so we aren't calling it for every rate limited entry?

cstyan

comment created time in 4 days

PullRequestReviewEvent

issue commentgrafana/loki

AWS Cloudwatch logs scrape config

Yeah @james-callahan, you're correct. I'm hoping we can make it a little easier with some additional tooling that @cstyan is looking into :hugs:

owen-d

comment created time in 4 days

PullRequestReviewEvent

Pull request review commentgrafana/loki

Chore: Use dskit/spanlogger

 func NewEngine(opts EngineOpts, q Querier, l Limits) *Engine {  // Query creates a new LogQL query. Instant/Range type is derived from the parameters. func (ng *Engine) Query(params Params) Query {+	if ng.logger == nil {+		panic("doof")+	}
aknuds1

comment created time in 4 days

Pull request review commentgrafana/loki

Chore: Use dskit/spanlogger

 type query struct {  // Exec Implements `Query`. It handles instrumentation & defers to Eval. func (q *query) Exec(ctx context.Context) (logqlmodel.Result, error) {-	log, ctx := spanlogger.New(ctx, "query.Exec")+	if q.logger == nil {+		panic("oof")+	}
aknuds1

comment created time in 4 days

Pull request review commentgrafana/loki

Chore: Use dskit/spanlogger

 func (opts *EngineOpts) applyDefault() {  // Engine is the LogQL engine. type Engine struct {+	logger    log.Logger 	timeout   time.Duration 	evaluator Evaluator 	limits    Limits }  // NewEngine creates a new LogQL Engine.-func NewEngine(opts EngineOpts, q Querier, l Limits) *Engine {+func NewEngine(opts EngineOpts, q Querier, l Limits, logger log.Logger) *Engine { 	opts.applyDefault()
	opts.applyDefault()
	if logger == nil {
		logger = log.NewNopLogger()
	}
aknuds1

comment created time in 4 days

PullRequestReviewEvent
PullRequestReviewEvent
PullRequestReviewEvent