v1.20.0 seems to be only slightly slower than c9a92552385e742555fe341f813e6beba93defae, which I was using before (not dramatically slower like just after #699).

With v1.20.0 I'm seeing a lot of strange failures that I can't reliably reproduce like WARN [runner] Can't run linter goanalysis_metalinter: fact_purity: failed prerequisites: buildssa@<redacted> [<redacted>]. Sometimes one run will fail that way, and the next run, with no changes to the source code, will work fine.

@FrancoisPoinsot: Trying to set kafkaConfig.Version to V0_10_1_0 gives me:

kafka: invalid configuration (consumer groups require Version to be >= V0_10_2_0)

Upgrading kafka may well be the answer. I've been using the shopify/kafka all-in-one Docker container for this testing use case. I'll need to look around for a newer container image.

@FrancoisPoinsot: It seems like the version of kafka-consumer-groups that ships with the very old version of kafka I use for test automation (0.10.1.0) doesn't support --members or --verbose. I tried running the command several times without those options while running my test case, and got some strange output:

root@7fbf113a95c5:/# /opt/kafka_2.11-0.10.1.0/bin/kafka-consumer-groups.sh --bootstrap-server 127.0.0.1:9092 --group aaron_postgres_log_consumer_group --describe
Error while executing consumer group command Error reading field 'version': java.nio.BufferUnderflowException
org.apache.kafka.common.protocol.types.SchemaException: Error reading field 'version': java.nio.BufferUnderflowException
        at org.apache.kafka.common.protocol.types.Schema.read(Schema.java:73)
        at org.apache.kafka.clients.consumer.internals.ConsumerProtocol.deserializeAssignment(ConsumerProtocol.java:106)
        at kafka.admin.AdminClient$$anonfun$describeConsumerGroup$1.apply(AdminClient.scala:156)
        at kafka.admin.AdminClient$$anonfun$describeConsumerGroup$1.apply(AdminClient.scala:155)
        at scala.collection.immutable.List.map(List.scala:277)
        at kafka.admin.AdminClient.describeConsumerGroup(AdminClient.scala:155)
        at kafka.admin.ConsumerGroupCommand$KafkaConsumerGroupService.describeGroup(ConsumerGroupCommand.scala:308)
        at kafka.admin.ConsumerGroupCommand$ConsumerGroupService$class.describe(ConsumerGroupCommand.scala:89)
        at kafka.admin.ConsumerGroupCommand$KafkaConsumerGroupService.describe(ConsumerGroupCommand.scala:296)
        at kafka.admin.ConsumerGroupCommand$.main(ConsumerGroupCommand.scala:68)
        at kafka.admin.ConsumerGroupCommand.main(ConsumerGroupCommand.scala)

root@7fbf113a95c5:/# /opt/kafka_2.11-0.10.1.0/bin/kafka-consumer-groups.sh --bootstrap-server 127.0.0.1:9092 --group aaron_postgres_log_consumer_group --describe
GROUP                          TOPIC                          PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             OWNER
aaron_postgres_log_consumer_group log_stream_aaron               0          17494           17494           0               176c1760-9269-4c55-b21d-265985f4a303_/10.0.135.129
root@7fbf113a95c5:/# /opt/kafka_2.11-0.10.1.0/bin/kafka-consumer-groups.sh --bootstrap-server 127.0.0.1:9092 --group aaron_postgres_log_consumer_group --describe
Consumer group `aaron_postgres_log_consumer_group` is rebalancing.
root@7fbf113a95c5:/# /opt/kafka_2.11-0.10.1.0/bin/kafka-consumer-groups.sh --bootstrap-server 127.0.0.1:9092 --group aaron_postgres_log_consumer_group --describe
Error while executing consumer group command Error reading field 'version': java.nio.BufferUnderflowException
org.apache.kafka.common.protocol.types.SchemaException: Error reading field 'version': java.nio.BufferUnderflowException
        at org.apache.kafka.common.protocol.types.Schema.read(Schema.java:73)
        at org.apache.kafka.clients.consumer.internals.ConsumerProtocol.deserializeAssignment(ConsumerProtocol.java:106)
        at kafka.admin.AdminClient$$anonfun$describeConsumerGroup$1.apply(AdminClient.scala:156)
        at kafka.admin.AdminClient$$anonfun$describeConsumerGroup$1.apply(AdminClient.scala:155)
        at scala.collection.immutable.List.map(List.scala:277)
        at kafka.admin.AdminClient.describeConsumerGroup(AdminClient.scala:155)
        at kafka.admin.ConsumerGroupCommand$KafkaConsumerGroupService.describeGroup(ConsumerGroupCommand.scala:308)
        at kafka.admin.ConsumerGroupCommand$ConsumerGroupService$class.describe(ConsumerGroupCommand.scala:89)
        at kafka.admin.ConsumerGroupCommand$KafkaConsumerGroupService.describe(ConsumerGroupCommand.scala:296)
        at kafka.admin.ConsumerGroupCommand$.main(ConsumerGroupCommand.scala:68)
        at kafka.admin.ConsumerGroupCommand.main(ConsumerGroupCommand.scala)

root@7fbf113a95c5:/# /opt/kafka_2.11-0.10.1.0/bin/kafka-consumer-groups.sh --bootstrap-server 127.0.0.1:9092 --group aaron_postgres_log_consumer_group --describe
Consumer group `aaron_postgres_log_consumer_group` is rebalancing.
root@7fbf113a95c5:/# /opt/kafka_2.11-0.10.1.0/bin/kafka-consumer-groups.sh --bootstrap-server 127.0.0.1:9092 --group aaron_postgres_log_consumer_group --describe
Consumer group `aaron_postgres_log_consumer_group` is rebalancing.
root@7fbf113a95c5:/# /opt/kafka_2.11-0.10.1.0/bin/kafka-consumer-groups.sh --bootstrap-server 127.0.0.1:9092 --group aaron_postgres_log_consumer_group --describe
Consumer group `aaron_postgres_log_consumer_group` is rebalancing.
root@7fbf113a95c5:/# /opt/kafka_2.11-0.10.1.0/bin/kafka-consumer-groups.sh --bootstrap-server 127.0.0.1:9092 --group aaron_postgres_log_consumer_group --describe
Consumer group `aaron_postgres_log_consumer_group` is rebalancing.
root@7fbf113a95c5:/# /opt/kafka_2.11-0.10.1.0/bin/kafka-consumer-groups.sh --bootstrap-server 127.0.0.1:9092 --group aaron_postgres_log_consumer_group --describe
Consumer group `aaron_postgres_log_consumer_group` is rebalancing.
root@7fbf113a95c5:/# /opt/kafka_2.11-0.10.1.0/bin/kafka-consumer-groups.sh --bootstrap-server 127.0.0.1:9092 --group aaron_postgres_log_consumer_group --describe
Consumer group `aaron_postgres_log_consumer_group` is rebalancing.
root@7fbf113a95c5:/# /opt/kafka_2.11-0.10.1.0/bin/kafka-consumer-groups.sh --bootstrap-server 127.0.0.1:9092 --group aaron_postgres_log_consumer_group --describe
Consumer group `aaron_postgres_log_consumer_group` is rebalancing.
root@7fbf113a95c5:/# /opt/kafka_2.11-0.10.1.0/bin/kafka-consumer-groups.sh --bootstrap-server 127.0.0.1:9092 --group aaron_postgres_log_consumer_group --describe
GROUP                          TOPIC                          PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             OWNER
aaron_postgres_log_consumer_group log_stream_aaron               0          17494           17494           0               535c443b-a05f-438b-aaa7-b741325b2cc1_/10.0.135.129

This is a run every few seconds over the course of about 20 seconds.

Versions

Please specify real version numbers or git SHAs, not just "Latest" since that changes fairly regularly.

Configuration

What configuration values are you using for Sarama and Kafka?

        kafkaConfig := sarama.NewConfig()
        kafkaConfig.Consumer.Return.Errors = true
        kafkaConfig.Consumer.Offsets.Initial = sarama.OffsetOldest
        kafkaConfig.Consumer.Offsets.Retention = 35 * 24 * time.Hour
        kafkaConfig.Version = sarama.V0_10_2_0

Problem Description

I am trying to migrate from sarama-cluster. When I start multiple consumers and use the same groupID on a topic that only has a single partition, I expected only one of the consumers to receive messages (I believe this was the behavior with sarama-cluster). Instead, both consumers appear to receive claims which cover the single partition. This means they both receive the full stream of messages.

I start the two consumers at the same time, on a fresh topic that is just starting to receive messages. The behavior I see is that both consumers receive the messages I produce. They each receive a claim with Partition() == 0 and InitialOffset() == -2.

Please let me know if my expectations are incorrect, or if I'm doing something wrong.

#699 seems to have made things a lot slower.

Before:

real    0m26.390s
user    1m9.332s
sys     0m45.512s

After:

real    1m5.048s
user    3m11.196s
sys     3m29.260s

The first run with the new version was even slower (at least a few minutes), but I didn't manage to capture the timing. Also, memory use is much higher than before. On the first run, I saw it using 15 GB, up from 3.2 GB before.

Is there any way to turn offf the facts caching?

I'm not sure if it's because of the same change, but peak memory use also seems much lower now. I'm seeing about 3.2 GB, instead of 9 GB before I upgraded.

Do you run only govet in golangci-lint? why if yes?

Normally I run many linters, but performance was a lot slower than gometalinter, and I tracked down the difference to go vet.

Here are some benchmarks showing what I mean:

$ time go vet ./...

real    0m17.862s
user    1m43.516s
sys     0m22.268s
$ time go vet ./...

real    0m5.302s
user    0m21.332s
sys     0m11.820s
$ time golangci-lint run

real    0m32.683s
user    1m7.324s
sys     2m39.892s
$ time golangci-lint run

real    0m34.040s
user    1m9.764s
sys     2m47.400s

$ go version
go version go1.13 linux/amd64
$ golangci-lint --version
golangci-lint has version v1.18.1-0.20190911144626-bf67a3eef3ec built from (unknown, mod sum: "h1:2hgPZ5e/Xx5qznnizFOYiiau5hz9rwpnE0Se0d3wasU=") on (unknown)

It's interesting to me that the "system" time is so high.

Here is my .golangci.yml:

run:
  skip-dirs:
    - lookups
  modules-download-mode: vendor
  deadline: 20m

linters-settings:
  govet:
    check-shadowing: false
    settings:
      printf:
        funcs:
          - (github.com/blueowldev/backend/logger.RequestLog).Infof
          - (github.com/blueowldev/backend/logger.RequestLog).Tracef
          - (github.com/blueowldev/backend/logger.RequestLog).Errorf
          - (github.com/blueowldev/backend/logger.RequestLog).Alertf

linters:
  # inverted configuration with `enable-all` and `disable` is not scalable during updates of golangci-lint
  disable-all: true
  enable:
    - govet
issues:
  max-same-issues: 0
  exclude-rules:
    - text: "weak cryptographic primitive"
      linters:
        - gosec
    - text: "Rollback` is not checked"
      linters:
        - errcheck
    - text: "StopAll` is not checked"
      linters:
        - errcheck
    # scopelint breaks on this file for some reason
    - path: billing/billcore/generate.go
      linters:
        - scopelint
    # SQL injection is not a concern inside migrations or tests
    - path: migrate.go
      linters:
        - gosec
    - path: _test.go
      linters:
        - gosec

I've reduced govet work time by 25% on this repo in #693.

This made a big difference!

$ time golangci-lint run
real    0m9.965s
user    0m22.556s
sys     0m30.876s

golangci-lint has version v1.18.1-0.20190914154818-c9a92552385e built from (unknown, mod sum: "h1:oiCHfLWXxKdZAHrNK0uOpiCkqfq/wESZUrNTAuf54To=") on (unknown)

Thank you for looking into this. I consider the issue resolved.

I'm experiencing this same problem with golangci-lint 1.18.0, but it seems like the use-installed-packages flag no longer exists. Is there a current solution to the problem?