profile
viewpoint
If you are wondering where the data of this site comes from, please visit https://api.github.com/users/dajac/events. GitMemory does not store any data, but only uses NGINX to cache data for a period of time. The idea behind GitMemory is simply to give users a better reading experience.
David Jacot dajac @confluentinc Switzerland Software Engineer at Confluent, Apache Kafka Committer.

dajac/kfn 8

KFn - Function as a Service (FaaS) for Apache Kafka

dajac/react 8

Reactive programming for Clojure.

dajac/check-graphite 2

nagios check for graphite metrics

dajac/aleph 1

asynchronous communication for clojure

dajac/hector 1

a high level client for cassandra

dajac/lamina 1

event-driven workflows for clojure

dajac/snacktory 1

Readability clone in Java for Jetslide

dajac/cascalog 0

Data processing on Hadoop without the hassle.

dajac/clj-statsd 0

simple client library to interface with statsd

Pull request review commentapache/kafka

KAFKA-13111: Re-evaluate Fetch Sessions when using topic IDs

 class CachedPartition(val topic: String,     mustRespond   } -  override def hashCode: Int = Objects.hash(new TopicPartition(topic, partition), topicId)+  override def hashCode: Int = if (topicId != Uuid.ZERO_UUID) (31 * partition) + topicId.hashCode else

Could we add a scaladoc for this method which explains what we do and why?

jolshan

comment created time in 12 hours

Pull request review commentapache/kafka

KAFKA-13111: Re-evaluate Fetch Sessions when using topic IDs

 class SessionlessFetchContext(val fetchData: util.Map[TopicPartition, FetchReque   * @param reqMetadata        The request metadata.   * @param fetchData          The partition data from the fetch request.   * @param usesTopicIds       True if this session should use topic IDs.-  * @param topicIds           The map from topic names to topic IDs.   * @param isFromFollower     True if this fetch request came from a follower.   */ class FullFetchContext(private val time: Time,                        private val cache: FetchSessionCache,                        private val reqMetadata: JFetchMetadata,-                       private val fetchData: util.Map[TopicPartition, FetchRequest.PartitionData],+                       private val fetchData: util.Map[TopicIdPartition, FetchRequest.PartitionData],                        private val usesTopicIds: Boolean,-                       private val topicIds: util.Map[String, Uuid],                        private val isFromFollower: Boolean) extends FetchContext {-  override def getFetchOffset(part: TopicPartition): Option[Long] =+  override def getFetchOffset(part: TopicIdPartition): Option[Long] =     Option(fetchData.get(part)).map(_.fetchOffset) -  override def foreachPartition(fun: (TopicPartition, Uuid, FetchRequest.PartitionData) => Unit): Unit = {-    fetchData.forEach((tp, data) => fun(tp, topicIds.get(tp.topic), data))+  override def foreachPartition(fun: (TopicIdPartition, FetchRequest.PartitionData) => Unit): Unit = {+    fetchData.forEach((tp, data) => fun(tp, data))   }    override def getResponseSize(updates: FetchSession.RESP_MAP, versionId: Short): Int = {-    FetchResponse.sizeOf(versionId, updates.entrySet.iterator, topicIds)+    FetchResponse.sizeOf(versionId, updates.entrySet.iterator)   }    override def updateAndGenerateResponseData(updates: FetchSession.RESP_MAP): FetchResponse = {     var hasInconsistentTopicIds = false-    def createNewSession: (FetchSession.CACHE_MAP, FetchSession.TOPIC_ID_MAP) = {+    def createNewSession: FetchSession.CACHE_MAP = {       val cachedPartitions = new FetchSession.CACHE_MAP(updates.size)-      val sessionTopicIds = new util.HashMap[String, Uuid](updates.size)       updates.forEach { (part, respData) =>         if (respData.errorCode() == Errors.INCONSISTENT_TOPIC_ID.code()) {           info(s"Session encountered an inconsistent topic ID for topicPartition $part.")           hasInconsistentTopicIds = true         }         val reqData = fetchData.get(part)-        val id = topicIds.getOrDefault(part.topic(), Uuid.ZERO_UUID)-        cachedPartitions.mustAdd(new CachedPartition(part, id, reqData, respData))-        if (id != Uuid.ZERO_UUID)-          sessionTopicIds.put(part.topic, id)+        cachedPartitions.mustAdd(new CachedPartition(part.topicPartition, part.topicId, reqData, respData))       }-      (cachedPartitions, sessionTopicIds)+      cachedPartitions     }     val responseSessionId = cache.maybeCreateSession(time.milliseconds(), isFromFollower,         updates.size, usesTopicIds, () => createNewSession)     if (hasInconsistentTopicIds) {-      FetchResponse.of(Errors.INCONSISTENT_TOPIC_ID, 0, responseSessionId, new FetchSession.RESP_MAP, Collections.emptyMap())+      FetchResponse.of(Errors.INCONSISTENT_TOPIC_ID, 0, responseSessionId, new FetchSession.RESP_MAP)

Do we still need to return INCONSISTENT_TOPIC_ID a top level error? Fetcher prior to this change would need it, for sure. With this PR, we actually don't want the fetcher to treat it as a top level error but rather as a partition error. We need to think/discuss this a little more, I think.

jolshan

comment created time in 12 hours

Pull request review commentapache/kafka

KAFKA-13111: Re-evaluate Fetch Sessions when using topic IDs

 class FetchSession(val id: Int,    def metadata: JFetchMetadata = synchronized { new JFetchMetadata(id, epoch) } -  def getFetchOffset(topicPartition: TopicPartition): Option[Long] = synchronized {-    Option(partitionMap.find(new CachedPartition(topicPartition,-      sessionTopicIds.getOrDefault(topicPartition.topic(), Uuid.ZERO_UUID)))).map(_.fetchOffset)+  def getFetchOffset(topicIdPartition: TopicIdPartition): Option[Long] = synchronized {+    Option(partitionMap.find(new CachedPartition(topicIdPartition.topicPartition, topicIdPartition.topicId))).map(_.fetchOffset)   } -  type TL = util.ArrayList[TopicPartition]+  type TL = util.ArrayList[TopicIdPartition]    // Update the cached partition data based on the request.   def update(fetchData: FetchSession.REQ_MAP,-             toForget: util.List[TopicPartition],+             toForget: util.List[TopicIdPartition],              reqMetadata: JFetchMetadata,-             topicIds: util.Map[String, Uuid]): (TL, TL, TL, TL) = synchronized {+             usesTopicIds: Boolean): (TL, TL, TL) = synchronized {     val added = new TL     val updated = new TL     val removed = new TL-    val inconsistentTopicIds = new TL     fetchData.forEach { (topicPart, reqData) =>-      // Get the topic ID on the broker, if it is valid and the topic is new to the session, add its ID.-      // If the topic already existed, check that its ID is consistent.-      val id = topicIds.getOrDefault(topicPart.topic, Uuid.ZERO_UUID)-      val newCachedPart = new CachedPartition(topicPart, id, reqData)-      if (id != Uuid.ZERO_UUID) {-        val prevSessionTopicId = sessionTopicIds.putIfAbsent(topicPart.topic, id)-        if (prevSessionTopicId != null && prevSessionTopicId != id)-          inconsistentTopicIds.add(topicPart)-      }+      val newCachedPart = new CachedPartition(topicPart.topicPartition, topicPart.topicId, reqData)       val cachedPart = partitionMap.find(newCachedPart)       if (cachedPart == null) {         partitionMap.mustAdd(newCachedPart)         added.add(topicPart)       } else {         cachedPart.updateRequestParams(reqData)+        if (cachedPart.topic == null)+        // Update the topic name in place+          cachedPart.resolveUnknownName(topicPart.topicPartition.topic)

This might not be necessary if we won't resolve topic ids in the request in all cases (see my previous comment).

jolshan

comment created time in 12 hours

Pull request review commentapache/kafka

KAFKA-13111: Re-evaluate Fetch Sessions when using topic IDs

 class KafkaApis(val requestChannel: RequestChannel,       None     } -    val erroneous = mutable.ArrayBuffer[(TopicPartition, FetchResponseData.PartitionData)]()-    val interesting = mutable.ArrayBuffer[(TopicPartition, FetchRequest.PartitionData)]()+    val erroneous = mutable.ArrayBuffer[(TopicIdPartition, FetchResponseData.PartitionData)]()+    val interesting = mutable.ArrayBuffer[(TopicIdPartition, FetchRequest.PartitionData)]()     val sessionTopicIds = mutable.Map[String, Uuid]()     if (fetchRequest.isFromFollower) {       // The follower must have ClusterAction on ClusterResource in order to fetch partition data.       if (authHelper.authorize(request.context, CLUSTER_ACTION, CLUSTER, CLUSTER_NAME)) {-        fetchContext.foreachPartition { (topicPartition, topicId, data) =>-          sessionTopicIds.put(topicPartition.topic(), topicId)-          if (!metadataCache.contains(topicPartition))-            erroneous += topicPartition -> FetchResponse.partitionResponse(topicPartition.partition, Errors.UNKNOWN_TOPIC_OR_PARTITION)+        fetchContext.foreachPartition { (topicIdPartition, data) =>+          sessionTopicIds.put(topicIdPartition.topicPartition.topic, topicIdPartition.topicId)+          if (topicIdPartition.topicPartition.topic == null )+            erroneous += topicIdPartition -> FetchResponse.partitionResponse(topicIdPartition.topicPartition.partition, Errors.UNKNOWN_TOPIC_ID)

nit: Could we add an overload to partitionResponse which takes a TopicIdPartition? This would reduce the boiler plate code a bit here.

jolshan

comment created time in 12 hours

Pull request review commentapache/kafka

KAFKA-13111: Re-evaluate Fetch Sessions when using topic IDs

 class KafkaApis(val requestChannel: RequestChannel,       None     } -    val erroneous = mutable.ArrayBuffer[(TopicPartition, FetchResponseData.PartitionData)]()-    val interesting = mutable.ArrayBuffer[(TopicPartition, FetchRequest.PartitionData)]()+    val erroneous = mutable.ArrayBuffer[(TopicIdPartition, FetchResponseData.PartitionData)]()+    val interesting = mutable.ArrayBuffer[(TopicIdPartition, FetchRequest.PartitionData)]()     val sessionTopicIds = mutable.Map[String, Uuid]()     if (fetchRequest.isFromFollower) {       // The follower must have ClusterAction on ClusterResource in order to fetch partition data.       if (authHelper.authorize(request.context, CLUSTER_ACTION, CLUSTER, CLUSTER_NAME)) {-        fetchContext.foreachPartition { (topicPartition, topicId, data) =>-          sessionTopicIds.put(topicPartition.topic(), topicId)-          if (!metadataCache.contains(topicPartition))-            erroneous += topicPartition -> FetchResponse.partitionResponse(topicPartition.partition, Errors.UNKNOWN_TOPIC_OR_PARTITION)+        fetchContext.foreachPartition { (topicIdPartition, data) =>+          sessionTopicIds.put(topicIdPartition.topicPartition.topic, topicIdPartition.topicId)+          if (topicIdPartition.topicPartition.topic == null )+            erroneous += topicIdPartition -> FetchResponse.partitionResponse(topicIdPartition.topicPartition.partition, Errors.UNKNOWN_TOPIC_ID)+          else if (!metadataCache.contains(topicIdPartition.topicPartition))+            erroneous += topicIdPartition -> FetchResponse.partitionResponse(topicIdPartition.topicPartition.partition, Errors.UNKNOWN_TOPIC_OR_PARTITION)           else-            interesting += (topicPartition -> data)+            interesting += (topicIdPartition -> data)         }       } else {-        fetchContext.foreachPartition { (part, topicId, _) =>-          sessionTopicIds.put(part.topic(), topicId)-          erroneous += part -> FetchResponse.partitionResponse(part.partition, Errors.TOPIC_AUTHORIZATION_FAILED)+        fetchContext.foreachPartition { (topicIdPartition, _) =>+          sessionTopicIds.put(topicIdPartition.topicPartition.topic, topicIdPartition.topicId)+          erroneous += topicIdPartition -> FetchResponse.partitionResponse(topicIdPartition.topicPartition.partition, Errors.TOPIC_AUTHORIZATION_FAILED)         }       }     } else {       // Regular Kafka consumers need READ permission on each partition they are fetching.-      val partitionDatas = new mutable.ArrayBuffer[(TopicPartition, FetchRequest.PartitionData)]-      fetchContext.foreachPartition { (topicPartition, topicId, partitionData) =>-        partitionDatas += topicPartition -> partitionData-        sessionTopicIds.put(topicPartition.topic(), topicId)-      }-      val authorizedTopics = authHelper.filterByAuthorized(request.context, READ, TOPIC, partitionDatas)(_._1.topic)-      partitionDatas.foreach { case (topicPartition, data) =>-        if (!authorizedTopics.contains(topicPartition.topic))-          erroneous += topicPartition -> FetchResponse.partitionResponse(topicPartition.partition, Errors.TOPIC_AUTHORIZATION_FAILED)-        else if (!metadataCache.contains(topicPartition))-          erroneous += topicPartition -> FetchResponse.partitionResponse(topicPartition.partition, Errors.UNKNOWN_TOPIC_OR_PARTITION)+      val partitionDatas = new mutable.ArrayBuffer[(TopicIdPartition, FetchRequest.PartitionData)]+      fetchContext.foreachPartition { (topicIdPartition, partitionData) =>+        partitionDatas += topicIdPartition -> partitionData+        sessionTopicIds.put(topicIdPartition.topicPartition.topic(), topicIdPartition.topicId)

Could we direclty check if the topic name is null here and put the unresolved ones to erroneous? This would avoid the filter on the next line.

jolshan

comment created time in 12 hours

Pull request review commentapache/kafka

KAFKA-13111: Re-evaluate Fetch Sessions when using topic IDs

 class CachedPartition(val topic: String,         this.eq(that) ||           (that.canEqual(this) &&             this.partition.equals(that.partition) &&-            this.topic.equals(that.topic) &&-            this.topicId.equals(that.topicId))+            (if (this.topicId != Uuid.ZERO_UUID) this.topicId.equals(that.topicId)+            else this.topic.equals(that.topic)))

Side note here: I think that we should implement override def elementKeysAreEqual(that: Any): Boolean from the ImplicitLinkedHashCollection.Element interface to make it clear that we do this for comparing elements in the collections.

jolshan

comment created time in 12 hours

Pull request review commentapache/kafka

KAFKA-13111: Re-evaluate Fetch Sessions when using topic IDs

 class KafkaApis(val requestChannel: RequestChannel,          // Prepare fetch response from converted data         val response =-          FetchResponse.of(unconvertedFetchResponse.error, throttleTimeMs, unconvertedFetchResponse.sessionId, convertedData, sessionTopicIds.asJava)+          FetchResponse.of(unconvertedFetchResponse.error, throttleTimeMs, unconvertedFetchResponse.sessionId, convertedData)         // record the bytes out metrics only when the response is being sent         response.data().responses().forEach { topicResponse =>           topicResponse.partitions().forEach { data =>-            val tp = new TopicPartition(topicResponse.topic(), data.partitionIndex())-            brokerTopicStats.updateBytesOut(tp.topic, fetchRequest.isFromFollower, reassigningPartitions.contains(tp), FetchResponse.recordsSize(data))+            val tp = new TopicIdPartition(topicResponse.topicId, new TopicPartition(topicResponse.topic(), data.partitionIndex()))+            // If the topic name was not known, we will have no bytes out.+            if (tp.topicPartition.topic != null)

Should we create tp after this check? We could also create a TopicPartition as we don't really use TopicIdPartition for the metric.

jolshan

comment created time in 12 hours

Pull request review commentapache/kafka

KAFKA-13111: Re-evaluate Fetch Sessions when using topic IDs

 class KafkaApis(val requestChannel: RequestChannel,     val versionId = request.header.apiVersion     val clientId = request.header.clientId     val fetchRequest = request.body[FetchRequest]-    val (topicIds, topicNames) =+    val topicNames =       if (fetchRequest.version() >= 13)-        metadataCache.topicIdInfo()+        metadataCache.topicIdsToNames()       else-        (Collections.emptyMap[String, Uuid](), Collections.emptyMap[Uuid, String]())+        Collections.emptyMap[Uuid, String]() -    // If fetchData or forgottenTopics contain an unknown topic ID, return a top level error.-    var fetchData: util.Map[TopicPartition, FetchRequest.PartitionData] = null-    var forgottenTopics: util.List[TopicPartition] = null-    try {-      fetchData = fetchRequest.fetchData(topicNames)-      forgottenTopics = fetchRequest.forgottenTopics(topicNames)-    } catch {-      case e: UnknownTopicIdException => throw e-    }+    val fetchData = fetchRequest.fetchData(topicNames)+    val forgottenTopics = fetchRequest.forgottenTopics(topicNames)

When a session is used, resolving the topic ids is not really necessary here because we should already have the names in the session or we would resolve them later anyway. I wonder if it would be better to do this entirely in the fetchManager.newConext based on the context type. Have you considered something like this?

jolshan

comment created time in 12 hours

Pull request review commentapache/kafka

KAFKA-13111: Re-evaluate Fetch Sessions when using topic IDs

 class ReplicaFetcherThread(name: String,         Map.empty       }     } else {-      fetchResponse.responseData(fetchSessionHandler.sessionTopicNames, clientResponse.requestHeader().apiVersion()).asScala+      fetchResponse.responseData(fetchSessionHandler.sessionTopicNames, clientResponse.requestHeader().apiVersion()).asScala.map {

Not related to this line. Don't wee need to update the fetcher to handle the topic id errors at the partition level?

jolshan

comment created time in 12 hours

Pull request review commentapache/kafka

KAFKA-13111: Re-evaluate Fetch Sessions when using topic IDs

 class KafkaApis(val requestChannel: RequestChannel,       None     } -    val erroneous = mutable.ArrayBuffer[(TopicPartition, FetchResponseData.PartitionData)]()-    val interesting = mutable.ArrayBuffer[(TopicPartition, FetchRequest.PartitionData)]()+    val erroneous = mutable.ArrayBuffer[(TopicIdPartition, FetchResponseData.PartitionData)]()+    val interesting = mutable.ArrayBuffer[(TopicIdPartition, FetchRequest.PartitionData)]()     val sessionTopicIds = mutable.Map[String, Uuid]()     if (fetchRequest.isFromFollower) {       // The follower must have ClusterAction on ClusterResource in order to fetch partition data.       if (authHelper.authorize(request.context, CLUSTER_ACTION, CLUSTER, CLUSTER_NAME)) {-        fetchContext.foreachPartition { (topicPartition, topicId, data) =>-          sessionTopicIds.put(topicPartition.topic(), topicId)-          if (!metadataCache.contains(topicPartition))-            erroneous += topicPartition -> FetchResponse.partitionResponse(topicPartition.partition, Errors.UNKNOWN_TOPIC_OR_PARTITION)+        fetchContext.foreachPartition { (topicIdPartition, data) =>+          sessionTopicIds.put(topicIdPartition.topicPartition.topic, topicIdPartition.topicId)

Do we still need this sessionTopicIds mapping if we have the topic id in the topicIdPartition?

jolshan

comment created time in 12 hours

PullRequestReviewEvent
PullRequestReviewEvent

push eventapache/kafka

Justine Olshan

commit sha b76bcaf3a8f28a5ad553a9edd477c4d6d887f08a

KAFKA-13102: Topic IDs not propagated to metadata cache quickly enough for Fetch path (#11170) Before we used the metadata cache to determine whether or not to use topic IDs. Unfortunately, metadata cache updates with ZK controllers are in a separate request and may be too slow for the fetcher thread. This results in switching between topic names and topic IDs for topics that could just use IDs. This patch adds topic IDs to FetcherState created in LeaderAndIsr requests. It also supports updating this state for follower threads as soon as a LeaderAndIsr request provides a topic ID. We've opted to only update replica fetcher threads. AlterLogDir threads will use either topic name or topic ID depending on what was present when they were created. Reviewers: David Jacot <djacot@confluent.io>

view details

push time in 18 hours

PR merged apache/kafka

KAFKA-13102: Topic IDs not propagated to metadata cache quickly enough for Fetch path

Before we used the metadata cache to determine whether or not to use topic IDs. Unfortunately, metadata cache updates with ZK controllers are in a separate request and may be too slow for the fetcher thread. This results in switching between topic names and topic IDs for topics that could just use IDs.

This change adds topic IDs to FetcherState created in LeaderAndIsr requests. It also supports updating this state for follower threads as soon as a LeaderAndIsr request provides a topic ID.

I've opted to only update replica fetcher threads. Alter Log Dir threads will use either topic name or topic ID depending on what was present when they were created.

Committer Checklist (excluded from commit message)

  • [ ] Verify design and implementation
  • [ ] Verify test coverage and CI build status
  • [ ] Verify documentation (including upgrade notes)
+398 -101

2 comments

13 changed files

jolshan

pr closed time in 18 hours

pull request commentapache/kafka

KAFKA-13102: Topic IDs not propagated to metadata cache quickly enough for Fetch path

Two tests failed in the last build:

Build / JDK 17 and Scala 2.13 / [1] Type=Raft, Name=testTopicPartition, Security=PLAINTEXT – kafka.admin.LeaderElectionCommandTest
1m 15s
Build / JDK 17 and Scala 2.13 / [1] Type=Raft, Name=testTopicPartition, Security=PLAINTEXT – kafka.admin.LeaderElectionCommandTest

I have seen similar failures in other PR so they are likely flaky at the moment. I have run them locally and they have passed.

jolshan

comment created time in 19 hours

PullRequestReviewEvent

Pull request review commentapache/kafka

KAFKA-13102: Topic IDs not propagated to metadata cache quickly enough for Fetch path

 class AbstractFetcherThreadTest {     fetcher.verifyLastFetchedEpoch(partition, Some(5))   } +  @Test+  def testMaybeUpdateTopicIds(): Unit = {+    val partition = new TopicPartition("topic1", 0)+    val fetcher = new MockFetcherThread++    // Start with no topic IDs+    fetcher.setReplicaState(partition, MockFetcherThread.PartitionState(leaderEpoch = 0))+    fetcher.addPartitions(Map(partition -> initialFetchState(None, 0L, leaderEpoch = 0)))+++    def verifyFetchState(fetchState: Option[PartitionFetchState], expectedTopicId: Option[Uuid]): Unit = {+      assertTrue(fetchState.isDefined)+      assertEquals(expectedTopicId, fetchState.get.topicId)+    }++    verifyFetchState(fetcher.fetchState(partition), None)++    // Add topic ID+    fetcher.maybeUpdateTopicIds(Set(partition), topicName => topicIds.get(topicName))

We could add tests for the ReplicaManager which verifies that the topic id is updated and propagated to the fetcher manager. Is it what you are thinking about?

Regarding this particular test, what would you add? If the topic id is zero, we would just set it. As you said, the logic which prevents this from happening is before.

jolshan

comment created time in a day

PullRequestReviewEvent

Pull request review commentapache/kafka

KAFKA-13102: Topic IDs not propagated to metadata cache quickly enough for Fetch path

 class ReplicaManagerTest {       ClientQuotasImage.EMPTY     )   }++  def assertFetcherHasTopicId[T <: AbstractFetcherThread](manager: AbstractFetcherManager[T],+                                                          tp: TopicPartition,+                                                          expectedTopicId: Option[Uuid]): Unit = {+    val fetchState = manager.getFetcher(tp).flatMap(_.fetchState(tp))+    assertTrue(fetchState.isDefined)+    assertEquals(expectedTopicId, fetchState.get.topicId)+  }++  @Test+  def testPartitionFetchStateUpdatesWithTopicIdAdded(): Unit = {+    val aliveBrokersIds = Seq(0, 1)+    val replicaManager = setupReplicaManagerWithMockedPurgatories(new MockTimer(time),+      brokerId = 0, aliveBrokersIds)+    try {+      val tp = new TopicPartition(topic, 0)+      val leaderAndIsr = new LeaderAndIsr(1, 0, aliveBrokersIds.toList, 0)++      val leaderAndIsrRequest1 = leaderAndIsrRequest(Uuid.ZERO_UUID, tp, aliveBrokersIds, leaderAndIsr)+      val leaderAndIsrResponse1 = replicaManager.becomeLeaderOrFollower(0, leaderAndIsrRequest1, (_, _) => ())+      assertEquals(Errors.NONE, leaderAndIsrResponse1.error)++      assertFetcherHasTopicId(replicaManager.replicaFetcherManager, tp, None)++      val leaderAndIsrRequest2 = leaderAndIsrRequest(topicId, tp, aliveBrokersIds, leaderAndIsr)+      val leaderAndIsrResponse2 = replicaManager.becomeLeaderOrFollower(0, leaderAndIsrRequest2, (_, _) => ())+      assertEquals(Errors.NONE, leaderAndIsrResponse2.error)++      assertFetcherHasTopicId(replicaManager.replicaFetcherManager, tp, Some(topicId))++    } finally {+      replicaManager.shutdown(checkpointHW = false)+    }+  }++  @ParameterizedTest+  @ValueSource(booleans = Array(true, false))+  def testReplicaAlterLogDirsWithAndWithoutIds(usesTopicIds: Boolean): Unit = {+    val version = if (usesTopicIds) LeaderAndIsrRequestData.HIGHEST_SUPPORTED_VERSION else 4.toShort+    val topicIdRaw = if (usesTopicIds) topicId else Uuid.ZERO_UUID

Ah.. It is because topicId is also used in the statement. You could indeed use this.topicId there.

jolshan

comment created time in a day

PullRequestReviewEvent

Pull request review commentapache/kafka

KAFKA-13102: Topic IDs not propagated to metadata cache quickly enough for Fetch path

 class AbstractFetcherThreadTest {     fetcher.verifyLastFetchedEpoch(partition, Some(5))   } +  @Test+  def testMaybeUpdateTopicIds(): Unit = {+    val partition = new TopicPartition("topic1", 0)+    val fetcher = new MockFetcherThread++    // Start with no topic IDs+    fetcher.setReplicaState(partition, MockFetcherThread.PartitionState(leaderEpoch = 0))+    fetcher.addPartitions(Map(partition -> initialFetchState(None, 0L, leaderEpoch = 0)))+++    def verifyFetchState(fetchState: Option[PartitionFetchState], expectedTopicId: Option[Uuid]): Unit = {+      assertTrue(fetchState.isDefined)+      assertEquals(expectedTopicId, fetchState.get.topicId)+    }++    verifyFetchState(fetcher.fetchState(partition), None)++    // Add topic ID+    fetcher.maybeUpdateTopicIds(Set(partition), topicName => topicIds.get(topicName))

Right. We should get here if the partition has a zero topic ID, I think.

jolshan

comment created time in 2 days

Pull request review commentapache/kafka

KAFKA-13102: Topic IDs not propagated to metadata cache quickly enough for Fetch path

 class ReplicaManager(val config: KafkaConfig,                     stateChangeLogger.info(s"Updating log for $topicPartition to assign topic ID " +                       s"$topicId from LeaderAndIsr request from controller $controllerId with correlation " +                       s"id $correlationId epoch $controllerEpoch")+                    if (partitionState.leader != localBrokerId && metadataCache.hasAliveBroker(partitionState.leader))

We need to address this point.

jolshan

comment created time in 2 days

Pull request review commentapache/kafka

KAFKA-13102: Topic IDs not propagated to metadata cache quickly enough for Fetch path

 class ReplicaManagerTest {       ClientQuotasImage.EMPTY     )   }++  def assertFetcherHasTopicId[T <: AbstractFetcherThread](manager: AbstractFetcherManager[T],+                                                          tp: TopicPartition,+                                                          expectedTopicId: Option[Uuid]): Unit = {+    val fetchState = manager.getFetcher(tp).flatMap(_.fetchState(tp))+    assertTrue(fetchState.isDefined)+    assertEquals(expectedTopicId, fetchState.get.topicId)+  }++  @Test+  def testPartitionFetchStateUpdatesWithTopicIdAdded(): Unit = {+    val aliveBrokersIds = Seq(0, 1)+    val replicaManager = setupReplicaManagerWithMockedPurgatories(new MockTimer(time),+      brokerId = 0, aliveBrokersIds)+    try {+      val tp = new TopicPartition(topic, 0)+      val leaderAndIsr = new LeaderAndIsr(1, 0, aliveBrokersIds.toList, 0)++      val leaderAndIsrRequest1 = leaderAndIsrRequest(Uuid.ZERO_UUID, tp, aliveBrokersIds, leaderAndIsr)+      val leaderAndIsrResponse1 = replicaManager.becomeLeaderOrFollower(0, leaderAndIsrRequest1, (_, _) => ())+      assertEquals(Errors.NONE, leaderAndIsrResponse1.error)++      assertFetcherHasTopicId(replicaManager.replicaFetcherManager, tp, None)++      val leaderAndIsrRequest2 = leaderAndIsrRequest(topicId, tp, aliveBrokersIds, leaderAndIsr)+      val leaderAndIsrResponse2 = replicaManager.becomeLeaderOrFollower(0, leaderAndIsrRequest2, (_, _) => ())+      assertEquals(Errors.NONE, leaderAndIsrResponse2.error)++      assertFetcherHasTopicId(replicaManager.replicaFetcherManager, tp, Some(topicId))++    } finally {+      replicaManager.shutdown(checkpointHW = false)+    }+  }++  @ParameterizedTest+  @ValueSource(booleans = Array(true, false))+  def testReplicaAlterLogDirsWithAndWithoutIds(usesTopicIds: Boolean): Unit = {+    val version = if (usesTopicIds) LeaderAndIsrRequestData.HIGHEST_SUPPORTED_VERSION else 4.toShort+    val topicIdRaw = if (usesTopicIds) topicId else Uuid.ZERO_UUID

nit: topicId instead of topicIdRaw?

jolshan

comment created time in 2 days

Pull request review commentapache/kafka

KAFKA-13102: Topic IDs not propagated to metadata cache quickly enough for Fetch path

 abstract class AbstractFetcherThread(name: String,     } finally partitionMapLock.unlock()   } +  def maybeAddTopicIdsToThread(partitions: Set[TopicPartition], topicIds: String => Option[Uuid]) = {+    partitionMapLock.lockInterruptibly()+    try {+      partitions.foreach { tp =>+        val currentState = partitionStates.stateValue(tp)+        if (currentState != null) {+          val updatedState = currentState.updateTopicId(topicIds(tp.topic))+          partitionStates.updateAndMoveToEnd(tp, updatedState)

We need to address this point.

jolshan

comment created time in 2 days

Pull request review commentapache/kafka

KAFKA-13102: Topic IDs not propagated to metadata cache quickly enough for Fetch path

 class ReplicaManagerTest {       ClientQuotasImage.EMPTY     )   }++  def assertFetcherHasTopicId[T <: AbstractFetcherThread](manager: AbstractFetcherManager[T],+                                                          tp: TopicPartition,+                                                          expectedTopicId: Option[Uuid]): Unit = {+    val fetchState = manager.getFetcher(tp).flatMap(_.fetchState(tp))+    assertTrue(fetchState.isDefined)+    assertEquals(expectedTopicId, fetchState.get.topicId)+  }++  @Test+  def testPartitionFetchStateUpdatesWithTopicIdAdded(): Unit = {+    val aliveBrokersIds = Seq(0, 1)+    val replicaManager = setupReplicaManagerWithMockedPurgatories(new MockTimer(time),+      brokerId = 0, aliveBrokersIds)+    try {+      val tp = new TopicPartition(topic, 0)+      val leaderAndIsr = new LeaderAndIsr(1, 0, aliveBrokersIds.toList, 0)++      val leaderAndIsrRequest1 = leaderAndIsrRequest(Uuid.ZERO_UUID, tp, aliveBrokersIds, leaderAndIsr)+      val leaderAndIsrResponse1 = replicaManager.becomeLeaderOrFollower(0, leaderAndIsrRequest1, (_, _) => ())+      assertEquals(Errors.NONE, leaderAndIsrResponse1.error)++      assertFetcherHasTopicId(replicaManager.replicaFetcherManager, tp, None)++      val leaderAndIsrRequest2 = leaderAndIsrRequest(topicId, tp, aliveBrokersIds, leaderAndIsr)+      val leaderAndIsrResponse2 = replicaManager.becomeLeaderOrFollower(0, leaderAndIsrRequest2, (_, _) => ())+      assertEquals(Errors.NONE, leaderAndIsrResponse2.error)++      assertFetcherHasTopicId(replicaManager.replicaFetcherManager, tp, Some(topicId))++    } finally {+      replicaManager.shutdown(checkpointHW = false)+    }+  }++  @ParameterizedTest+  @ValueSource(booleans = Array(true, false))+  def testReplicaAlterLogDirsWithAndWithoutIds(usesTopicIds: Boolean): Unit = {+    val version = if (usesTopicIds) LeaderAndIsrRequestData.HIGHEST_SUPPORTED_VERSION else 4.toShort+    val topicIdRaw = if (usesTopicIds) topicId else Uuid.ZERO_UUID+    val topicIdOpt = if (usesTopicIds) Some(topicId) else None+    val replicaManager = setupReplicaManagerWithMockedPurgatories(new MockTimer(time))+    try {+      val topicPartition = new TopicPartition(topic, 0)+      val aliveBrokersIds = Seq(0, 1)+      replicaManager.createPartition(topicPartition)+        .createLogIfNotExists(isNew = false, isFutureReplica = false,+          new LazyOffsetCheckpoints(replicaManager.highWatermarkCheckpoints), None)+      val tp = new TopicPartition(topic, 0)+      val leaderAndIsr = new LeaderAndIsr(0, 0, aliveBrokersIds.toList, 0)++      val leaderAndIsrRequest1 = leaderAndIsrRequest(topicIdRaw, tp, aliveBrokersIds, leaderAndIsr, version = version)+      replicaManager.becomeLeaderOrFollower(0, leaderAndIsrRequest1, (_, _) => ())+      val partition = replicaManager.getPartitionOrException(tp)+      assertEquals(1, replicaManager.logManager.liveLogDirs.filterNot(_ == partition.log.get.dir.getParentFile).size)++      // Append a couple of messages.+      for (i <- 1 to 40) {+        val records = TestUtils.singletonRecords(s"message $i".getBytes)+        appendRecords(replicaManager, tp, records).onFire { response =>+          assertEquals(Errors.NONE, response.error)+        }+      }++      // find the live and different folder

nit: Could we update the comments to start with a capital letter and to finish with a dot similarly to the comment above? Or, update the comment above to look like this one. I don't mind about the format but only about the consistency.

jolshan

comment created time in 2 days

Pull request review commentapache/kafka

KAFKA-13102: Topic IDs not propagated to metadata cache quickly enough for Fetch path

 case class PartitionFetchState(fetchOffset: Long,       s", delay=${delay.map(_.delayMs).getOrElse(0)}ms" +

Could we add the topic id to the toString?

jolshan

comment created time in 2 days

PullRequestReviewEvent
PullRequestReviewEvent

Pull request review commentapache/kafka

KAFKA-13102: Topic IDs not propagated to metadata cache quickly enough for Fetch path

 class ReplicaManager(val config: KafkaConfig,                     stateChangeLogger.info(s"Updating log for $topicPartition to assign topic ID " +                       s"$topicId from LeaderAndIsr request from controller $controllerId with correlation " +                       s"id $correlationId epoch $controllerEpoch")+                    if (partitionState.leader != localBrokerId && metadataCache.hasAliveBroker(partitionState.leader))

Don't forget to remove && metadataCache.hasAliveBroker(partitionState.leader) here.

jolshan

comment created time in 2 days

PullRequestReviewEvent