profile
viewpoint

wepay/waltz 359

Waltz is a quorum-based distributed write-ahead log for replicating transactions

smithakoduri/incubator-airflow 0

Apache Airflow (Incubating)

smithakoduri/kafka-connect-bigquery 0

A Kafka Connect BigQuery sink connector

smithakoduri/kafka-connect-protobuf-converter 0

Protobuf converter plugin for Kafka Connect

wepay/ducktape 0

System integration and performance tests

push eventwepay/waltz

Smitha Koduri

commit sha 11b2f9db4204d40102d23132aa9c715d876c6e5a

Updated server partition metric to populate latest hwm in the metric when partition object is created.

view details

Smitha Koduri

commit sha b935860048bcac19737b7a235d0b2336f36595dc

nit fix

view details

Smitha Koduri

commit sha fabebcf5e9fffa084d4dd4ddd324cf8121b8106a

Modified code to update commitHighWaterMark in AppendTask.

view details

Smitha Koduri

commit sha 9383b1053f15a8646bdb4d0a64cb33625ef2975f

nit fix

view details

smithakoduri

commit sha 7f82abea00d9336e6bc2a0f5a5fc1ea1685ea505

Merge pull request #150 from wepay/server_partition_hwm_metric_fix Fix to populate current high-water-mark of a partition to commitHighWaterMark variable in AppendTask.init()

view details

push time in 12 days

PR merged wepay/waltz

Fix to populate current high-water-mark of a partition to commitHighWaterMark variable in AppendTask.init()

Currently, the high-water-mark metric in waltz-server/../Partition.java returns the value of "commitHighWaterMark" which is set to Long.MIN_VALUE when the Partition object is created. And, this variable is updated to reflect the latest high-water-mark of a partition only when there is a write to it.

Because of which when a new WaltzServer is created or existing WaltzServer is restarted or when partitions are reassigned to a different server, the high-water-mark metric displays the value as Long.MIN_VALUE for partitions which had data written to it earlier. This ticket will help in resolving that issue and populate the latest high-water-mark of a partition even after the WaltzServer restart.

+2 -1

1 comment

1 changed file

smithakoduri

pr closed time in 12 days

PullRequestReviewEvent

Pull request review commentwepay/waltz

Fix to populate current hwm in the server partition's hwm metric when Partition object is created.

 private void registerMetrics() {         REGISTRY.gauge(metricsGroup, "total-real-time-feed-context-removed", (Gauge<Long>) () -> getTotalRealtimeFeedContextRemoved());         REGISTRY.gauge(metricsGroup, "total-catchup-feed-context-added", (Gauge<Integer>) () -> getTotalCatchupFeedContextAdded());         REGISTRY.gauge(metricsGroup, "total-catchup-feed-context-removed", (Gauge<Integer>) () -> getTotalCatchupFeedContextRemoved());-        REGISTRY.gauge(metricsGroup, "high-water-mark", (Gauge<Long>) () -> commitHighWaterMark);+        REGISTRY.gauge(metricsGroup, "high-water-mark", (Gauge<Long>) () -> {+            try {

By calling "storePartition.highWaterMark()" in registeredMetrics(), there might be a possibility that "high-water-mark" metrics value might be stuck if the recovery is not complete. In order to avoid that scenario, I modified the code to update the "commitHighWaterMark" in AppendTask.init(). This is executed as a separate thread and hence will not impact in partition creation or metrics fetching.

smithakoduri

comment created time in a month

PullRequestReviewEvent

push eventwepay/waltz

Smitha Koduri

commit sha 9383b1053f15a8646bdb4d0a64cb33625ef2975f

nit fix

view details

push time in a month

push eventwepay/waltz

Smitha Koduri

commit sha fabebcf5e9fffa084d4dd4ddd324cf8121b8106a

Modified code to update commitHighWaterMark in AppendTask.

view details

push time in a month

PR opened wepay/waltz

Fix to populate latest hwm in the server partition's hwm metric when Partition object is created.

Currently, the high-water-mark metric in waltz-server/../Partition.java returns the value of "commitHighWaterMark" which is set to Long.MIN_VALUE when the Partition object is created. And, this variable is updated to reflect the latest high-water-mark of a partition only when there is a write to it.

Because of which when a new WaltzServer is created or existing WaltzServer is restarted, the high-water-mark metric displays the value as Long.MIN_VALUE for partitions which had data returned to it earlier. This ticket will help in resolving that issue and populate the latest high-water-mark of a partition even after restart.

+28 -1

0 comment

2 changed files

pr created time in 2 months

push eventwepay/waltz

Smitha Koduri

commit sha b935860048bcac19737b7a235d0b2336f36595dc

nit fix

view details

push time in 2 months

create barnchwepay/waltz

branch : server_partition_hwm_metric_fix

created branch time in 2 months

created tagwepay/waltz

tag0.11.2

Waltz is a quorum-based distributed write-ahead log for replicating transactions

created time in 3 months

release wepay/waltz

0.11.2

released time in 3 months

PullRequestReviewEvent
PullRequestReviewEvent
more