profile
viewpoint
Kamal Aboul-Hosn kamalaboulhosn Google Brooklyn, NY TL working on Google Cloud Pub/Sub

GoogleCloudPlatform/training-data-analyst 5754

Labs and demos for courses for GCP Training (http://cloud.google.com/training).

GoogleCloudPlatform/pubsub 190

This repository contains open-source projects managed by the owners of Google Cloud Pub/Sub.

kamalaboulhosn/dotnet-docs-samples 0

.NET code samples used on https://cloud.google.com

kamalaboulhosn/fruitstrap 0

Install and debug iPhone apps from the command line, without using Xcode

kamalaboulhosn/gapic-generator 0

Tools for generating API client libraries from API Service Configuration descriptions.

kamalaboulhosn/google-cloud-dotnet 0

Google Cloud Client Libraries for .NET

kamalaboulhosn/google-cloud-go 0

Google Cloud Client Libraries for Go.

kamalaboulhosn/google-cloud-php 0

Google Cloud Client Library for PHP

kamalaboulhosn/google-cloud-ruby 0

Google Cloud Client Library for Ruby

kamalaboulhosn/googleapis 0

Public interface definitions of Google APIs.

PR merged GoogleCloudPlatform/pubsub

Remove log4j
+740 -1516

0 comment

17 changed files

kamalaboulhosn

pr closed time in a month

push eventGoogleCloudPlatform/pubsub

Kamal Aboul-Hosn

commit sha 43cc7dea0d479c7c39edc7b3e83ffe8692e096be

Remove log4j (#304) * feat: Add ability to specify endpoint in CPS connector. * Set default endpoint. * Add note about low throughput when using partition * fix: Actually set endpoint provided by command-line argument * fix: Allow one to specify unordered delivery * Remove log4j usage and Kafka testing so we can remove the log4j dependency entirely.

view details

push time in a month

MemberEvent

PR opened GoogleCloudPlatform/pubsub

Remove log4j
+740 -1516

0 comment

17 changed files

pr created time in a month

push eventkamalaboulhosn/pubsub

David Xia

commit sha c9d4a4e8d6bbedd6306746f4cdce07eedf2db7ee

fix: use ubuntu-2004-focal-v20210720 as GCE VM source image fixes #293

view details

dpcollins-google

commit sha a07f310bb45b99b0f249521499cdea3884089557

Merge branch 'master' into patch1

view details

dpcollins-google

commit sha c8c0fc415c9a2ecf4b9542dae8416407001e659f

Merge pull request #294 from davidxia/patch1 fix: use ubuntu-2004-focal-v20210720 as GCE VM source image

view details

samarthsingal

commit sha 211e61f56d6c1d8029eaf50eb734e7f021544be7

Update Netty dependency

view details

samarthsingal

commit sha e7e3a6e8a178d20f883fa6dc58a63056c1d3ef44

Merge pull request #302 from GoogleCloudPlatform/samarthsingal/build-updates Update Netty dependency

view details

Kamal Aboul-Hosn

commit sha 327948da1b1e6b779c733db13ff1202aa452417b

Merge remote-tracking branch 'upstream/master'

view details

Kamal Aboul-Hosn

commit sha f0c56fee96663d9508f9e2c3bae1c914c43e9c1d

Remove log4j usage and Kafka testing so we can remove the log4j dependency entirely.

view details

push time in a month

issue commentgoogleapis/nodejs-pubsub

pubsub Subscription: should detach subscriptions failed

The internal ticket was awaiting information from @feywind about the project for progress to be made, so no changes were made in response to this.

flaky-bot[bot]

comment created time in a month

issue commentgoogleapis/java-pubsub

Exposing `publishAllOutstanding` as a public API in `PublisherInterface`

The PublisherInterface is used by the Pub/Sub Lite library. It does not implement the publishAllOutstanding() method, though. Therefore, adding the method to the interface would break the Pub/Sub Lite client library unless we added support for this method. I'm not sure how well that fits into Lite. Perhaps @dpcollins-google has some thoughts.

What was the reason for switching from Publisher to PublisherInterface?

elefeint

comment created time in a month

issue commentgoogleapis/python-pubsub

Subscriber flow control is ineffective for ordered messages.

I can't speak to the specifics of the Python library, but I can say that "max_messages" and "max_bytes" is meant to indicate the number of messages/bytes outstanding that have been sent to the user callback, but have not yet been acked or nacked.

There might be a little bit of overflow in the client library of messages that are pending to be sent to the user callback. Assuming a single stream to the server, we'd expect the number of messages in the stuck state in the client to be very small because the server side would stop sending messages when the flow control limits were reached. With multiple streams, yes, there can be more of a discrepancy because we copy the flow control values to each stream rather than dividing them up among the streams. Long-term, I think this is something we will change across all of the languages in some way.

jimfulton

comment created time in 2 months

issue commentgoogleapis/python-pubsub

Lots of unexpected logging in Python library

They say they have no adjusted log levels in their environment.

They use these Python libraries:

  • pymysql==0.8.1
  • google-cloud-pubsub==2.3.0
  • sqlalchemy==1.3.18
  • pdf2image==1.14.0

And these Alpine Linux libraries:

  • gcc
  • libc-dev
  • libffi-dev
  • mariadb-dev
  • libstdc++
  • openssh,jpeg-dev
  • zlib-dev
  • poppler
  • poppler-utils
kamalaboulhosn

comment created time in 2 months

issue commentGoogleCloudPlatform/pubsub

Avro with Kafka Connect sink connector

Yeah, so in that case, there is no way to convert the Avro into another format within the connector. You could store your schema in Pub/Sub and then manually attach the path to it as an attribute in your messages so that you can pull the schema and decode messages, though this would require your Kafka publisher to publish with the metadata in the headers.

OuesFa

comment created time in 2 months

issue commentgoogleapis/google-cloud-ruby

PubSub subscriber not processing messages after starting sometimes

Determining if it was a server-side issue would require a support request being put in via the Cloud Console as we would have to look up information about the subscriber on the backend. If the messages are getting delivered to the subscriber client and upon shutting down the subscriber, the messages are successfully processed by other subscribers, that points more to a client-side issue than a server-side issue (which is possibly what the "expired acks" graph is showing.

stevesng

comment created time in 2 months

PullRequestReviewEvent

issue commentgoogleapis/google-cloud-dotnet

PubSub SubscriberServiceApiClient.PullAsync doesn't respect maxMessages

The guarantee with pull is that you will receive up to maxMessages, but may receive fewer, even if there are more than maxMessages in the backlog. The reason is that the service is trading off fuller pull responses for shorter end-to-end latency. From the time a request comes in, messages are fetched to send in response. If some messages are quickly available, the choice is between waiting for more messages to become available or returning the messages that were quickly available. If the system waits, then the end-to-end latency increases for the messages that were already fetched. Therefore, Pub/Sub only waits a short period of time before sending the messages it has.

The code provided is an inefficient way to use Cloud Pub/Sub. The synchronous pull section of the documentation says:

Note: To achieve low message delivery latency with synchronous pull, it is important to have many simultaneously outstanding pull requests. As the throughput of the topic increases, more pull requests are necessary. In general, asynchronous pull [via the client library using streaming pull] is preferable for latency-sensitive applications.

When there are a lot of simultaneous requests, the server knows to keep a constant flow of messages ready that can be returned in responses. When there is only one request outstanding at a time, the server doesn't know that there will necessarily be subsequent requests to fulfill and therefore doesn't keep a lot of messages waiting because we don't want to waste moving messages around.

There is no way to to ensure that you receive the number of messages you request with the Pull method in Cloud Pub/Sub.

Rassi

comment created time in 2 months

issue commentgoogleapis/python-pubsub

Lots of unexpected logging in Python library

@plamut I will ask about the additional information. I agree that it doesn't look to be something at the Pub/Sub library level. My suspicion is that debug logging is turned on the gRPC level somehow.

kamalaboulhosn

comment created time in 2 months

issue commentGoogleCloudPlatform/pubsub

Avro with Kafka Connect sink connector

You can dump the Avro messages into the Cloud Pub/Sub as-is since they are just bytes. You'd then have to rely on your subscribers to decode the messages. If all messages on a topic use the same schema, then you could potentially take advantage of Pub/Sub's schema support.

OuesFa

comment created time in 2 months

issue commentgoogleapis/python-pubsub

Lots of unexpected logging in Python library

@plamut would you be able to take a look into this while @pradn is out? Thanks!

kamalaboulhosn

comment created time in 2 months

issue openedgoogleapis/python-pubsub

Lots of unexpected logging in Python library

Summarized from a customer issue:

Somehow, something with this google PubSub service is printing out Mutex.cc logs at an enormous scale of 3.5M / 5 minutes. The logs look as follows:

Error
2021-11-03 15:08:55.861 EDT[mutex.cc : 435] RAW: Lock blocking 0x7f0c66b9cb78 @
Error
2021-11-03 15:08:55.861 EDT[mutex.cc : 435] RAW: Lock returning 0x7f0c66b9cb78 @
Error
2021-11-03 15:08:55.861 EDT[mutex.cc : 435] RAW: Unlock 0x7f0c66b9cb78 @
Error
2021-11-03 15:08:55.861 EDT[mutex.cc : 435] RAW: Lock blocking 0x7f0c66b9cb78 @
Error
2021-11-03 15:08:55.861 EDT[mutex.cc : 435] RAW: Lock returning 0x7f0c66b9cb78 @
Error
2021-11-03 15:08:55.861 EDT[mutex.cc : 435] RAW: Unlock 0x7f0c66b9cb78 @
Error
2021-11-03 15:08:55.861 EDT[mutex.cc : 435] RAW: Lock blocking 0x7f0c66b9cb78 @
Error
2021-11-03 15:08:55.861 EDT[mutex.cc : 435] RAW: Lock returning 0x7f0c66b9cb78 @
Error
2021-11-03 15:08:55.861 EDT[mutex.cc : 435] RAW: Unlock 0x7f0c66b9cb78 @
Error
2021-11-03 15:08:55.861 EDT[mutex.cc : 435] RAW: Lock blocking 0x7f0c66b9cb78 @
Error
2021-11-03 15:08:55.861 EDT[mutex.cc : 435] RAW: Lock returning 0x7f0c66b9cb78 @
Error
2021-11-03 15:08:55.861 EDT[mutex.cc : 435] RAW: Unlock 0x7f0c66b9cb78 @
Error
2021-11-03 15:08:55.861 EDT[mutex.cc : 435] RAW: Lock blocking 0x7f0c66b9cb78 @
Error
2021-11-03 15:08:55.861 EDT[mutex.cc : 435] RAW: Lock returning 0x7f0c66b9cb78 @
Error
2021-11-03 15:08:55.861 EDT[mutex.cc : 435] RAW: Unlock 0x7f0c66b9cb78 @
Error
2021-11-03 15:08:55.861 EDT[mutex.cc : 435] RAW: Lock blocking 0x7f0c66b9cb78 @
Error
2021-11-03 15:08:55.861 EDT[mutex.cc : 435] RAW: Lock returning 0x7f0c66b9cb78 @
Error
2021-11-03 15:08:55.861 EDT[mutex.cc : 435] RAW: Unlock 0x7f0c66b9cb78 @
Error
2021-11-03 15:08:55.861 EDT[mutex.cc : 435] RAW: Lock blocking 0x7f0c66b9cb78 @
Error
2021-11-03 15:08:55.861 EDT[mutex.cc : 435] RAW: Lock returning 0x7f0c66b9cb78 @

The information provided about version:

  1. We are using version 2.3.0 of google-cloud-pubsub
  2. The spike in logs was not tied to an upgrade of pubsub or another library, we actually haven't touched the code in months

The customer suggested "I believe the core of the issue is that pubsub is sending messages without a delay and no exponential backoff, thus causing massive spikes in utilization and an over-use of the logging module."

created time in 2 months

issue commentGoogleCloudPlatform/pubsub

Avro with Kafka Connect sink connector

I believe the notion of the schema registry is a concept specific to Confluent Kafka and not part of the generic, open-source Kafka connect infrastructure. The GCE connector you link to is provided by Confluent, whereas this one is not. At this time, we do not support any lookup of schema in a schema registry via this connector.

OuesFa

comment created time in 2 months

issue commentGoogleCloudPlatform/pubsub

Avro with Kafka Connect sink connector

Can you explain a little more what you are trying to do? What is the format you want the data written in when sent to Pub/Sub? In what format is the data you've written to Kafka stored?

OuesFa

comment created time in 2 months

issue commentgoogleapis/nodejs-pubsub

Publisher total timeout value changed after upgrading library version

If the issue is taking 2.5 hours to manifest, it sounds to me like some kind of memory or resource leak that is slowing down processing over time. The change from 600s to 60s as the timeout may have just made that issue be realized sooner due to the smaller timeout. In general, I'd expect the number of publishes that should fail in 60s to be quite small. If it gets into the state where none were failing in that time and then most or all start failing within that time, it sounds like a resource leak.

That leak could be in the client library, potentially. Have we run any tests that run for 2+ hours with the Node library to see if problems develop? If not, it might be worth doing that.

We do ultimately need to understand why the total timeout switched from 600s to 60s without any changes we made that we expected to cause that. Am I reading googleapis/gax-nodejs#1100 correctly that if one specifies a timeout in the config, then it is used for all three timeout values (initial, max, and total)? What is the config we have that is specifying this timeout?

ShyamShah11

comment created time in 2 months

PullRequestReviewEvent
PullRequestReviewEvent

issue commentgoogleapis/google-cloud-dotnet

No information if SubscriberServiceApiClient.AcknowledgeAsync has exceeded deadline

The ack deadline is not a strong guarantee either way: there is no guarantee that a message won't be delivered before the ack deadline has expired and there is no guarantee that an ack that comes after the deadline won't be accepted. In general, acks are meant to be fire-and-forget operations for the client and in the server, they are not processed synchronously when received. At this time, there is no way to get a response from an ack and know definitively if it was accepted or not. This is part of Pub/Sub's at-least-once delivery model.

Rassi

comment created time in 3 months

issue commentgoogleapis/nodejs-pubsub

Topic creation fails with DEADLINE_EXCEEDED error

@feywind The intention was for the change to 5 seconds to only affected Publish calls, not CreateTopic calls. CreateTopic should have a deadline of 60s.

kirpichenko

comment created time in 3 months

issue commentgoogleapis/nodejs-pubsub

pubsub Snapshot seeking: "before each" hook for "should seek to a date" failed

@feywind You will have to remind me which server-side issues.

flaky-bot[bot]

comment created time in 3 months

more