profile
viewpoint

mreiferson/go-httpclient 290

a Go HTTP client with timeouts

mreiferson/go-install-as 70

a Go tool to install a package with a specific import path

mreiferson/go-options 58

a Go package to structure and resolve options

jehiah/git-open-pull 49

convert a github issue into a pull request

mreiferson/go-snappystream 47

a Go package for framed snappy streams

mreiferson/encfs-macfusion2 5

enhanced version of encfs-macfusion2 plugin http://code.google.com/p/encfs-macfusion2/

jehiah/nsq 4

realtime distributed message processing at scale

mreiferson/go-stat 4

performant instrumentation/profiling for Go

mreiferson/ddstatsd 3

statsd proxy to convert metric namespaces to tags

mreiferson/2014-talks 1

This is the official repository for slides and talks from GopherCon 2014

pull request commentnsqio/nsq

topic: allow unbuffered memoryMsgChan (partial revert of b3b29b7eff41)

sorry I'm late on this... I think I follow what's happening now, but I'm still not sold. In the event consumers are slow, this creates undesirable back-pressure when you'd rather those messages get written to disk.

Why don't we just revert #1159 and take a step back with what (if anything) we want to achieve with --mem-queue-size=0?

Alternatively, I think the fix in 786c1f2 is fine — creating an ephemeral channel should arguably never write to disk. For DPUB, perhaps there's a similar highly-targeted fix (like keeping a non-nil memory message chan and only using it when a DPUB occurs?

ploxiln

comment created time in 5 days

issue commentnsqio/pynsq

Reader keeps querying nsqlookupd after `reader.close()` if it's called too quickly

Yep, good catch. Are you interested in opening a PR to fix this?

dettonijr

comment created time in 5 days

issue commentnsqio/go-nsq

Client doesn't know it's disconnected

Doesn't sound like correct behavior. Can you share the logs from the consumer side? I assume you filed the issue here because the consumer uses go-nsq?

What versions are you running of nsqd and go-nsq?

watchforstock

comment created time in 5 days

issue closednsqio/nsq

How to get first message in each consumer call.

Hi Team,

Assume that one of my backend application will insert data in to the channel in a sequential order. Other end, consumer will read the data from the channel. Our requirement is , once the consumer connected to the nsq and read the channel, it should read 1 message from the channel at a time instead of all messages together. Means in a FIFO method.

Is this doable in NSQ?

Please advice.

Best Regards,

Sujan

closed time in 5 days

bmcinnovasionstudio

startedpanjf2000/ants

started time in a month

startedpanjf2000/gnet

started time in a month

issue commentnsqio/nsq

nsqadmin: Non admin user can delete the topic on a node

Yep, it also doesn't present a confirmation dialog.

guishoudaoge

comment created time in 2 months

startedrung/threat-matrix-cicd

started time in 2 months

pull request commentnsqio/nsq

topic: allow unbuffered memoryMsgChan (partial revert of b3b29b7eff41)

@ploxiln catching up on this...

Doesn't this effectively reduce the likelihood of messages being written to disk when mem-queue-size < 32, which is the opposite of what most users intend?

Seems like users are trying to accomplish a few things with mem-queue-size = 0:

  1. Round trip all messages to disk
  2. Preserve a more strict ordering (retries aside)
  3. Still be able to use deferred pub

This fix breaks (1), somewhat handles (2), and fixes (3).

Note, it looks like we also broke ephemeral channels according to https://github.com/metal-stack/metal-roles/pull/74. This makes sense intuitively, if you don't want any messages stored in memory ephemeral channels shouldn't work? Not sure we did that intentionally though.

ploxiln

comment created time in 2 months

push eventnsqio/go-nsq

Péter Szilágyi

commit sha 5f35ce8f2a0ee2acb2ae7626bf90ea12666d4025

consumer: lower old RDYs first, then assign to new connection Currently when a consumer is connected to a new NSQD, the connection is added to the pool of connections and then a loop iterates over all of them, rebalancing the RDY values. The iteration order of a map is random, thus it can happen that the consumer tries to assign the RDY value to the new connection, before decreasing the existing ones. This leads to a RDY of 0 being assigned. This issue is even more pronounced when there are only two connections: initially all the RDY are assigned the existing connection, and when a new NSQD is connected, the old one needs to be cut in half and the half just taken away gets assigned to the new connection. If map iteration happens in reverse order - starting with the new connection - it will get assigned 0 RDY and then the old connection gets cut in half. The end result is blocked communication on the new NSQD instance until a new round of rebalance is triggered. This issue may be less relevant in long running processes, but it is very annoying in tests where we're adding and remocing NSQD instances and the test hangs from time to time due to a flaky RDY allocation. The issue is fixed by fist iterating over all the existing connections and rebalancing them, and only at the very end calling maybeUpdateRDY on the new NSQD instance.

view details

Matt Reiferson

commit sha 827b83641ce6d8fe911c8da7c4374fdc5a61abbb

Merge pull request #341 from karalabe/fix-rdy-distribution consumer: lower old RDYs first, then assign to new connection

view details

push time in 2 months

PR merged nsqio/go-nsq

consumer: lower old RDYs first, then assign to new connection bug

Currently when a consumer is connected to a new NSQD, the connection is added to the pool of connections and then a loop iterates over all of them, rebalancing the RDY values. The iteration order of a map is random, thus it can happen that the consumer tries to assign the RDY value to the new connection, before decreasing the existing ones. This leads to a RDY of 0 being assigned.

This issue is even more pronounced when there are only two connections: initially all the RDY are assigned the existing connection, and when a new NSQD is connected, the old one needs to be cut in half and the half just taken away gets assigned to the new connection. If map iteration happens in reverse order - starting with the new connection - it will get assigned 0 RDY and then the old connection gets cut in half. The end result is blocked communication on the new NSQD instance until a new round of rebalance is triggered.

This issue may be less relevant in long running processes, but it is very annoying in tests where we're adding and remocing NSQD instances and the test hangs from time to time due to a flaky RDY allocation.

The issue is fixed by fist iterating over all the existing connections and rebalancing them, and only at the very end calling maybeUpdateRDY on the new NSQD instance.

+7 -2

1 comment

1 changed file

karalabe

pr closed time in 2 months

pull request commentnsqio/go-nsq

consumer: lower old RDYs first, then assign to new connection

LGTM, thanks

karalabe

comment created time in 2 months

release nsqio/pynsq

v0.9.1

released time in 2 months

created tagnsqio/pynsq

tagv0.9.1

The official Python client library for NSQ

created time in 2 months

push eventnsqio/pynsq

Matt Reiferson

commit sha 65a822ebaf53ed0451eb62d0b870595b25d1bf66

bump v0.9.1 stable

view details

push time in 2 months

push eventmreiferson/pynsq

Tim Gates

commit sha 34be6ed324239601941862681c66ac888c17b490

docs: Fix a few typos There are small typos in: - nsq/backoff_timer.py - nsq/protocol.py - nsq/writer.py Fixes: - Should read `successful` rather than `successfull`. - Should read `publishing` rather than `publshing`. - Should read `commands` rather than `commmands`.

view details

Matt Reiferson

commit sha eea05ac35a24d4ddd0c90cb4dec4240ad667147e

Merge pull request #256 from timgates42/bugfix_typos docs: Fix a few typos

view details

Matt Reiferson

commit sha 938435f37b53076e719e12497d8b7605a3d71d46

readme: update test badge, test dependencies

view details

Matt Reiferson

commit sha a1fac14792c07610d83cf908532be2b78741bd7d

Merge pull request #257 from mreiferson/readme-update readme: update test badge, test dependencies

view details

Alp Aker

commit sha 24a84c297c3352a8cf976c1ffe738e9adff41d9e

reader: Don't send RDY 1 on a new connection if that would violate the max-in-flight constraint.

view details

Alp Aker

commit sha 88b5c59392cbac3ff2f13926a594f8e490712b9f

reader: Avoid max-in-flight violations when exiting backoff.

view details

Alp Aker

commit sha df9ffcc5fc68365610a85c8db8a4cc862ba67c33

reader: Avoid max-in-flight violations when dynamically setting max-in-flight.

view details

Alp Aker

commit sha 16abeb71821137614c19b60b83e5e9aae334ff89

tests: Add RDY redistribution tests.

view details

Matt Reiferson

commit sha ab14efef32adf869d15752c391508701b84c50c7

Merge pull request #254 from alpaker/respect-max-in-flight-on-startup reader: don't send RDY 1 on a new connection if that would violate max-in-flight

view details

Matt Reiferson

commit sha 65a822ebaf53ed0451eb62d0b870595b25d1bf66

bump v0.9.1 stable

view details

push time in 2 months

issue commentnsqio/nsq

Disconnects seem to be quite ungraceful

I'm inclined to move this issue over to go-nsq as I think it is likely the major contributing factor here.

karalabe

comment created time in 2 months

push eventnsqio/nsq

Pierce Lopez

commit sha 60ae5badfa37c88f23026b1fd89540b9974c7262

nsqd: update dependencies go-nsq v1.1.0 go-diskqueue v1.1.0 golang/snappy v0.0.4 BurntSushi/toml v0.4.1

view details

Pierce Lopez

commit sha e6b949db33785f518e05ee44ff7f28230c5832b5

nsqlookupd: less flaky protocol test NSQLookupd.New() listens on TCPAddress and HTTPAddress, which NewOptions() gives default values of 4160 and 4161, and sometimes this conflicts with other tests running concurrently.

view details

Matt Reiferson

commit sha 198a8840f767dacb4ccbc0c560aadc25125e8e39

Merge pull request #1387 from ploxiln/diskqueue_110 nsqd: update dependencies

view details

push time in 3 months

PR merged nsqio/nsq

Reviewers
nsqd: update dependencies chore
+18 -15

6 comments

3 changed files

ploxiln

pr closed time in 3 months

pull request commentnsqio/nsq

nsqd: update go-diskqueue to v1.1.0

ready?

ploxiln

comment created time in 3 months

pull request commentnsqio/nsq

nsqd: update go-diskqueue to v1.1.0

Not sure what to make of the race detector error given that it's internal to Go source code?

ploxiln

comment created time in 3 months

Pull request review commentnsqio/nsq

nsqd: update go-diskqueue to v1.1.0

 func testIOLoopReturnsClientErr(t *testing.T, fakeConn test.FakeNetConn) { 	opts := NewOptions() 	opts.Logger = test.NewTestLogger(t) 	opts.LogLevel = LOG_DEBUG+	opts.TCPAddress = "127.0.0.1:0"+	opts.HTTPAddress = "127.0.0.1:0"

I guess there are a few tests that don't use mustStartLookupd, which does this automatically as a bootstrap step.

ploxiln

comment created time in 3 months

PullRequestReviewEvent

issue commentnsqio/pynsq

Client configured with deflate=true loses connection on Tornado 6

I can confirm that this does indeed fail as described.

ayamnikovx

comment created time in 3 months

pull request commentnsqio/go-nsq

consumer: fix send on closed channel when reconnecting

I'm not sure this is sufficient. Stop() doesn't close exitChan, so lookupdLoop() and any single-connection reconnects will still be racing. We need to ensure they all return before exit() gets called.

Allenxuxu

comment created time in 3 months

push eventmreiferson/go-nsq

Matt Reiferson

commit sha b5c6a15ca7e41ee057ab68f8c9ab15c029f0318b

*: switch to github actions

view details

Matt Reiferson

commit sha 372c9e4f9ea1e9f19830cc3e6b1b6489ff91aa46

Merge pull request #329 from mreiferson/github-actions *: switch to github actions

view details

Matt Reiferson

commit sha 37690ba6ad41d3b6f91b8d84c5981a3fb7bb6369

*: fix working directory for github actions

view details

Matt Reiferson

commit sha 2f8d0cc3d9a354899309c7f258028d5661910f16

config: use explicit int64 for random durations

view details

Matt Reiferson

commit sha 3bce8d0263da9117b74015a7c466475920077519

Merge pull request #330 from mreiferson/github-actions-fix *: fix github actions

view details

Jason

commit sha c238b8d7ebd00bb723fb4198fb66ac03cfad5b7f

consumer: re-use http client with keepalives for lookup requests

view details

Pierce Lopez

commit sha 61df38efcdbd770b6698fde3edcec39fadda569b

Merge pull request #333 from JieTrancender/perfHTTPTransport consumer: reuse lookupd http client

view details

Péter Szilágyi

commit sha c4f8b83b818cb7fad89dd46ba3d4745e3bea7f93

producer: fix shutdown logs to be uniform with others

view details

Pierce Lopez

commit sha 54583b581cf3a568a114d37ea9e0e172e9f7ed2a

Merge pull request #336 from karalabe/producer-log-tagging producer: fix shutdown logs to be uniform with others

view details

Pierce Lopez

commit sha 9205a845e60c0a41caedbd967c9458199a1b282e

bump version to 1.1.0 and update ChangeLog

view details

Matt Reiferson

commit sha 259dc590b2e791e4b11ad7496bc312ec81d1004b

Merge pull request #340 from ploxiln/bump_110 bump version to 1.1.0 and update ChangeLog

view details

push time in 3 months

Pull request review commentnsqio/nsq

nsqd: update go-diskqueue to v1.1.0

 func testIOLoopReturnsClientErr(t *testing.T, fakeConn test.FakeNetConn) { 	opts := NewOptions() 	opts.Logger = test.NewTestLogger(t) 	opts.LogLevel = LOG_DEBUG+	opts.TCPAddress = "127.0.0.1:14163" // avoid possible conflict with other tests+	opts.HTTPAddress = "127.0.0.1:14164"

why is this change necessary? does it not handle the :0 "random free port"?

ploxiln

comment created time in 3 months

PullRequestReviewEvent

push eventnsqio/go-nsq

Pierce Lopez

commit sha 9205a845e60c0a41caedbd967c9458199a1b282e

bump version to 1.1.0 and update ChangeLog

view details

Matt Reiferson

commit sha 259dc590b2e791e4b11ad7496bc312ec81d1004b

Merge pull request #340 from ploxiln/bump_110 bump version to 1.1.0 and update ChangeLog

view details

push time in 3 months

PR merged nsqio/go-nsq

Reviewers
bump version to 1.1.0 and update ChangeLog chore

I bumped to 1.1.0 because it's been a while, and there are a couple API additions:

  • SetLoggerLevel() and SetLoggerForLevel() in #275
  • Config.LookupdPollTimeout and SetLookupdHttpClient() in #333

ChangeLog is roughly chronological but with docs/cosmetic/project stuff moved to the end.

+29 -1

0 comment

2 changed files

ploxiln

pr closed time in 3 months

more