profile
viewpoint

issue commentRichardKnop/machinery

Default settings in RedisConfig not optimal for connection pooling

Merged to master.

See commit https://github.com/RichardKnop/machinery/commit/ec54a187c6b2bf3f5951f992f08c193446d5b12c

charleswhchan

comment created time in 19 days

pull request commentRichardKnop/machinery

support redis cluster

Thanks @WingGao

WingGao

comment created time in a month

pull request commentRichardKnop/machinery

Fix #468 - Optimize RedisConfig for connection pool

Thanks @RichardKnop 👍

charleswhchan

comment created time in a month

delete branch charleswhchan/machinery

delete branch : fix-468

delete time in a month

create barnchcharleswhchan/machinery

branch : fix-445

created branch time in a month

PR opened RichardKnop/machinery

Fix #468 - Optimize RedisConfig for connection pool

The default Machinery settings was opening/closing lots of new Redis connections every time SendTask() is called. This lead to major CPU load for Redis and eventually exhaust the max number of client connections as the Redis process has trouble keeping up with all the open/close connections.

+13 -2

0 comment

2 changed files

pr created time in a month

create barnchcharleswhchan/machinery

branch : fix-468

created branch time in a month

pull request commentRichardKnop/machinery

support redis cluster

@WingGao : Can you please clarify the purpose of DelayedTasksKey for me? What is the default if not specified in RedisConfig?

Thanks.

WingGao

comment created time in a month

push eventcharleswhchan/machinery

Charles Chan

commit sha f85e51e091f21501324c9d34afb72865a38b7f99

Ad escape for GroupTaskCount in README

view details

push time in a month

delete branch charleswhchan/machinery

delete branch : patch-1

delete time in a month

pull request commentRichardKnop/machinery

Update Retry Tasks description

Thanks! 😃

charleswhchan

comment created time in a month

create barnchcharleswhchan/django-oidc-provider

branch : charleswhchan-patch-1

created branch time in a month

pull request commentRichardKnop/machinery

Update Retry Tasks description

I am not sure why the CI job fails since I only touch the README file. 😅

charleswhchan

comment created time in a month

issue commentRichardKnop/machinery

Default settings in RedisConfig not optimal for connection pooling

Thanks @RichardKnop. I will put up a PR for it in a bit.

charleswhchan

comment created time in a month

PR opened RichardKnop/machinery

Update Retry Tasks description
+1 -1

0 comment

1 changed file

pr created time in a month

push eventcharleswhchan/machinery

Charles Chan

commit sha 35ad717e24d2b90082e3e6eae5c92c73366539ec

Update Retry Tasks description

view details

push time in a month

startedgoogle/eng-practices

started time in a month

issue openedRichardKnop/machinery

Default settings in RedisConfig not optimal for connection pooling

From: https://godoc.org/github.com/RichardKnop/machinery/v1/config#RedisConfig

type RedisConfig struct {
    // Maximum number of idle connections in the pool.
    MaxIdle int `yaml:"max_idle" envconfig:"REDIS_MAX_IDLE"`

    // Maximum number of connections allocated by the pool at a given time.
    // When zero, there is no limit on the number of connections in the pool.
    MaxActive int `yaml:"max_active" envconfig:"REDIS_MAX_ACTIVE"`

    // Close connections after remaining idle for this duration in seconds. If the value
    // is zero, then idle connections are not closed. Applications should set
    // the timeout to a value less than the server's timeout.
    IdleTimeout int `yaml:"max_idle_timeout" envconfig:"REDIS_IDLE_TIMEOUT"`

    // If Wait is true and the pool is at the MaxActive limit, then Get() waits
    // for a connection to be returned to the pool before returning.
    Wait bool `yaml:"wait" envconfig:"REDIS_WAIT"`

During load testing, we found that Machinery was opening/closing lots of new connections every time and creating major CPU load for Redis. It eventually exhaust the number of connections as the Redis process has trouble keeping up with all the open/close connections.

Looking the documentation and code, it make sense because MaxActive is 0/unlimited, and MaxIdle is 0.

Perhaps a better setting out of the box is to setup a small pool of connections?

Suggestion:

MaxActive = 100
Wait = true
MaxIdle = 10
IdleTimeout = 10

Thoughts?

created time in a month

push eventcharleswhchan/machinery

Charles Chan

commit sha fdd8669187205c86e5eb75f4d744ebbc7bece5e7

Fix #465 - Sync DelayedTasksPollPeriod in config.go/redis.go

view details

Richard Knop

commit sha b18ab3f4841397fa7c8e83ae96a28ef597dd6da5

Merge pull request #467 from charleswhchan/fix-465 Fix #465 - Sync DelayedTasksPollPeriod in config.go/redis.go

view details

push time in a month

pull request commentRichardKnop/machinery

Fix #465 - Sync DelayedTasksPollPeriod in config.go/redis.go

Thanks @RichardKnop 👍

charleswhchan

comment created time in 2 months

delete branch charleswhchan/machinery

delete branch : fix-465

delete time in 2 months

create barnchcharleswhchan/machinery

branch : fix-465

created branch time in 2 months

push eventcharleswhchan/machinery

Charles Chan

commit sha 342ef9fe4be7c62af890ecdf595a44d1bfa2e461

Fix typo in config.go yamk --> yaml

view details

Richard Knop

commit sha 1bd3be901751563f6ba6de122f6ffc997fd903aa

Merge pull request #453 from charleswhchan/charleswhchan-patch-1 Fix typo in config.go

view details

mjarkk

commit sha ba2f296c4f61074ea1b8eaa0d3603d7f6b8b757e

Added an extra line of logging

view details

Richard Knop

commit sha 3c6fa19c82de5b45ca0ad6129dbc5925f486779d

Merge pull request #455 from mjarkk/fix-redis-task-not-registered-in-worker Added logging line to the redis broker when task is not registered

view details

mjarkk

commit sha 60872fba75ce661e1ce7f61239c16c4545f3b902

Added support for []byte and fixed []uint8 value types

view details

Charles Chan

commit sha 41e61c22099f69afbc2ff68276ca2a9d3f89c0db

Fix a race condition in Redis broker This happens to me when I have an e2e test fixture setup as follows: - miniredis.Run() // setup - worker.Launch() - short test -- takes almost no time - worker.Quit() // teardown - miniredis.Close() As worker.Launch() (or LaunchAsync()) starts redis.StartConsumer() in a goroutine, it is possible for Launch() (or LaunchAsync()) to return immediately and teardown code to be executed. If worker.Quit() has been called, then it is unlikely for the ping result to matter. So modified the code to return an appropriate error for client side checking.

view details

Richard Knop

commit sha 6f7b63c9a5dbb8c18444aff89d22578efe4c39b0

Merge pull request #458 from charleswhchan/fix-race-condition-with-redis-open-and-worker-quit Enhance error handling for a race condition in Redis broker

view details

Richard Knop

commit sha 91e34732f457f2d3229acd6a4bb03148ff0e3572

Merge pull request #457 from mjarkk/fix-uint8-slice Added support for []byte and fixed []uint8 value types

view details

vincentcui

commit sha 9ec1661b792abed25c0188c2832e20db5951c375

Fix issue #461: replace thrift.git using github repo

view details

Richard Knop

commit sha 634bbbe832b2c8905c8eaa4514b4dbe0555888fe

Merge pull request #462 from vincenthcui/master Fix issue #461: replace thrift.git using github repo

view details

vincentcui

commit sha f231d3ecc1f833230d3d186b7b31d00e5119f4c7

minify go.sum run go mod tidy to monify go.sum, `git.apache.org/thrift` has been removed

view details

Richard Knop

commit sha 7b1dc904a64d9c88b8f12c2b757a851c1b009b04

Merge pull request #463 from vincenthcui/master minify go.sum

view details

Richard Knop

commit sha a81960024072baf6c4bd7071e48b346aa08af741

fix: fixed a unit test which actually tried to connect to mongo

view details

Richard Knop

commit sha 45c722bb48504fcc9cf9090ea137d96402d2c96a

Merge pull request #464 from RichardKnop/fix-mongodb-backend-factory-test-issue fix: fixed a unit test which actually tried to connect to mongo

view details

push time in 2 months

issue openedRichardKnop/machinery

RFP: Option to disable delayed task / for Redis broker?

Currently Redis broker polls Redis DB to look for delayed tasks https://github.com/RichardKnop/machinery/blob/6f7b63c9a5dbb8c18444aff89d22578efe4c39b0/v1/brokers/redis/redis.go#L331

If the software don't use delay tasks, polling nextDelayedTask() seems wasteful. While there is a workaround now to set a long DelayedTasksPollPeriod, I am not sure if there is any side-effect (for example, a long sleep may block worker shutdown?).

Would it be useful to have an option to disable polling? or allow DelayedTasksPollPeriod to be set to -1 to represent disabled state.

I am not sure if this is useful? And I don't know if other broker request similar changes.

created time in 2 months

issue commentRichardKnop/machinery

Mismatch default DelayedTasksPollPeriod

Also 20 seems too short given this comment in the code:

		// Space out queries to ZSET so we don't bombard redis
		// server with relentless ZRANGEBYSCOREs
		time.Sleep(time.Duration(pollPeriod) * time.Millisecond)
charleswhchan

comment created time in 2 months

issue openedRichardKnop/machinery

Mismatch default DelayedTasksPollPeriod

In https://github.com/RichardKnop/machinery/blob/master/v1/config/config.go#L44

			DelayedTasksPollPeriod: 20,

In https://github.com/RichardKnop/machinery/blob/6f7b63c9a5dbb8c18444aff89d22578efe4c39b0/v1/brokers/redis/redis.go#L331

	pollPeriod := 500 // default poll period for delayed tasks

Given NormalTasksPollPeriod: 1000, I suspect the default for DelayedTasksPollPeriod is incorrect and should be 500 as specified in redis.go?

created time in 2 months

pull request commentRichardKnop/machinery

Fix issue #461: replace thrift.git using github repo

@vincenthcui : Thanks for the explanation. Could you create a new PR with "go mod tidy". That would be great.

vincenthcui

comment created time in 2 months

pull request commentRichardKnop/machinery

Fix issue #461: replace thrift.git using github repo

I wonder if the changes in go.mod should be:

replace (
	git.apache.org/thrift.git v0.0.0-20180902110319-2566ecd5d999 => github.com/apache/thrift v0.0.0-20180902110319-2566ecd5d999
	git.apache.org/thrift.git v0.0.0-20181218151757-9b75e4fe745a => github.com/apache/thrift v0.0.0-20181218151757-9b75e4fe745a
)

If you look at https://github.com/RichardKnop/machinery/blob/master/go.sum#L11, it is currently referencing git.apache.org/thrift.git in the following lines

git.apache.org/thrift.git v0.0.0-20180902110319-2566ecd5d999/go.mod h1:fPE2ZNJGynbRyZ4dJvy6G277gSllfV2HJqblrnkyeyg=
git.apache.org/thrift.git v0.0.0-20181218151757-9b75e4fe745a/go.mod h1:fPE2ZNJGynbRyZ4dJvy6G277gSllfV2HJqblrnkyeyg=
vincenthcui

comment created time in 2 months

push eventcharleswhchan/demos

Charles Chan

commit sha 6d1a53929ee1df1467947cf397d6d5d5eaeda425

Add .gitpod.yml

view details

push time in 3 months

fork charleswhchan/demos

D-Wave Ocean code examples

fork in 3 months

issue commentstretchr/testify

Provision for Skipping a test

@Mistobaan: yes. suite.T().Skip() will skip only the current test method? (workaround until the feature is implemented)

nanjekyejoannah-zz

comment created time in 3 months

push eventcharleswhchan/machinery

Charles Chan

commit sha 41e61c22099f69afbc2ff68276ca2a9d3f89c0db

Fix a race condition in Redis broker This happens to me when I have an e2e test fixture setup as follows: - miniredis.Run() // setup - worker.Launch() - short test -- takes almost no time - worker.Quit() // teardown - miniredis.Close() As worker.Launch() (or LaunchAsync()) starts redis.StartConsumer() in a goroutine, it is possible for Launch() (or LaunchAsync()) to return immediately and teardown code to be executed. If worker.Quit() has been called, then it is unlikely for the ping result to matter. So modified the code to return an appropriate error for client side checking.

view details

push time in 3 months

push eventcharleswhchan/machinery

Charles Chan

commit sha 12fb04746fbcaf3563638c11b6326a6fcc5559b1

Fix a race condition in Redis broker This happens to me when I have an e2e test fixture setup as follows: - miniredis.Run() // setup - worker.Launch() - short test -- takes almost no time - worker.Quit() // teardown - miniredis.Close() As worker.Launch() (or LaunchAsync()) starts redis.StartConsumer() in a goroutine, it is possible for Launch() (or LaunchAsync()) to return immediately and teardown code to be executed. If worker.Quit() has been called, then it is unlikely for the ping result to matter. So modified the code to return an appropriate error for client side checking.

view details

push time in 3 months

PR opened RichardKnop/machinery

Fix a race condition in Redis broker

This happens to me when I have an e2e test fixture setup as follows:

  • miniredis.Run() // setup
  • worker.Launch()
  • short test -- takes almost no time
  • worker.Quit() // teardown
  • miniredis.Close()

As worker.Launch() (or LaunchAsync()) starts redis.StartConsumer() in a goroutine, it is possible for worker.Launch() to return immediately and teardown code to be executed.

If worker.Quit() has been called, then it is unlikely for the Redis ping result to matter. So modified the code to discard the error.

+10 -1

0 comment

1 changed file

pr created time in 3 months

pull request commentRichardKnop/machinery

fix redis broker quit very slow

@Liv1020 : I made a PR awhile ago and the fix is now included in v.1.6.8. Can you give it a try?

Liv1020

comment created time in 3 months

more