issue commentRichardKnop/machinery
Default settings in RedisConfig not optimal for connection pooling
Merged to master.
See commit https://github.com/RichardKnop/machinery/commit/ec54a187c6b2bf3f5951f992f08c193446d5b12c
comment created time in 19 days
pull request commentRichardKnop/machinery
Fix #468 - Optimize RedisConfig for connection pool
Thanks @RichardKnop 👍
comment created time in a month
PR opened RichardKnop/machinery
The default Machinery settings was opening/closing lots of new Redis connections every time SendTask() is called. This lead to major CPU load for Redis and eventually exhaust the max number of client connections as the Redis process has trouble keeping up with all the open/close connections.
pr created time in a month
pull request commentRichardKnop/machinery
@WingGao : Can you please clarify the purpose of DelayedTasksKey for me?
What is the default if not specified in RedisConfig?
Thanks.
comment created time in a month
push eventcharleswhchan/machinery
commit sha f85e51e091f21501324c9d34afb72865a38b7f99
Ad escape for GroupTaskCount in README
push time in a month
pull request commentRichardKnop/machinery
Update Retry Tasks description
Thanks! 😃
comment created time in a month
PR opened juanifioren/django-oidc-provider
pr created time in a month
create barnchcharleswhchan/django-oidc-provider
branch : charleswhchan-patch-1
created branch time in a month
fork charleswhchan/django-oidc-provider
OpenID Connect and OAuth2 provider implementation for Djangonauts.
http://django-oidc-provider.readthedocs.org
fork in a month
pull request commentRichardKnop/machinery
Update Retry Tasks description
I am not sure why the CI job fails since I only touch the README file. 😅
comment created time in a month
issue commentRichardKnop/machinery
Default settings in RedisConfig not optimal for connection pooling
Thanks @RichardKnop. I will put up a PR for it in a bit.
comment created time in a month
push eventcharleswhchan/machinery
commit sha 35ad717e24d2b90082e3e6eae5c92c73366539ec
Update Retry Tasks description
push time in a month
startedgoogle/eng-practices
started time in a month
issue openedRichardKnop/machinery
Default settings in RedisConfig not optimal for connection pooling
From: https://godoc.org/github.com/RichardKnop/machinery/v1/config#RedisConfig
type RedisConfig struct {
// Maximum number of idle connections in the pool.
MaxIdle int `yaml:"max_idle" envconfig:"REDIS_MAX_IDLE"`
// Maximum number of connections allocated by the pool at a given time.
// When zero, there is no limit on the number of connections in the pool.
MaxActive int `yaml:"max_active" envconfig:"REDIS_MAX_ACTIVE"`
// Close connections after remaining idle for this duration in seconds. If the value
// is zero, then idle connections are not closed. Applications should set
// the timeout to a value less than the server's timeout.
IdleTimeout int `yaml:"max_idle_timeout" envconfig:"REDIS_IDLE_TIMEOUT"`
// If Wait is true and the pool is at the MaxActive limit, then Get() waits
// for a connection to be returned to the pool before returning.
Wait bool `yaml:"wait" envconfig:"REDIS_WAIT"`
During load testing, we found that Machinery was opening/closing lots of new connections every time and creating major CPU load for Redis. It eventually exhaust the number of connections as the Redis process has trouble keeping up with all the open/close connections.
Looking the documentation and code, it make sense because MaxActive is 0/unlimited, and MaxIdle is 0.
Perhaps a better setting out of the box is to setup a small pool of connections?
Suggestion:
MaxActive = 100
Wait = true
MaxIdle = 10
IdleTimeout = 10
Thoughts?
created time in a month
push eventcharleswhchan/machinery
commit sha fdd8669187205c86e5eb75f4d744ebbc7bece5e7
Fix #465 - Sync DelayedTasksPollPeriod in config.go/redis.go
commit sha b18ab3f4841397fa7c8e83ae96a28ef597dd6da5
Merge pull request #467 from charleswhchan/fix-465 Fix #465 - Sync DelayedTasksPollPeriod in config.go/redis.go
push time in a month
pull request commentRichardKnop/machinery
Fix #465 - Sync DelayedTasksPollPeriod in config.go/redis.go
Thanks @RichardKnop 👍
comment created time in 2 months
push eventcharleswhchan/machinery
commit sha 342ef9fe4be7c62af890ecdf595a44d1bfa2e461
Fix typo in config.go yamk --> yaml
commit sha 1bd3be901751563f6ba6de122f6ffc997fd903aa
Merge pull request #453 from charleswhchan/charleswhchan-patch-1 Fix typo in config.go
commit sha ba2f296c4f61074ea1b8eaa0d3603d7f6b8b757e
Added an extra line of logging
commit sha 3c6fa19c82de5b45ca0ad6129dbc5925f486779d
Merge pull request #455 from mjarkk/fix-redis-task-not-registered-in-worker Added logging line to the redis broker when task is not registered
commit sha 60872fba75ce661e1ce7f61239c16c4545f3b902
Added support for []byte and fixed []uint8 value types
commit sha 41e61c22099f69afbc2ff68276ca2a9d3f89c0db
Fix a race condition in Redis broker This happens to me when I have an e2e test fixture setup as follows: - miniredis.Run() // setup - worker.Launch() - short test -- takes almost no time - worker.Quit() // teardown - miniredis.Close() As worker.Launch() (or LaunchAsync()) starts redis.StartConsumer() in a goroutine, it is possible for Launch() (or LaunchAsync()) to return immediately and teardown code to be executed. If worker.Quit() has been called, then it is unlikely for the ping result to matter. So modified the code to return an appropriate error for client side checking.
commit sha 6f7b63c9a5dbb8c18444aff89d22578efe4c39b0
Merge pull request #458 from charleswhchan/fix-race-condition-with-redis-open-and-worker-quit Enhance error handling for a race condition in Redis broker
commit sha 91e34732f457f2d3229acd6a4bb03148ff0e3572
Merge pull request #457 from mjarkk/fix-uint8-slice Added support for []byte and fixed []uint8 value types
commit sha 9ec1661b792abed25c0188c2832e20db5951c375
Fix issue #461: replace thrift.git using github repo
commit sha 634bbbe832b2c8905c8eaa4514b4dbe0555888fe
Merge pull request #462 from vincenthcui/master Fix issue #461: replace thrift.git using github repo
commit sha f231d3ecc1f833230d3d186b7b31d00e5119f4c7
minify go.sum run go mod tidy to monify go.sum, `git.apache.org/thrift` has been removed
commit sha 7b1dc904a64d9c88b8f12c2b757a851c1b009b04
Merge pull request #463 from vincenthcui/master minify go.sum
commit sha a81960024072baf6c4bd7071e48b346aa08af741
fix: fixed a unit test which actually tried to connect to mongo
commit sha 45c722bb48504fcc9cf9090ea137d96402d2c96a
Merge pull request #464 from RichardKnop/fix-mongodb-backend-factory-test-issue fix: fixed a unit test which actually tried to connect to mongo
push time in 2 months
issue openedRichardKnop/machinery
RFP: Option to disable delayed task / for Redis broker?
Currently Redis broker polls Redis DB to look for delayed tasks https://github.com/RichardKnop/machinery/blob/6f7b63c9a5dbb8c18444aff89d22578efe4c39b0/v1/brokers/redis/redis.go#L331
If the software don't use delay tasks, polling nextDelayedTask() seems wasteful. While there is a workaround now to set a long DelayedTasksPollPeriod, I am not sure if there is any side-effect (for example, a long sleep may block worker shutdown?).
Would it be useful to have an option to disable polling? or allow DelayedTasksPollPeriod to be set to -1 to represent disabled state.
I am not sure if this is useful? And I don't know if other broker request similar changes.
created time in 2 months
issue commentRichardKnop/machinery
Mismatch default DelayedTasksPollPeriod
Also 20 seems too short given this comment in the code:
// Space out queries to ZSET so we don't bombard redis
// server with relentless ZRANGEBYSCOREs
time.Sleep(time.Duration(pollPeriod) * time.Millisecond)
comment created time in 2 months
issue openedRichardKnop/machinery
Mismatch default DelayedTasksPollPeriod
In https://github.com/RichardKnop/machinery/blob/master/v1/config/config.go#L44
DelayedTasksPollPeriod: 20,
In https://github.com/RichardKnop/machinery/blob/6f7b63c9a5dbb8c18444aff89d22578efe4c39b0/v1/brokers/redis/redis.go#L331
pollPeriod := 500 // default poll period for delayed tasks
Given NormalTasksPollPeriod: 1000, I suspect the default for DelayedTasksPollPeriod is incorrect and should be 500 as specified in redis.go?
created time in 2 months
pull request commentRichardKnop/machinery
Fix issue #461: replace thrift.git using github repo
@vincenthcui : Thanks for the explanation. Could you create a new PR with "go mod tidy". That would be great.
comment created time in 2 months
pull request commentRichardKnop/machinery
Fix issue #461: replace thrift.git using github repo
I wonder if the changes in go.mod should be:
replace (
git.apache.org/thrift.git v0.0.0-20180902110319-2566ecd5d999 => github.com/apache/thrift v0.0.0-20180902110319-2566ecd5d999
git.apache.org/thrift.git v0.0.0-20181218151757-9b75e4fe745a => github.com/apache/thrift v0.0.0-20181218151757-9b75e4fe745a
)
If you look at https://github.com/RichardKnop/machinery/blob/master/go.sum#L11, it is currently referencing git.apache.org/thrift.git in the following lines
git.apache.org/thrift.git v0.0.0-20180902110319-2566ecd5d999/go.mod h1:fPE2ZNJGynbRyZ4dJvy6G277gSllfV2HJqblrnkyeyg=
git.apache.org/thrift.git v0.0.0-20181218151757-9b75e4fe745a/go.mod h1:fPE2ZNJGynbRyZ4dJvy6G277gSllfV2HJqblrnkyeyg=
comment created time in 2 months
push eventcharleswhchan/demos
commit sha 6d1a53929ee1df1467947cf397d6d5d5eaeda425
Add .gitpod.yml
push time in 3 months
issue commentstretchr/testify
@Mistobaan: yes. suite.T().Skip() will skip only the current test method? (workaround until the feature is implemented)
comment created time in 3 months
push eventcharleswhchan/machinery
commit sha 41e61c22099f69afbc2ff68276ca2a9d3f89c0db
Fix a race condition in Redis broker This happens to me when I have an e2e test fixture setup as follows: - miniredis.Run() // setup - worker.Launch() - short test -- takes almost no time - worker.Quit() // teardown - miniredis.Close() As worker.Launch() (or LaunchAsync()) starts redis.StartConsumer() in a goroutine, it is possible for Launch() (or LaunchAsync()) to return immediately and teardown code to be executed. If worker.Quit() has been called, then it is unlikely for the ping result to matter. So modified the code to return an appropriate error for client side checking.
push time in 3 months
push eventcharleswhchan/machinery
commit sha 12fb04746fbcaf3563638c11b6326a6fcc5559b1
Fix a race condition in Redis broker This happens to me when I have an e2e test fixture setup as follows: - miniredis.Run() // setup - worker.Launch() - short test -- takes almost no time - worker.Quit() // teardown - miniredis.Close() As worker.Launch() (or LaunchAsync()) starts redis.StartConsumer() in a goroutine, it is possible for Launch() (or LaunchAsync()) to return immediately and teardown code to be executed. If worker.Quit() has been called, then it is unlikely for the ping result to matter. So modified the code to return an appropriate error for client side checking.
push time in 3 months
PR opened RichardKnop/machinery
This happens to me when I have an e2e test fixture setup as follows:
miniredis.Run()// setupworker.Launch()- short test -- takes almost no time
worker.Quit()// teardownminiredis.Close()
As worker.Launch() (or LaunchAsync()) starts redis.StartConsumer() in a goroutine, it is possible for worker.Launch() to return immediately and teardown code to be executed.
If worker.Quit() has been called, then it is unlikely for the Redis ping result to matter. So modified the code to discard the error.
pr created time in 3 months
create barnchcharleswhchan/machinery
branch : fix-race-condition-with-redis-open-and-worker-quit
created branch time in 3 months
pull request commentRichardKnop/machinery
fix redis broker quit very slow
@Liv1020 : I made a PR awhile ago and the fix is now included in v.1.6.8. Can you give it a try?
comment created time in 3 months