profile
viewpoint
Murphy Randle mrmurphy Bloom Built (Day One) Hawaii https://mrmurphy.dev

mbuscemi/elm-lens 50

Elm code visualizations for maximum productivity in Atom

bloom/reason-server-tools 7

A collection of modules that we've written to be helpful when writing Node.js servers in Reason.

bloom/aviators 4

The most coolest Elm UI kit specifically made for Day One that there is.

bloom/bs-pom 2

A collection of functions for working with JS promises in ReasonML

bloom/re-localforage 2

A small Reason library for interacting with localforage. Very early and incomplete. Contributions welcome.

bloom/elm-return 1

Writer powered pipelining for `(model, Cmd msg)`

bloom/purescript-bloomdown 1

A Purescript library for parsing SlamData's dialect of Markdown.

bloom/remotedata 1

Tools for fetching data from remote sources (incl. HTTP).

elm-conf-us/2016-q-and-a 1

Q&A for elm-conf US 2016

issue commentgraphile/worker

Database grinding to a halt waiting on transaction ID

I should have closed the issue before, my apologies!

mrmurphy

comment created time in 5 days

issue closedgraphile/worker

Database grinding to a halt waiting on transaction ID

Hey there! We are just experimenting with migrating to Graphile Worker as our main worker driver, but it seems to have brought us down in production today, and I'm trying to find out why.

  • Version: 0.6.0
  • Node: 12.x
  • I'm passing my existing pool from node-postgres when constructing workerUtils.
  • I'm creating 5 jobs per second in one queue. Each of the 5 jobs has a consistent jobKey.

The idea is that these 5 jobs will run every second, unless they take longer than a second to complete, then the job will be re-queued up to one second after the previous attempt completes.

These jobs trigger a select from a table of data, and an insert into a table in another database. This shouldn't matter to the problem at hand, but both the select and the insert use their own transactions.

Sometimes this source table that the jobs are select from can build up to contain millions of rows because the database they are going to be inserted into becomes unavailable and the jobs that process them timeout. We have indexes on the table that should make it so that selecting from the table is still fast. But we've observed that when this table starts to build up, after a few hours the add_job graphile function starts to go insane waiting on transactionid:

image

image

Snapshots from RDS's dashboard^

What are the possibilities here? Am I doing something wrong with my transactions in the insertion jobs? is this an indication that I'm leaking transaction IDs? The database got so saturated that the application ground to a halt. A reboot of the Web servers didn't help. Only completely canceling the publishing of new jobs to graphile helped. When I commented out the code that published new jobs, and restarted the servers, everything came instantly back up.

That does mean that the servers stopped processing the jobs, however, so maybe our jobs are taking way too long to fail when their target DB becomes unavailable, and they're using up all available transactions / connections?

closed time in 5 days

mrmurphy

issue commentgraphile/worker

Database grinding to a halt waiting on transaction ID

Ah so sorry I didn't get back to you earlier. Thank you for the guess! We ended up abandoning this approach entirely for the specific feature we were working in.

mrmurphy

comment created time in 5 days

IssuesEvent

issue commentgraphile/worker

Database grinding to a halt waiting on transaction ID

I'm pretty sure this is an issue with how we're handling our transactions.

mrmurphy

comment created time in 14 days

issue closedgraphile/worker

Database grinding to a halt waiting on transaction ID

Hey there! We are just experimenting with migrating to Graphile Worker as our main worker driver, but it seems to have brought us down in production today, and I'm trying to find out why.

  • Version: 0.6.0
  • Node: 12.x
  • I'm passing my existing pool from node-postgres when constructing workerUtils.
  • I'm creating 5 jobs per second in one queue. Each of the 5 jobs has a consistent jobKey.

The idea is that these 5 jobs will run every second, unless they take longer than a second to complete, then the job will be re-queued up to one second after the previous attempt completes.

These jobs trigger a select from a table of data, and an insert into a table in another database. This shouldn't matter to the problem at hand, but both the select and the insert use their own transactions.

Sometimes this source table that the jobs are select from can build up to contain millions of rows because the database they are going to be inserted into becomes unavailable and the jobs that process them timeout. We have indexes on the table that should make it so that selecting from the table is still fast. But we've observed that when this table starts to build up, after a few hours the add_job graphile function starts to go insane waiting on transactionid:

image

image

Snapshots from RDS's dashboard^

What are the possibilities here? Am I doing something wrong with my transactions in the insertion jobs? is this an indication that I'm leaking transaction IDs? The database got so saturated that the application ground to a halt. A reboot of the Web servers didn't help. Only completely canceling the publishing of new jobs to graphile helped. When I commented out the code that published new jobs, and restarted the servers, everything came instantly back up.

That does mean that the servers stopped processing the jobs, however, so maybe our jobs are taking way too long to fail when their target DB becomes unavailable, and they're using up all available transactions / connections?

closed time in 14 days

mrmurphy

issue openedgraphile/worker

Database grinding to a halt waiting on transaction ID

Hey there! We are just experimenting with migrating to Graphile Worker as our main worker driver, but it seems to have brought us down in production today, and I'm trying to find out why.

  • Version: 0.6.0
  • Node: 12.x
  • I'm passing my existing pool from node-postgres when constructing workerUtils.
  • I'm creating 5 jobs per second in one queue. Each of the 5 jobs has a consistent jobKey.

The idea is that these 5 jobs will run every second, unless they take longer than a second to complete, then the job will be re-queued up to one second after the previous attempt completes.

These jobs trigger a select from a table of data, and an insert into a table in another database. This shouldn't matter to the problem at hand, but both the select and the insert use their own transactions.

Sometimes this source table that the jobs are select from can build up to contain millions of rows because the database they are going to be inserted into becomes unavailable and the jobs that process them timeout. We have indexes on the table that should make it so that selecting from the table is still fast. But we've observed that when this table starts to build up, after a few hours the add_job graphile function starts to go insane waiting on transactionid:

image

image

Snapshots from RDS's dashboard^

What are the possibilities here? Am I doing something wrong with my transactions in the insertion jobs? is this an indication that I'm leaking transaction IDs? The database got so saturated that the application ground to a halt. A reboot of the Web servers didn't help. Only completely canceling the publishing of new jobs to graphile helped. When I commented out the code that published new jobs, and restarted the servers, everything came instantly back up.

That does mean that the servers stopped processing the jobs, however, so maybe our jobs are taking way too long to fail when their target DB becomes unavailable, and they're using up all available transactions / connections?

created time in 17 days

startedcontribsys/faktory

started time in 21 days

push eventmrmurphy/loki

Murphy Randle

commit sha 0e7e47bcdd90a55821de56f725cf9ee54011dd2d

Debugging

view details

push time in a month

push eventmrmurphy/loki

Murphy Randle

commit sha 127122862e0b956c6761b5277be62e8b06af96d8

Fix config file command

view details

push time in a month

create barnchmrmurphy/loki

branch : master

created branch time in a month

created repositorymrmurphy/loki

created time in a month

push eventmrmurphy/grafana

Murphy Randle

commit sha 9bd1ac6b1cd06c20eec5427f7f686c0ce6bfc7d0

Update README.md

view details

push time in a month

push eventmrmurphy/grafana

Murphy Randle

commit sha c1110d13b07536c9cad62fadae9d2af4469355d1

Update render.yaml

view details

push time in a month

push eventmrmurphy/grafana

Murphy Randle

commit sha 68bc3abb02b68734be40ad7e28c3def208bbc2d5

Delete redis.conf

view details

push time in a month

push eventmrmurphy/redis

Murphy Randle

commit sha 9815a3ab79c6dd53c7b33c73f47d01256f41c479

Update Dockerfile

view details

push time in a month

fork mrmurphy/redis

Persistent Redis as a private Docker service on Render

https://render.com/docs/deploy-redis

fork in a month

startedvapor/vapor

started time in 2 months

startedalpinejs/alpine

started time in 2 months

startedcaprover/caprover

started time in 2 months

startedkateinoigakukun/JavaScriptKit

started time in 2 months

more