profile
viewpoint

brianc/hipchat-eraser 7

Erase your 1 on 1 HipChat history

brianc/bencher 3

node.js benchmarking & performance analysis utilities

brianc/fluxed 3

tiny flux implementation

brianc/hyperterm-focus-reporting 3

Adds focus reporting to hyperterm for tmux & vim support

brianc/drain 2

A testing helper for node.js async code

brianc/emacs-starter-kit 2

All the code you need to get started, with an emphasis on dynamic languages.

brianc/brianc.github.com 1

stuff stuff. gimme some stuff.

brianc/connect-assetmanager 1

Middleware for Connect (node.js) for handling your static assets.

brianc/cookie-sessions 1

Secure cookie-based session middleware for Connect

push eventbrianc/node-postgres

Lewis Cowles

commit sha fd2c3563a57f19ca49cefa6b1de999d9aaa9b5f5

Security: simplify defineProperty non-enumerables * `password` already has this set, but was a little long considering we only want to override default of one property * `ssl.key` was showing up in tracebacks

view details

Lewis Cowles

commit sha e82137e6d3fcb0a84e90e0107a3606085da73806

Tests

view details

Lewis Cowles

commit sha 80c500ffbffff8c2445dce44661e85590dc026e3

Update packages/pg-pool/index.js Co-authored-by: Charmander <~@charmander.me>

view details

push time in 2 days

PR merged brianc/node-postgres

Security: Don't leak SSL key to logs, by making non-enumerable

Problem-space

  • password already has this set, but was a little long considering we only want to override default of one property
  • ssl.key was showing up in tracebacks

This came up locally when disconnecting a client after connection, but before close

Emitted 'error' event on BoundPool instance at:
    at Client.idleListener (C:\Users\lewis\projects\nodejs-app\node_modules\pg-pool\index.js:57:10)
    at Client.emit (events.js:321:20)
    at connectedErrorHandler (C:\Users\lewis\projects\nodejs-app\node_modules\pg\lib\client.js:221:10)
    at Connection.<anonymous> (C:\Users\lewis\projects\nodejs-app\node_modules\pg\lib\client.js:289:9)
    at Object.onceWrapper (events.js:427:28)
    [... lines matching original stack trace ...]
    at processTicksAndRejections (internal/process/task_queues.js:84:21) {
  client: Client {

with

key: '-----BEGIN RSA PRIVATE KEY-----\n' +
        'MIIEpAIBAAKCAQEAxAnztpOXj+6OuEONaUkpW2CPxCOoKyuzlsF5PTcjETHGgSN+\n' +
        ...
        'qq/1uRgkxeakQ9gl9sXL94+ALu8bHKFVReZbSsdYEcdJwdEH5F/4HA==\n' +
        '-----END RSA PRIVATE KEY-----\n'

Secrets in logs are not great.

Temporary fix

Without this code, you can already make the key non-enumerable. This just passes that responsibility into the library so that it does not impact users who don't know about this.

+68 -0

13 comments

4 changed files

Lewiscowles1986

pr closed time in 2 days

pull request commentbrianc/node-postgres

Security: Don't leak SSL key to logs, by making non-enumerable

today not going well i'm sorry - will try again tomorrow 😭

Lewiscowles1986

comment created time in 2 days

push eventbrianc/node-postgres-docs

dependabot[bot]

commit sha 413eedf5e7d492f4b32409800237aa6658c423ec

Bump http-proxy from 1.17.0 to 1.18.1 (#100) Bumps [http-proxy](https://github.com/http-party/node-http-proxy) from 1.17.0 to 1.18.1. - [Release notes](https://github.com/http-party/node-http-proxy/releases) - [Changelog](https://github.com/http-party/node-http-proxy/blob/master/CHANGELOG.md) - [Commits](https://github.com/http-party/node-http-proxy/compare/1.17.0...1.18.1) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

view details

push time in 3 days

PR merged brianc/node-postgres-docs

Bump http-proxy from 1.17.0 to 1.18.1 dependencies

Bumps http-proxy from 1.17.0 to 1.18.1. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/http-party/node-http-proxy/blob/master/CHANGELOG.md">http-proxy's changelog</a>.</em></p> <blockquote> <h2><a href="https://github.com/http-party/node-http-proxy/compare/1.18.0...v1.18.1">v1.18.1</a> - 2020-05-17</h2> <h3>Merged</h3> <ul> <li>Skip sending the proxyReq event when the expect header is present <a href="https://github-redirect.dependabot.com/http-party/node-http-proxy/pull/1447"><code>#1447</code></a></li> <li>Remove node6 support, add node12 to build <a href="https://github-redirect.dependabot.com/http-party/node-http-proxy/pull/1397"><code>#1397</code></a></li> </ul> <h2><a href="https://github.com/http-party/node-http-proxy/compare/1.17.0...1.18.0">1.18.0</a> - 2019-09-18</h2> <h3>Merged</h3> <ul> <li>Added in auto-changelog module set to keepachangelog format <a href="https://github-redirect.dependabot.com/http-party/node-http-proxy/pull/1373"><code>#1373</code></a></li> <li>fix 'Modify Response' readme section to avoid unnecessary array copying <a href="https://github-redirect.dependabot.com/http-party/node-http-proxy/pull/1300"><code>#1300</code></a></li> <li>Fix incorrect target name for reverse proxy example <a href="https://github-redirect.dependabot.com/http-party/node-http-proxy/pull/1135"><code>#1135</code></a></li> <li>Fix modify response middleware example <a href="https://github-redirect.dependabot.com/http-party/node-http-proxy/pull/1139"><code>#1139</code></a></li> <li>[dist] Update dependency async to v3 <a href="https://github-redirect.dependabot.com/http-party/node-http-proxy/pull/1359"><code>#1359</code></a></li> <li>Fix path to local http-proxy in examples. <a href="https://github-redirect.dependabot.com/http-party/node-http-proxy/pull/1072"><code>#1072</code></a></li> <li>fix reverse-proxy example require path <a href="https://github-redirect.dependabot.com/http-party/node-http-proxy/pull/1067"><code>#1067</code></a></li> <li>Update README.md <a href="https://github-redirect.dependabot.com/http-party/node-http-proxy/pull/970"><code>#970</code></a></li> <li>[dist] Update dependency request to ~2.88.0 [SECURITY] <a href="https://github-redirect.dependabot.com/http-party/node-http-proxy/pull/1357"><code>#1357</code></a></li> <li>[dist] Update dependency eventemitter3 to v4 <a href="https://github-redirect.dependabot.com/http-party/node-http-proxy/pull/1365"><code>#1365</code></a></li> <li>[dist] Update dependency colors to v1 <a href="https://github-redirect.dependabot.com/http-party/node-http-proxy/pull/1360"><code>#1360</code></a></li> <li>[dist] Update all non-major dependencies <a href="https://github-redirect.dependabot.com/http-party/node-http-proxy/pull/1356"><code>#1356</code></a></li> <li>[dist] Update dependency agentkeepalive to v4 <a href="https://github-redirect.dependabot.com/http-party/node-http-proxy/pull/1358"><code>#1358</code></a></li> <li>[dist] Update dependency nyc to v14 <a href="https://github-redirect.dependabot.com/http-party/node-http-proxy/pull/1367"><code>#1367</code></a></li> <li>[dist] Update dependency concat-stream to v2 <a href="https://github-redirect.dependabot.com/http-party/node-http-proxy/pull/1363"><code>#1363</code></a></li> <li>x-forwarded-host overwrite for mutli level proxies <a href="https://github-redirect.dependabot.com/http-party/node-http-proxy/pull/1267"><code>#1267</code></a></li> <li>[refactor doc] Complete rename to http-party org. <a href="https://github-redirect.dependabot.com/http-party/node-http-proxy/pull/1362"><code>#1362</code></a></li> <li>Highlight correct lines for createProxyServer <a href="https://github-redirect.dependabot.com/http-party/node-http-proxy/pull/1117"><code>#1117</code></a></li> <li>Fix docs for rewrite options - 201 also handled <a href="https://github-redirect.dependabot.com/http-party/node-http-proxy/pull/1147"><code>#1147</code></a></li> <li>Update .nyc_output <a href="https://github-redirect.dependabot.com/http-party/node-http-proxy/pull/1339"><code>#1339</code></a></li> <li>Configure Renovate <a href="https://github-redirect.dependabot.com/http-party/node-http-proxy/pull/1355"><code>#1355</code></a></li> <li>[examples] Restream body before proxying, support for Content-Type of application/x-www-form-urlencoded <a href="https://github-redirect.dependabot.com/http-party/node-http-proxy/pull/1264"><code>#1264</code></a></li> </ul> <h3>Commits</h3> <ul> <li>[dist] New test fixtures. <a href="https://github.com/http-party/node-http-proxy/commit/7e4a0e511bc30c059216860153301de2cdd1e97f"><code>7e4a0e5</code></a></li> <li>[dist] End of an era. <a href="https://github.com/http-party/node-http-proxy/commit/a9b09cce43f072db99fb5170030a05536177ccb7"><code>a9b09cc</code></a></li> <li>[dist] Version bump. 1.18.0 <a href="https://github.com/http-party/node-http-proxy/commit/9bbe486c5efcc356fb4d189ef38eee275bbde345"><code>9bbe486</code></a></li> <li>[fix] Latest versions. <a href="https://github.com/http-party/node-http-proxy/commit/59c4403e9dc15ab9b19ee2a3f4aecbfc6c3d94c4"><code>59c4403</code></a></li> <li>[fix test] Update tests. <a href="https://github.com/http-party/node-http-proxy/commit/dd1d08b6319d1def729554446a5b0176978a8dad"><code>dd1d08b</code></a></li> <li>[dist] Update dependency ws to v3 [SECURITY] <a href="https://github.com/http-party/node-http-proxy/commit/b00911c93740a00c5cfbacbb91565cb6912ed255"><code>b00911c</code></a></li> <li>[dist] .gitattributes all the things. <a href="https://github.com/http-party/node-http-proxy/commit/fc93520d741ec80be8ae31ca005f3e9c199e330e"><code>fc93520</code></a></li> <li>[dist] Regenerate package-lock.json. <a href="https://github.com/http-party/node-http-proxy/commit/16d4f8a95162b2e2e4ee6657c500f1208c044b2d"><code>16d4f8a</code></a></li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/http-party/node-http-proxy/commit/9b96cd725127a024dabebec6c7ea8c807272223d"><code>9b96cd7</code></a> 1.18.1</li> <li><a href="https://github.com/http-party/node-http-proxy/commit/335aeeba2f0c286dc89c402eeb76af47834c89a3"><code>335aeeb</code></a> Skip sending the proxyReq event when the expect header is present (<a href="https://github-redirect.dependabot.com/http-party/node-http-proxy/issues/1447">#1447</a>)</li> <li><a href="https://github.com/http-party/node-http-proxy/commit/dba39668ba4c9ad461316e834b2d64b77e1ca88e"><code>dba3966</code></a> Remove node6 support, add node12 to build (<a href="https://github-redirect.dependabot.com/http-party/node-http-proxy/issues/1397">#1397</a>)</li> <li><a href="https://github.com/http-party/node-http-proxy/commit/9bbe486c5efcc356fb4d189ef38eee275bbde345"><code>9bbe486</code></a> [dist] Version bump. 1.18.0</li> <li><a href="https://github.com/http-party/node-http-proxy/commit/6e4bef4d1cd96e7a284717941e0fc274acbd3712"><code>6e4bef4</code></a> Added in auto-changelog module set to keepachangelog format (<a href="https://github-redirect.dependabot.com/http-party/node-http-proxy/issues/1373">#1373</a>)</li> <li><a href="https://github.com/http-party/node-http-proxy/commit/d05624167ce75e860770c13afeacec2ce0f67add"><code>d056241</code></a> fix 'Modify Response' readme section to avoid unnecessary array copying (<a href="https://github-redirect.dependabot.com/http-party/node-http-proxy/issues/1300">#1300</a>)</li> <li><a href="https://github.com/http-party/node-http-proxy/commit/244303b994525684e1ec8dff2e8055f89b62b1ee"><code>244303b</code></a> Fix incorrect target name for reverse proxy example (<a href="https://github-redirect.dependabot.com/http-party/node-http-proxy/issues/1135">#1135</a>)</li> <li><a href="https://github.com/http-party/node-http-proxy/commit/b4028ba78bc4616e6969e0e66b0fe4634849b68b"><code>b4028ba</code></a> Fix modify response middleware example (<a href="https://github-redirect.dependabot.com/http-party/node-http-proxy/issues/1139">#1139</a>)</li> <li><a href="https://github.com/http-party/node-http-proxy/commit/77a98159d2da0f20a03e2819c79662f36069f234"><code>77a9815</code></a> [dist] Update dependency async to v3 (<a href="https://github-redirect.dependabot.com/http-party/node-http-proxy/issues/1359">#1359</a>)</li> <li><a href="https://github.com/http-party/node-http-proxy/commit/c662f9ebcd8d623db374dbc7bef231b2b0af0c3a"><code>c662f9e</code></a> Fix path to local http-proxy in examples. (<a href="https://github-redirect.dependabot.com/http-party/node-http-proxy/issues/1072">#1072</a>)</li> <li>Additional commits viewable in <a href="https://github.com/http-party/node-http-proxy/compare/1.17.0...1.18.1">compare view</a></li> </ul> </details> <br />

Dependabot compatibility score

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


<details> <summary>Dependabot commands and options</summary> <br />

You can trigger Dependabot actions by commenting on this PR:

  • @dependabot rebase will rebase this PR
  • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
  • @dependabot merge will merge this PR after your CI passes on it
  • @dependabot squash and merge will squash and merge this PR after your CI passes on it
  • @dependabot cancel merge will cancel a previously requested merge and block automerging
  • @dependabot reopen will reopen this PR if it is closed
  • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
  • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
  • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
  • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
  • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
  • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
  • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
  • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the Security Alerts page.

</details>

+13 -10

1 comment

1 changed file

dependabot[bot]

pr closed time in 3 days

push eventbrianc/node-postgres-docs

Matt Riedemann

commit sha 2c0a8eef95fe0f6300b17d9a03ce3dd016e89f78

Document idle_in_transaction_session_timeout (#106) * Document idle_in_transaction_session_timeout Simple documentation for the pass-through connection parameter added in node-postgres/pull/2049. The description is taken from https://www.postgresql.org/docs/current/runtime-config-client.html. Closes node-postgres/issues/2381 * Small wording tweak

view details

push time in 3 days

PR merged brianc/node-postgres-docs

Document idle_in_transaction_session_timeout

Simple documentation for the pass-through connection parameter added in https://github.com/brianc/node-postgres/pull/2049.

The description is taken from https://www.postgresql.org/docs/current/runtime-config-client.html.

Closes https://github.com/brianc/node-postgres/issues/2381

+1 -0

2 comments

1 changed file

mriedem

pr closed time in 3 days

issue closedbrianc/node-postgres

Documentation - How to pass in idle_in_transaction_session_timeout

Hello,

I'm having trouble passing this config into the client: idle_in_transaction_session_timeout implemented here: https://github.com/brianc/node-postgres/pull/2049

However I couldn't figure out where to pass it in given the starting example of

const { Pool } = require('pg');
const pool = new Pool();

pool.query('SQL...')

I looked into the config document and did not see idle_in_transaction_session_timeout as a documented property of the PoolConfig type

It would be nice to have the available options documented here: https://node-postgres.com/api/pool#constructor

Thanks

closed time in 3 days

legopin

pull request commentbrianc/node-postgres-docs

Document idle_in_transaction_session_timeout

nice! thanks!

mriedem

comment created time in 3 days

issue commentbrianc/node-postgres

How I can have two different connection pools for two different databases?

if you want two pools it's pretty straight forward:

const pool1 = new pg.Pool({ database: 'database1' })
const pool2 = new pg.Pool({ database: database2' })

Up to you in your app to decide which pool to use.

As far as serverless + pgbouncer...it should work though I don't run pgbouncer myself in production.

Also is it possible to execute multiple queries in the same statement?

You can only run multiple queries in a single statement if you do not use parameters. This is a postgres protocol restriction & there's no working around it. 99% of the time just do multiple statements w/ async/await

const res1 = await pool.query('SELECT 1')
const res2 = await pool.query('SELECT 2')

If you absolutely need to execute multiple statements in the same query, don't use parameters:

const res = await pool.query('SELECT 1; SELECT 2;).
alekbarszczewski

comment created time in 3 days

pull request commentbrianc/node-postgres

Security: Don't leak SSL key to logs, by making non-enumerable

yo - i can merge this. I'll do so first thing tomorrow (I like to be around all day in case issues come up). Having to deal w/ some personal tragedy this week....our dog of 11 years is getting euthanized this week....so there might be some delays depending on how things go. 😭

Lewiscowles1986

comment created time in 3 days

pull request commentbrianc/node-postgres

Remove console.error on pg-native module not found

Im out of town until Monday. I will look then.

On Fri, Oct 16, 2020 at 6:08 AM Michael Salinger notifications@github.com wrote:

@brianc https://github.com/brianc Can this be merged?

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/brianc/node-postgres/pull/2372#issuecomment-709981149, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAMHIJYYYNPJM3PU4PL7HLSLASTRANCNFSM4SKRO3RA .

caseywebdev

comment created time in 6 days

pull request commentbrianc/node-postgres

chore(): update eslint

@chyzwar what did you want me to label this pull request with? I think this looks good to merge.

chyzwar

comment created time in 9 days

issue commentbrianc/node-postgres

Available clients in pool

yup that looks correct! I should make some first class getters to return that info, really...but that'll work for now! Good luck with your upgrade & lmk if you have any other questions.

ranode

comment created time in 9 days

pull request commentbrianc/node-postgres

chore(): added dependabot

I think dependabot is nice but it would raise PR for versions of packages that no longer support node v8. For example lestest ts-node and mocha no longer suport node v8.

hmm that might be hard to deal with, any suggestions?

chyzwar

comment created time in 9 days

PullRequestReviewEvent

Pull request review commentbrianc/node-postgres

[WIP] pg-query-stream typescript

+{+  "compilerOptions": {+    "module": "commonjs",+    "esModuleInterop": true,+    "allowSyntheticDefaultImports": true,+    "strict": false,

it will be nice to turn this to true one day, but this is fine for the initial port.

chyzwar

comment created time in 10 days

PullRequestReviewEvent
PullRequestReviewEvent

issue commentbrianc/node-postgres

Export DatabaseError in pg package

so you want the main pg package to export DatabaseError? that sounds reasonable

golergka

comment created time in 10 days

issue commentbrianc/node-postgres

All results from SELECT query have the same ID!?!?

You're joining 3 tables, each with an id field. pg tries to map the column names to an object. In the event of a column name collision it doesn't know what to do. Try something like this:

SELECT task.id as "taskId", room.id as "roomId"

etc - then the names will map into the object without collision. You could also try

client.query({  text: 'SELECT * ....', rowMode: 'array' })

to return an array of fields and map it into an object however you'd like

KemikalGeneral

comment created time in 10 days

pull request commentbrianc/node-postgres

Exposing TypeOverrides from the library's root

It's just type exposure through variable, why do you need tests for that?

Otherwise in the future it might get unexposed & then we wouldn't know we're breaking anything until we release a version w/o it exposed. So just a simple check to make sure it's there is fine.

vitaly-t

comment created time in 10 days

issue commentbrianc/node-postgres

pool.connect() waits forever

Yeah pg<=8.0 is not compatible with node>=14

joker-8989

comment created time in 10 days

pull request commentbrianc/node-postgres

Security: Don't leak SSL key to logs, by making non-enumerable

I like the amount of tests per the level of lines of library code changed & appreciate your contribution! Since @charmander previously reviewed I'll leave it up to them to make the final call. If you don't hear back from either of us in a week or so LMK & I can take over and drive this across the finish line but want to give charmander some time...we're working for free after all so sometimes its a bit slower but that's how it goes!

Lewiscowles1986

comment created time in 13 days

pull request commentbrianc/node-postgres

Exposing TypeOverrides from the library's root

Thanks for this - this seems reasonable, but could you write some tests which will both prevent this from regressing & also demonstrate how this would be useful?

vitaly-t

comment created time in 13 days

issue commentbrianc/node-postgres

Using async/await with pg-cursor?

@felixfbecker

It would be amazing if Cursor could implement Symbol.asyncIterator so that it can be iterated with for await (const row of cursor) {}.

For this you should probably use pg-query-stream which already supports this. pg-cursor is used by pg-query-stream. I'm definitely cool with promisifying the cursor, but really the cursor is a bit lower level than pg-query-stream and can be useful primarily when you want to read exactly a certain number of rows at a time in chunks to carefully control how you're consuming. pg-query-stream is probably more generally useful for "pipe this query to this other thing" but due to high water marks, back pressure, and internal buffering in node streams you don't have as fine grained control over the exact number of rows read. In practice this is usually fine, and kinda preferred as you usually don't need to think about things at that level...so would recommend pg-query-stream for most use cases, including async iteration.

zachsa

comment created time in 13 days

issue closedbrianc/node-postgres

Losing connections in pg_pool?

Greetings,

I am running 30 nodejs instances in AWS talking to a postgreSQL rds instance. Each nodejs instance is using pg and a pg_pool for all database interaction. All calls are using simply pg_pool.query to interact with the pool.

Everything runs along well for several hours - we watch the waitingCount, idleCount and runningCount during execution and the values rise and fall nicely during the run.

As some point in time however things go off the rails. The totalCount rises to the limit of 90, the idleCount remains at zero and the waitingCount continues to rise. (up to and over 100,000).

The netstat command on the ec2 instances show they have 90 sessions established to the database server - however pg_stat_activity on the postgreSQL server indicates no queries are active from the instance.

It appears the connections are idle - however the pool does not seem to be placing them back into the pool for the waiting entries?

NodeJS Version: 4.2.6 pg Version: 6.4.2 pg-pool: 2.0.10 PostgreSQL Version 9.5.15 RDS instance size: db.r4.16xlarge

We are creating a single global pool - and are handling error, remove, connect and aquire messages.
Have the pool configured with an connectionTimeoutMillis: 60000 and idleTimeoutMillis: 120000 Using the bluebird promise.

Am I actually losing connections in the pool?

As an aside - eventually the postgreSQL rds instance eventually runs out of freeableMemory and crashes. AWS claims the crash is caused by excessive load.

closed time in 13 days

pcbentz

issue commentbrianc/node-postgres

Losing connections in pg_pool?

@pcbentz - charmander is right you're running pretty old stuff. It's hard to say if the bugs are related to your code, version of node, or older version of PG....a few things I can recommend, some of which you probably already thought of.

Make sure you have a single place in your app to do queries & make sure all the clients are returned to the pool...the most common issue of starved connection pools is not returning the client to the pool. At the very bottom of this page I put a little bit of info on how you could possibly track down unreturned clients.

As an aside - eventually the postgreSQL rds instance eventually runs out of freeableMemory and crashes. AWS claims the crash is caused by excessive load.

If your instance is running out of memory but your sessions connected to the database is remaining steady at 90 then it's likely something fishy going on with your actual database. Connections to postgres backend use a non-trivial amount of ram because postgres uses a threaded model - but it's something like 4 megabytes per connection. In the distant past I've crashed a postgres instance on an 4xl instance in ec2 but we connected like 3,000+ connections to it at once (oops!). Anyways, if you only have 90 connections steadily open to your database and it's running out of freeable memory, something else is going on there...possibly some runaway query or something else on the box.

The final thing you can try doing, if upgrading pg & node isn't an option for you (I run apps in prod too, I know it can be a pain) is using something like https://www.npmjs.com/package/generic-pool and implement your own pool if you suspect the issue is in pg-pool. You can have fine-grained control over acquire and release semantics there & do any sort of instrumentation you need.

Hope this helps!

pcbentz

comment created time in 13 days

issue commentbrianc/node-postgres

TypeError: Cannot read property 'name' of null

@pckilgore my pleasure! Let me know if it still happens...was quite a monster to track down but @timotm is the real hero here. Steps to reproduce & deep analysis helped me know where to focus. I regret it took me so long to circle back to this issue.

greghart

comment created time in 14 days

created tagbrianc/node-postgres

tagpg-query-stream@3.3.1

PostgreSQL client for node.js.

created time in 14 days

push eventbrianc/node-postgres

Brian M. Carlson

commit sha 36342c9a84b68123f666879a9f34ac319a44727a

Publish - pg-cursor@2.4.1 - pg-query-stream@3.3.1 - pg@8.4.1

view details

push time in 14 days

created tagbrianc/node-postgres

tagpg@8.4.1

PostgreSQL client for node.js.

created time in 14 days

created tagbrianc/node-postgres

tagpg-cursor@2.4.1

PostgreSQL client for node.js.

created time in 14 days

push eventbrianc/node-postgres

Brian M. Carlson

commit sha 9c678e108c4ef73187d16bd7b6fae8cd71fe9895

Fix double-sync crash on postgres 9.x

view details

Brian M. Carlson

commit sha 17e7e9ed3d9037fcd57627653c8bb7089deb1969

Remove fix to fail tests

view details

Brian M. Carlson

commit sha f55d879c52f01a288686626a216b27b65498cc99

Apply fix

view details

Brian M. Carlson

commit sha b45051d72a96408a2c019d4e54490fba5f3270e3

Update comments

view details

Brian M. Carlson

commit sha d31486fb7c630ce0d10653ff731e8b563ba50af8

Change when sync is sent during pipelining

view details

Brian M. Carlson

commit sha dd3ce616d0fbdb92a7e146ecf4171bf3c1b3ea97

Fixes based on postgres maintainer advice

view details

Brian M. Carlson

commit sha d8681fc2cd1350731adec956367ff36aa1d67582

Comments & cleanup

view details

push time in 14 days

PR merged brianc/node-postgres

Fix double-sync crash on postgres 9.x

An attempt to fix #1105

It looks like in rare, random occasions postgres 9.x is sending both ErrorResponse and CommandComplete when a query times out. In an attempt to fix this, I have changed the query to only send a sync on an error message if it hasn't already sent a sync. Judging from the docs this should never happen, and indeed on newer versions of postgres the bug is not reproducible...but lots of folks still run postgres 9.x in production. I'm not super fond of this fix as it feels like it's working around a possible race condition in postgres 9.x so I'd love some additional eyes & ideas on this one. Also, want to run the test suite and make sure everything passes in the whole test matrix on travis.

One option is to put this behind some environment variable like PGSKIPDOUBLESYNC or something and allow people to opt in to this change. I'm worried about a (according to the docs, not possible) circumstance where we receive two messages requiring a sync and we only send one sync back.

+63 -21

8 comments

4 changed files

brianc

pr closed time in 14 days

issue closedbrianc/node-postgres

TypeError: Cannot read property 'name' of null

First off, thanks for the library! Overall it's been incredibly performant and straightforward to use. I am running into one issue which may be an issue with the library, though it's a bit hard to follow the protocol for me so my usage may be off as well.

I'm running into an uncaught error during the parseComplete handler in client.js (https://github.com/brianc/node-postgres/blob/v6.0.1/lib/client.js#L131).

I am doing a direct connect against the pool and then utilizing pg-query-stream

pool.connect().then(function(client){
  stream = client.query(new QueryStream(query, null));

  stream.on('error', function(streamError){
    stream.close(function(closeError){ 
      // What to do with a close error? At this point stream is already hosed :shrug:
      client.release(streamError);
    }
  });

  stream.on('close', function(){
    client.release();
  }); 
});

At some point during my app's life cycle, I get an uncaught error

TypeError: Cannot read property 'name' of null
    at null.<anonymous> (/var/app/current/node_modules/pg/lib/client.js:139:26)
    at emit (events.js:107:17)
    at Socket.<anonymous> (/var/app/current/node_modules/pg/lib/connection.js:121:12)
    at Socket.emit (events.js:107:17)
    at readableAddChunk (_stream_readable.js:163:16)
    at Socket.Readable.push (_stream_readable.js:126:10)
    at TCP.onread (net.js:540:20)

So obviously activeQuery is becoming null, and I am trying to narrow down why I would receive a parseComplete event when the active query is null.

According to my logs, the stream is not emitting an error, and seems to be closing normally. Therefore it seems like the connection client is getting a readyForQuery or end event, and then right after that getting the parseComplete event. Any idea why this would be happening, or see any issues with my usage? Thanks for any help you can give! I'll keep looking into it as well.

closed time in 14 days

greghart

push eventbrianc/node-postgres

Brian M. Carlson

commit sha ab995bbdcfbf5dfaa7b95248a2ce83c4ffb863a6

Comments & cleanup

view details

push time in 14 days

pull request commentbrianc/node-postgres

Fix double-sync crash on postgres 9.x

okay...so w/ some help from @sehrope and a postgres maintainer from IRC I've changed the approach somewhat. When discussing the maintainer (I believe their github name is @rhodiumtoad) suggested to pipeline the sync message at the end of the other pipelined messages. Sending sync immediately after execute allows the backend to process all the messages and only respond with either CommandComplete or ErrorResponse. This fixes the issue in the test w/o requiring any hacks or discarding messages. It also speeds up some of my crude benchmarks by over 10% which is a super awesome bonus side effect.

brianc

comment created time in 14 days

issue commentbrianc/node-postgres

TypeError: Cannot read property 'name' of null

@pvatterott that's good to know, but a bummer as my hunch was around a race condition in older versions of postgres. I've been going back and forth w/ some postgres maintainers & think this might fix it. I can confirm it fixes in 9.x & 10.x of postgres, it might fix it for 12.3 of postgres as well. The change involves pipelining the sync command in with the other extended query commands all at once. It also improves my benchmarks by up to 10% in some cases, which is a nice side effect.

greghart

comment created time in 14 days

push eventbrianc/node-postgres

Brian M. Carlson

commit sha 27637c1561f53f79f200463054c87b895aeb3a9b

Change when sync is sent during pipelining

view details

Brian M. Carlson

commit sha ba231d5a65a9c19c2ca064d63eddaf24e3d5522a

Fixes based on postgres maintainer advice

view details

push time in 14 days

issue commentbrianc/node-postgres

SSL connection - is there a way to provide certificates not with .pem format

The ssl config is forwarded unmodified directly to node's tls.connect function - anything that function supports is supported here. Anything it doesn't support is out of scope of this library (I try to keep as much magic out as possible) & could probably be handled by another npm module you run your certs through before passing them to the config for node-postgres.

TomRed97

comment created time in 14 days

push eventbrianc/node-postgres

Brian M. Carlson

commit sha 7f06bc062e99b989316cdf4cb0d63029d93c1e2e

Update comments

view details

push time in 14 days

push eventbrianc/node-postgres

Brian M. Carlson

commit sha 73b940a1e21f199e4dfb337e20889450ea3f3572

Apply fix

view details

push time in 14 days

pull request commentbrianc/node-postgres

Fix double-sync crash on postgres 9.x

If there's truly a protocol break in older versions might make more sense to scrap the entire connection as who knows what else could be out of sync.

It's not trivial to trash a client and error out any pending and in-flight queries and stop processing other incoming messages w/ the current architecture. I think this might risk introducing more places things could go wrong and unexpected errors or behaviors, but I'll fiddle with it. I am thinking one "safer" (less invasive) fix could be to have an environment variable like PGSKIPDOUBLESYNC=true or something & folks can opt-in to the fix?

brianc

comment created time in 14 days

pull request commentbrianc/node-postgres

Fix double-sync crash on postgres 9.x

Okay here is the matrix of test runs which fail without the patch applied.

brianc

comment created time in 14 days

pull request commentbrianc/node-postgres

Fix double-sync crash on postgres 9.x

Still feels weird working around a potential server FEBE bug. If there's truly a protocol break in older versions might make more sense to scrap the entire connection as who knows what else could be out of sync.

Yeah I'm asking in #postgres on IRC to see if anyone on the postgres development team knows more about this. This isn't a bad idea, i'll look at doing this as well.

brianc

comment created time in 14 days

Pull request review commentbrianc/node-postgres

Fix double-sync crash on postgres 9.x

+const pg = require('../../../lib')+const helper = require('../test-helper')+const suite = new helper.Suite()++suite.testAsync('timeout causing query crashes', async () => {+  const client = new helper.Client()+  await client.connect()+  await client.query('CREATE TEMP TABLE foobar( name TEXT NOT NULL, id SERIAL)')+  client.query('BEGIN')

ah yeah that's a hold over when i thought it might have been related to the query queue (which is a horrible unfortunate API design mistake I made 10 years ago). So I initially wrote the test kinda abusing that feature thinking it was something I'd done. I just added await back here, thanks!

brianc

comment created time in 14 days

PullRequestReviewEvent

push eventbrianc/node-postgres

Brian M. Carlson

commit sha e006b039bff76e2a00bc515ffac50389988ea457

Remove fix to fail tests

view details

push time in 14 days

pull request commentbrianc/node-postgres

Fix double-sync crash on postgres 9.x

@charmander @sehrope any thoughts here? All tests are passing which is 👍 but it feels like I'm working around an issue or possible race condition w/in older versions of postgres itself?

brianc

comment created time in 15 days

issue commentbrianc/node-postgres

TypeError: Cannot read property 'name' of null

I have a PR up for this: #2367. The fix feels dirty to me, but I think what's happening here might be a race condition within postgres itself we're working around.

greghart

comment created time in 15 days

PR opened brianc/node-postgres

Fix double-sync crash on postgres 9.x

An attempt to fix #1105

It looks like in rare, random occasions postgres 9.x is sending both ErrorResponse and CommandComplete when a query times out. In an attempt to fix this, I have changed the query to only send a sync on an error message if it hasn't already sent a sync. Judging from the docs this should never happen, and indeed on newer versions of postgres the bug is not reproducible...but lots of folks still run postgres 9.x in production. I'm not super fond of this fix as it feels like it's working around a possible race condition in postgres 9.x so I'd love some additional eyes & ideas on this one. Also, want to run the test suite and make sure everything passes in the whole test matrix on travis.

One option is to put this behind some environment variable like PGSKIPDOUBLESYNC or something and allow people to opt in to this change. I'm worried about a (according to the docs, not possible) circumstance where we receive two messages requiring a sync and we only send one sync back.

+36 -3

0 comment

3 changed files

pr created time in 15 days

create barnchbrianc/node-postgres

branch : bmc/fix-double-sync-on-old-postgres

created branch time in 15 days

issue commentbrianc/node-postgres

importing as es6 module creates error

Should not load module IMHO.

I agree with you. bad api design from a long forgotten age of the wild west of node.js. I'll try to remove it in pg@9.0

bkilinc

comment created time in 16 days

issue commentbrianc/node-postgres

Keepalive the Idle postgres connections

oof I need to document that!

gopal-abbineni

comment created time in 16 days

issue commentbrianc/node-postgres

pg-protocol support for Postgres Server v13

Glad you got it sorted out!

mightypenguin

comment created time in 17 days

issue commentbrianc/node-postgres

pg-protocol support for Postgres Server v13

what code is triggering this error? Looks like a SASL connection routine issue...we do have test coverage for that, but we haven't added postgres 13 to the matrix yet.

mightypenguin

comment created time in 17 days

issue commentbrianc/node-postgres

pg-protocol support for Postgres Server v13

hmmm that's not good - do you have any code to reproduce this or more details? What's the key type error? Also, I don't think typescript should have anything to do w/ the error if its a postgres thing. I can set up postgres 13 locally & start developing against it and add it to the CI matrix. I just ran the test suite against it and everything worked fine on my end, so might be an issue w/ your code.

mightypenguin

comment created time in 17 days

push eventbrianc/node-postgres-docs

Dustin J. Mitchell

commit sha fbc004d6fcd3bb750bc054e43a6060df5b2c5dfe

Include 8.0 in the Upgrading section (#103) This is where I went looking for information about what broke in 8. @eschwartz pointed me to the Announcements, which do contain full details but were not a place I thought to look. This PR includes the most critical breaking change in the "Upgrading" section, with reference to the announcement for the details.

view details

push time in 17 days

PR merged brianc/node-postgres-docs

Include 8.0 in the Upgrading section

This is where I went looking for information about what broke in 8. @eschwartz pointed me to the Announcements, which do contain full details but were not a place I thought to look.

This PR includes the most critical breaking change in the "Upgrading" section, with reference to the announcement for the details.

Fixes https://github.com/brianc/node-postgres/issues/2342

+19 -1

2 comments

1 changed file

djmitche

pr closed time in 17 days

issue closedbrianc/node-postgres

Changes in 8.0?

I see a nice summary of what changes when upgrading to 7.0, but nothing about 8.0. #2047 has some discussion of what might or might not go into that release, but I don't see any indication there or elsewhere about what the breaking changes are. Are there any, aside form node versions? If so, could a new section of the docs describe those changes?

closed time in 17 days

djmitche

pull request commentbrianc/node-postgres-docs

Include 8.0 in the Upgrading section

Amazing! thank you for this!! Sorry I missed it to begin with when I wrote the upgrading instructions in the announcements section! 🤦

djmitche

comment created time in 17 days

push eventbrianc/node-postgres-docs

Adewunmi Akinsanya

commit sha 87ca1bd45b67fce5973b9c1a0ffecb1f564e34c8

Add async/await example to 1-project-structure.md (#104)

view details

push time in 17 days

pull request commentbrianc/node-postgres-docs

Add async/await example to 1-project-structure.md

awesome! thanks!

the-wunmi

comment created time in 17 days

pull request commentbrianc/node-postgres

Fix null reference in pg-cursor #handleError

Yeah if you can include a test case which reproduces the issue then we can look at the code, suggest any changes, etc...but almost without exception I don't merge code without tests. Having to maintain contributed code for years after its landed and the contributor has moved on is incredibly difficult without test coverage.

juneidysoo

comment created time in 17 days

issue commentbrianc/node-postgres

how to avoid pg to parse a result object

Is there another way to prevent parsing? I've got a query with an execution time of 12ms, but with ::text on the jsonb field it takes 143ms.

how big is the jsonb text in bytes? If you can generate a self contained code sample of this being slow I can take a look, but it can be anything from the throughput between your DB and your app server to the size of the json, etc. If you're returning the results as a string then there's no parsing at all other than the mandatory parsing of turning the raw packets of row data into strings to return.

budarin

comment created time in 17 days

issue commentbrianc/node-postgres

importing as es6 module creates error

There must be some reason, but I don't get it.

backwards compatibility with older versions of pg. Node postgres has been around quite a bit longer than webpack or babel...some of the API designs are crufty now but have remained because the cost to change them is large...with over a million installs a month any breaking changes must be weighed very carefully. I will consider removing this property in pg@9.0 but I'm not sure when that will be released. I agree this property isn't super useful and is causing problems while not helping out anything (other than maintaining backwards compatiblity).

if I need to use native driver I can simply; import pg from 'pg-native';

technically you would do import pg from 'pg/native' pg-native is a sub-package of pg that doesn't have the same API as the js client.

bkilinc

comment created time in 17 days

issue commentbrianc/node-postgres

DatabaseError class should be public

Release went out! pg@8.4.0

On Sun, Oct 4, 2020 at 3:52 PM Tom notifications@github.com wrote:

@garaboncias https://github.com/garaboncias just a heads up, the fix for this was just merged. Keep an eye out for the next release 😁

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/brianc/node-postgres/issues/2340#issuecomment-703314381, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAMHIPQDAUEIHZPC6K4IKTSJDN7VANCNFSM4REZI5IA .

garaboncias

comment created time in 18 days

push eventbrianc/node-postgres

Brian M. Carlson

commit sha 7ffe68eba056b9a6d0fa88f928aa85e768c28838

Publish - pg-connection-string@2.4.0 - pg-cursor@2.4.0 - pg-protocol@1.3.0 - pg-query-stream@3.3.0 - pg@8.4.0

view details

push time in 18 days

push eventbrianc/node-postgres

Brian M. Carlson

commit sha 125a2686e81f6c7d0892bc65289bc4ef4e3d9986

Update changelog

view details

push time in 18 days

push eventbrianc/node-postgres

dependabot[bot]

commit sha da2bb859873d25a37343a5b9238cc018ce026179

Bump node-fetch from 2.6.0 to 2.6.1 Bumps [node-fetch](https://github.com/bitinn/node-fetch) from 2.6.0 to 2.6.1. - [Release notes](https://github.com/bitinn/node-fetch/releases) - [Changelog](https://github.com/node-fetch/node-fetch/blob/master/docs/CHANGELOG.md) - [Commits](https://github.com/bitinn/node-fetch/compare/v2.6.0...v2.6.1) Signed-off-by: dependabot[bot] <support@github.com>

view details

push time in 18 days

PR merged brianc/node-postgres

Bump node-fetch from 2.6.0 to 2.6.1 dependencies

Bumps node-fetch from 2.6.0 to 2.6.1. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/bitinn/node-fetch/releases">node-fetch's releases</a>.</em></p> <blockquote> <h2>v2.6.1</h2> <p><strong>This is an important security release. It is strongly recommended to update as soon as possible.</strong></p> <p>See <a href="https://github.com/node-fetch/node-fetch/blob/master/docs/CHANGELOG.md#v261">CHANGELOG</a> for details.</p> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/node-fetch/node-fetch/blob/master/docs/CHANGELOG.md">node-fetch's changelog</a>.</em></p> <blockquote> <h2>v2.6.1</h2> <p><strong>This is an important security release. It is strongly recommended to update as soon as possible.</strong></p> <ul> <li>Fix: honor the <code>size</code> option after following a redirect.</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/node-fetch/node-fetch/commit/b5e2e41b2b50bf2997720d6125accaf0dd68c0ab"><code>b5e2e41</code></a> update version number</li> <li><a href="https://github.com/node-fetch/node-fetch/commit/2358a6c2563d1730a0cdaccc197c611949f6a334"><code>2358a6c</code></a> Honor the <code>size</code> option after following a redirect and revert data uri support</li> <li><a href="https://github.com/node-fetch/node-fetch/commit/8c197f8982a238b3c345c64b17bfa92e16b4f7c4"><code>8c197f8</code></a> docs: Fix typos and grammatical errors in README.md (<a href="https://github-redirect.dependabot.com/bitinn/node-fetch/issues/686">#686</a>)</li> <li><a href="https://github.com/node-fetch/node-fetch/commit/1e99050f944ac435fce26a9549eadcc2419a968a"><code>1e99050</code></a> fix: Change error message thrown with redirect mode set to error (<a href="https://github-redirect.dependabot.com/bitinn/node-fetch/issues/653">#653</a>)</li> <li><a href="https://github.com/node-fetch/node-fetch/commit/244e6f63d42025465796e3ca4ce813bf2c31fc5b"><code>244e6f6</code></a> docs: Show backers in README</li> <li><a href="https://github.com/node-fetch/node-fetch/commit/6a5d192034a0f438551dffb6d2d8df2c00921d16"><code>6a5d192</code></a> fix: Properly parse meta tag when parameters are reversed (<a href="https://github-redirect.dependabot.com/bitinn/node-fetch/issues/682">#682</a>)</li> <li><a href="https://github.com/node-fetch/node-fetch/commit/47a24a03eb49a49d81b768892aee10074ed54a91"><code>47a24a0</code></a> chore: Add opencollective badge</li> <li><a href="https://github.com/node-fetch/node-fetch/commit/7b136627c537cb24430b0310638c9177a85acee1"><code>7b13662</code></a> chore: Add funding link</li> <li><a href="https://github.com/node-fetch/node-fetch/commit/5535c2ed478d418969ecfd60c16453462de2a53f"><code>5535c2e</code></a> fix: Check for global.fetch before binding it (<a href="https://github-redirect.dependabot.com/bitinn/node-fetch/issues/674">#674</a>)</li> <li><a href="https://github.com/node-fetch/node-fetch/commit/1d5778ad0d910dbd1584fb407a186f5a0bc1ea22"><code>1d5778a</code></a> docs: Add Discord badge</li> <li>Additional commits viewable in <a href="https://github.com/bitinn/node-fetch/compare/v2.6.0...v2.6.1">compare view</a></li> </ul> </details> <details> <summary>Maintainer changes</summary> <p>This version was pushed to npm by <a href="https://www.npmjs.com/~akepinski">akepinski</a>, a new releaser for node-fetch since your current version.</p> </details> <br />

Dependabot compatibility score

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


<details> <summary>Dependabot commands and options</summary> <br />

You can trigger Dependabot actions by commenting on this PR:

  • @dependabot rebase will rebase this PR
  • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
  • @dependabot merge will merge this PR after your CI passes on it
  • @dependabot squash and merge will squash and merge this PR after your CI passes on it
  • @dependabot cancel merge will cancel a previously requested merge and block automerging
  • @dependabot reopen will reopen this PR if it is closed
  • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
  • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
  • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
  • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
  • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
  • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
  • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
  • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the Security Alerts page.

</details>

+3 -3

0 comment

1 changed file

dependabot[bot]

pr closed time in 18 days

push eventbrianc/node-postgres

Brian C

commit sha 7649890bfafbf4dea890975a2c26114d8d16fe60

Update SPONSORS.md

view details

push time in 18 days

PR merged brianc/node-postgres

Update SPONSORS.md

@ianwalter thank you again! ❤️

+1 -0

0 comment

1 changed file

brianc

pr closed time in 18 days

PR opened brianc/node-postgres

Update SPONSORS.md

@ianwalter thank you again! ❤️

+1 -0

0 comment

1 changed file

pr created time in 18 days

create barnchbrianc/node-postgres

branch : bmc/update-sponsors

created branch time in 18 days

issue commentbrianc/node-postgres

"timeout exceeded when trying to connect" spike after upgrading to 8.2.1

@EdenAzulay

I'm having the same issue when using version 8.3.3 . Do you know if there are any recent changes on the last releases that might cause the same issue?

What issue are you having? can you describe it a bit more? Did you upgrade from an older version without the issue? I don't know of any changes which may have caused the issue. There have been several reports on this thread, but nothing conclusive yet.

@PsyTae

Looks like we are experiencing this same issue. We are using a pool and pool.query() for all our db queries, so if you would like additional info we can provide, we can try anything you suggest as well.

Thanks for the info and it sounds like you cleared up your own issue? Did the issue only show up w/ pg@8 and not pg@7?

As an aside: I have a separate pool in our app for long running queries (we have a few that take several seconds) with longer connection timeouts and so on. It can help.

mriedem

comment created time in 18 days

pull request commentbrianc/node-postgres

Support idle_in_transaction_session_timeout and statement_timeout for native driver

Yo - sorry I was late to the party here. Left a question on here....could we include a test when both options and statement_timeout is included? Once we got that I can merge & release a new minor ver.

shayonj

comment created time in 18 days

Pull request review commentbrianc/node-postgres

Support idle_in_transaction_session_timeout and statement_timeout for native driver

 test('libpq connection string building', function () {     )   }) +  test('builds conn string with options', function () {

What happens if both options and statement_timeout is included in the config?

shayonj

comment created time in 18 days

PullRequestReviewEvent

push eventbrianc/node-postgres

Benjie Gillam

commit sha c5445f028840bd2407ce74e9bd253cadbfc7e669

Fix metadata for pg-connection-string

view details

push time in 18 days

PR merged brianc/node-postgres

Fix metadata for pg-connection-string

When you visit the pg-connection-string module on npmjs.com it links to the wrong place.

+2 -2

1 comment

1 changed file

benjie

pr closed time in 18 days

pull request commentbrianc/node-postgres

Fix metadata for pg-connection-string

thanks!

benjie

comment created time in 18 days

push eventbrianc/node-postgres

Bogdan Chadkin

commit sha a02dfac5ad2e2abf0dc3a9817f953938acdc19b1

Replace semver with optional peer dependencies See example https://github.com/sindresorhus/gulp-chown/blob/bb74168c957b3a94f122aafcecf7ebc87088ec46/package.json#L42-L49 This feature is supported by both npm and yarn.

view details

push time in 18 days

PR merged brianc/node-postgres

Replace semver with optional peer dependencies

See example https://github.com/sindresorhus/gulp-chown/blob/bb74168c957b3a94f122aafcecf7ebc87088ec46/package.json#L42-L49

This feature is supported by both npm and yarn.

+9 -13

3 comments

3 changed files

TrySound

pr closed time in 18 days

pull request commentbrianc/node-postgres

Replace semver with optional peer dependencies

Wow this is awesome, thank you.

TrySound

comment created time in 18 days

push eventbrianc/node-postgres

Tom Carrio

commit sha 58258430d52ee446721cc3e6611e26f8bcaa67f5

Public export of DatabaseError - Updated root exports of 'pg-protocol' to include DatabaseError Ref: #2340

view details

push time in 18 days

PR merged brianc/node-postgres

Public export of DatabaseError

Description

When handling errors from this package it would be better to have the errors publicly exported

Change Summary

  • Updated root exports of 'pg-protocol' to include DatabaseError

Ref: #2340

+2 -2

1 comment

1 changed file

tcarrio

pr closed time in 18 days

pull request commentbrianc/node-postgres

Public export of DatabaseError

👍 thanks for the PR!

tcarrio

comment created time in 18 days

push eventbrianc/node-postgres

Benjie Gillam

commit sha 6be3b9022f83efc721596cc41165afaa07bfceb0

Add support for ?sslmode connection string param

view details

Benjie Gillam

commit sha 9cbea21587330155e2d88b25d50fdb9fe081af1d

Solve issues caused by config.ssl = true

view details

Benjie Gillam

commit sha e421167d4631cf887960f44b477cafabffb2e7ee

Add ssl=true into the test

view details

push time in 18 days

PR merged brianc/node-postgres

Add support for ?sslmode connection string param

Fixes #1949

If you're using connection strings, you can't use the PGSSLMODE=no-verify envvar because that's only used when config.ssl is not set. I use ?ssl=true&sslmode=no-verify&sslrootcert=/app/data/amazon-rds-ca-cert.pem which does set config.ssl, so we need to also support parsing sslmode from the connection string. I've copied the logic (roughly) from:

https://github.com/brianc/node-postgres/blob/8291b233b81312ce2fbfce12ccd98ceceb53f5b9/packages/pg/lib/connection-parameters.js#L21-L32

I didn't see a CONTRIBUTING file, so I've taken a guess at the steps to make a good PR:

  • [x] Added fix
  • [x] Noted the GitHub issue it closes
  • [x] Added test cases / 100% code coverage
  • [x] Matched surrounding code style / passed linting
  • [x] Sponsor @brianc :sparkling_heart:
+64 -1

2 comments

2 changed files

benjie

pr closed time in 18 days

issue closedbrianc/node-postgres

Postgres `sslmode` parameter support

I can't figure out clearly, is sslmode parameter supported for connection strings? As I can see in pg-connection-string, sslmode isn't parsed, but I can see its mention in the codebase.

I use pg-promise in my project, and I have sslmode parameter in the Postgres connection string from Google Cloud. So I can't connect due to the database "dbname?sslmode=disable" does not exist, that's why I'm asking :)

closed time in 18 days

charlie-wasp

pull request commentbrianc/node-postgres

Add support for ?sslmode connection string param

Sorry for being so late on this. Went on a much needed vacation at the start of Sept & came back to a pretty disruptive life event. I just got out of the woods on that on Friday...finally. 🕺

This PR is perfect. I'm going to merge it & release a new minor version right now. ❤️

Also, really appreciate the sponsoring! ❇️

benjie

comment created time in 18 days

issue closedbrianc/node-postgres

AWS RDS and Extended Query Protocol

Hi! Amazing library, thanks for all of the hard work.

TLDR: The extended query protocol makes the AWS RDS Proxy entirely useless. Is there any way to turn off the extended query protocol in pg?

I have a serious issue I'm facing in a serverless environment. I'm using AWS Lambda, and directly connecting to postgres has caused horrific performance and stability issues because of the vast numbers of connections being opened and closed. So, the solution from AWS seems to be to use their RDS Proxy. It sits in front of postgres and handles a warm connection pool. I've set it all up properly. Unfortunately, all of my connections are being session pinned, essentially rendering the proxy useless, and it even seems to be performing worse than without the proxy. I've looked at my error logs and found this error repeatedly:

The client session was pinned to the database connection [dbConnection=251482716] for the remainder of the session. The proxy can't reuse this connection until the session ends. Reason: A parse message was detected.

Reason: A parse message was detected.. This seems to come because of the extended query protocol. Apparently the extended query protocol causes pinning, as documented here: https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/rds-proxy.html#rds-proxy-pinning

So my real question is, is there any way to turn off the extended query protocol in pg? If not, does anyone have any insight into how I can properly manage connections to my database in a serverless environment? This is a major issue for us at this time, and I'm not sure how to proceed

closed time in 18 days

lastmjs

issue commentbrianc/node-postgres

AWS RDS and Extended Query Protocol

But in node-postgres and I imagine other clients, parameterized queries are generally implemented as prepared statements.

There is no difference at the protocol level, actually. Any query with parameters (whether or not you call it a prepared statement) requires a parse/bind/describe/execute series of messages. Either way it "prepares" the query during the parse phase. Sorry the RDS proxy is so picky about parameterized queries...that's quite limiting as doing parameters in queries with string concatenation and client side sanitization is not ideal. Let me know if there's anything you can think of from this libraries side to help...but sounds like an RDS issue for now.

lastmjs

comment created time in 18 days

issue commentbrianc/node-postgres

Feature: Memory limits

Please feel free to submit a pull request with proper test coverage & I'll be happy to take a look.

pauldraper

comment created time in a month

issue closedbrianc/node-postgres

Feature: Memory limits

I'd like a large result to produce an Error (which can be handled), rather than crashing the entire process with OutOfMemory.

This is similar in nature to limit I/O for HTTP requests:

const express = require('express')
const bodyParser = require('body-parser')

const app = express()
app.use(bodyParser.json({ limit: '5mb' }))

I imagine an option to Connection/Pool like maxResultSize. When the result exceeds the configured number of bytes, an Error is thrown. (I understand this limit would be subject to the choice of binary and text formats...this isn't super precise, just a way to gracefully handle unexpectedly large results.)

This would significantly improve the stability of multi-operation processes using this library.

closed time in a month

pauldraper

issue commentbrianc/node-postgres

Feature: Memory limits

If it's the total result set, have you considered using a cursor? That would let you operate on the row in chunks without needing to allocate memory for the entire result set. If you have commands that could return an arbitrary / huge number of rows, it's the way to go to ensure you do not blow up the node process memory usage.

Yup, 100% this is the way to handle this! The postgres protocol supports this, and node is obvs build w/ streaming in mind. This library already exists as well. We use pg and regularly stream 500-600 megabytes of results down to clients w/o issue using cursors.

As far a single row taking up more memory than your server...you should fix your data in that case. 🤷

One major difference between a db client & a web server & why body parser accepts a limit is a web server needs to protect you from bad actors sending you gigabytes in a post request and DDoSing you. Generally, you control both your web server & database client, so you are just DDoSing yourself in this case. Even if the postgres client threw an error (and forcibly disconnected) the database server has already gone through the work of shoving hundreds of megabytes into the network socket.

pauldraper

comment created time in a month

issue commentbrianc/node-postgres

switch docs/tests to use jsonstream-next

Hey totally open to a PR on this if you wanna try it out! Thanks for letting me know about this. Since it's only used in tests its not suuuper high priority but always nice to keep code hygiene and continuity in mind.

contra

comment created time in 2 months

more