Problem-space

• password already has this set, but was a little long considering we only want to override default of one property
• ssl.key was showing up in tracebacks

This came up locally when disconnecting a client after connection, but before close

Emitted 'error' event on BoundPool instance at:
    at Client.idleListener (C:\Users\lewis\projects\nodejs-app\node_modules\pg-pool\index.js:57:10)
    at Client.emit (events.js:321:20)
    at connectedErrorHandler (C:\Users\lewis\projects\nodejs-app\node_modules\pg\lib\client.js:221:10)
    at Connection.<anonymous> (C:\Users\lewis\projects\nodejs-app\node_modules\pg\lib\client.js:289:9)
    at Object.onceWrapper (events.js:427:28)
    [... lines matching original stack trace ...]
    at processTicksAndRejections (internal/process/task_queues.js:84:21) {
  client: Client {

with

key: '-----BEGIN RSA PRIVATE KEY-----\n' +
        'MIIEpAIBAAKCAQEAxAnztpOXj+6OuEONaUkpW2CPxCOoKyuzlsF5PTcjETHGgSN+\n' +
        ...
        'qq/1uRgkxeakQ9gl9sXL94+ALu8bHKFVReZbSsdYEcdJwdEH5F/4HA==\n' +
        '-----END RSA PRIVATE KEY-----\n'

Secrets in logs are not great.

Temporary fix

Without this code, you can already make the key non-enumerable. This just passes that responsibility into the library so that it does not impact users who don't know about this.

today not going well i'm sorry - will try again tomorrow 😭

Bumps http-proxy from 1.17.0 to 1.18.1. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/http-party/node-http-proxy/blob/master/CHANGELOG.md">http-proxy's changelog</a>.</em></p> <blockquote> <h2><a href="https://github.com/http-party/node-http-proxy/compare/1.18.0...v1.18.1">v1.18.1</a> - 2020-05-17</h2> <h3>Merged</h3> <ul> <li>Skip sending the proxyReq event when the expect header is present <a href="https://github-redirect.dependabot.com/http-party/node-http-proxy/pull/1447"><code>#1447</code></a></li> <li>Remove node6 support, add node12 to build <a href="https://github-redirect.dependabot.com/http-party/node-http-proxy/pull/1397"><code>#1397</code></a></li> </ul> <h2><a href="https://github.com/http-party/node-http-proxy/compare/1.17.0...1.18.0">1.18.0</a> - 2019-09-18</h2> <h3>Merged</h3> <ul> <li>Added in auto-changelog module set to keepachangelog format <a href="https://github-redirect.dependabot.com/http-party/node-http-proxy/pull/1373"><code>#1373</code></a></li> <li>fix 'Modify Response' readme section to avoid unnecessary array copying <a href="https://github-redirect.dependabot.com/http-party/node-http-proxy/pull/1300"><code>#1300</code></a></li> <li>Fix incorrect target name for reverse proxy example <a href="https://github-redirect.dependabot.com/http-party/node-http-proxy/pull/1135"><code>#1135</code></a></li> <li>Fix modify response middleware example <a href="https://github-redirect.dependabot.com/http-party/node-http-proxy/pull/1139"><code>#1139</code></a></li> <li>[dist] Update dependency async to v3 <a href="https://github-redirect.dependabot.com/http-party/node-http-proxy/pull/1359"><code>#1359</code></a></li> <li>Fix path to local http-proxy in examples. <a href="https://github-redirect.dependabot.com/http-party/node-http-proxy/pull/1072"><code>#1072</code></a></li> <li>fix reverse-proxy example require path <a href="https://github-redirect.dependabot.com/http-party/node-http-proxy/pull/1067"><code>#1067</code></a></li> <li>Update README.md <a href="https://github-redirect.dependabot.com/http-party/node-http-proxy/pull/970"><code>#970</code></a></li> <li>[dist] Update dependency request to ~2.88.0 [SECURITY] <a href="https://github-redirect.dependabot.com/http-party/node-http-proxy/pull/1357"><code>#1357</code></a></li> <li>[dist] Update dependency eventemitter3 to v4 <a href="https://github-redirect.dependabot.com/http-party/node-http-proxy/pull/1365"><code>#1365</code></a></li> <li>[dist] Update dependency colors to v1 <a href="https://github-redirect.dependabot.com/http-party/node-http-proxy/pull/1360"><code>#1360</code></a></li> <li>[dist] Update all non-major dependencies <a href="https://github-redirect.dependabot.com/http-party/node-http-proxy/pull/1356"><code>#1356</code></a></li> <li>[dist] Update dependency agentkeepalive to v4 <a href="https://github-redirect.dependabot.com/http-party/node-http-proxy/pull/1358"><code>#1358</code></a></li> <li>[dist] Update dependency nyc to v14 <a href="https://github-redirect.dependabot.com/http-party/node-http-proxy/pull/1367"><code>#1367</code></a></li> <li>[dist] Update dependency concat-stream to v2 <a href="https://github-redirect.dependabot.com/http-party/node-http-proxy/pull/1363"><code>#1363</code></a></li> <li>x-forwarded-host overwrite for mutli level proxies <a href="https://github-redirect.dependabot.com/http-party/node-http-proxy/pull/1267"><code>#1267</code></a></li> <li>[refactor doc] Complete rename to http-party org. <a href="https://github-redirect.dependabot.com/http-party/node-http-proxy/pull/1362"><code>#1362</code></a></li> <li>Highlight correct lines for createProxyServer <a href="https://github-redirect.dependabot.com/http-party/node-http-proxy/pull/1117"><code>#1117</code></a></li> <li>Fix docs for rewrite options - 201 also handled <a href="https://github-redirect.dependabot.com/http-party/node-http-proxy/pull/1147"><code>#1147</code></a></li> <li>Update .nyc_output <a href="https://github-redirect.dependabot.com/http-party/node-http-proxy/pull/1339"><code>#1339</code></a></li> <li>Configure Renovate <a href="https://github-redirect.dependabot.com/http-party/node-http-proxy/pull/1355"><code>#1355</code></a></li> <li>[examples] Restream body before proxying, support for Content-Type of application/x-www-form-urlencoded <a href="https://github-redirect.dependabot.com/http-party/node-http-proxy/pull/1264"><code>#1264</code></a></li> </ul> <h3>Commits</h3> <ul> <li>[dist] New test fixtures. <a href="https://github.com/http-party/node-http-proxy/commit/7e4a0e511bc30c059216860153301de2cdd1e97f"><code>7e4a0e5</code></a></li> <li>[dist] End of an era. <a href="https://github.com/http-party/node-http-proxy/commit/a9b09cce43f072db99fb5170030a05536177ccb7"><code>a9b09cc</code></a></li> <li>[dist] Version bump. 1.18.0 <a href="https://github.com/http-party/node-http-proxy/commit/9bbe486c5efcc356fb4d189ef38eee275bbde345"><code>9bbe486</code></a></li> <li>[fix] Latest versions. <a href="https://github.com/http-party/node-http-proxy/commit/59c4403e9dc15ab9b19ee2a3f4aecbfc6c3d94c4"><code>59c4403</code></a></li> <li>[fix test] Update tests. <a href="https://github.com/http-party/node-http-proxy/commit/dd1d08b6319d1def729554446a5b0176978a8dad"><code>dd1d08b</code></a></li> <li>[dist] Update dependency ws to v3 [SECURITY] <a href="https://github.com/http-party/node-http-proxy/commit/b00911c93740a00c5cfbacbb91565cb6912ed255"><code>b00911c</code></a></li> <li>[dist] .gitattributes all the things. <a href="https://github.com/http-party/node-http-proxy/commit/fc93520d741ec80be8ae31ca005f3e9c199e330e"><code>fc93520</code></a></li> <li>[dist] Regenerate package-lock.json. <a href="https://github.com/http-party/node-http-proxy/commit/16d4f8a95162b2e2e4ee6657c500f1208c044b2d"><code>16d4f8a</code></a></li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/http-party/node-http-proxy/commit/9b96cd725127a024dabebec6c7ea8c807272223d"><code>9b96cd7</code></a> 1.18.1</li> <li><a href="https://github.com/http-party/node-http-proxy/commit/335aeeba2f0c286dc89c402eeb76af47834c89a3"><code>335aeeb</code></a> Skip sending the proxyReq event when the expect header is present (<a href="https://github-redirect.dependabot.com/http-party/node-http-proxy/issues/1447">#1447</a>)</li> <li><a href="https://github.com/http-party/node-http-proxy/commit/dba39668ba4c9ad461316e834b2d64b77e1ca88e"><code>dba3966</code></a> Remove node6 support, add node12 to build (<a href="https://github-redirect.dependabot.com/http-party/node-http-proxy/issues/1397">#1397</a>)</li> <li><a href="https://github.com/http-party/node-http-proxy/commit/9bbe486c5efcc356fb4d189ef38eee275bbde345"><code>9bbe486</code></a> [dist] Version bump. 1.18.0</li> <li><a href="https://github.com/http-party/node-http-proxy/commit/6e4bef4d1cd96e7a284717941e0fc274acbd3712"><code>6e4bef4</code></a> Added in auto-changelog module set to keepachangelog format (<a href="https://github-redirect.dependabot.com/http-party/node-http-proxy/issues/1373">#1373</a>)</li> <li><a href="https://github.com/http-party/node-http-proxy/commit/d05624167ce75e860770c13afeacec2ce0f67add"><code>d056241</code></a> fix 'Modify Response' readme section to avoid unnecessary array copying (<a href="https://github-redirect.dependabot.com/http-party/node-http-proxy/issues/1300">#1300</a>)</li> <li><a href="https://github.com/http-party/node-http-proxy/commit/244303b994525684e1ec8dff2e8055f89b62b1ee"><code>244303b</code></a> Fix incorrect target name for reverse proxy example (<a href="https://github-redirect.dependabot.com/http-party/node-http-proxy/issues/1135">#1135</a>)</li> <li><a href="https://github.com/http-party/node-http-proxy/commit/b4028ba78bc4616e6969e0e66b0fe4634849b68b"><code>b4028ba</code></a> Fix modify response middleware example (<a href="https://github-redirect.dependabot.com/http-party/node-http-proxy/issues/1139">#1139</a>)</li> <li><a href="https://github.com/http-party/node-http-proxy/commit/77a98159d2da0f20a03e2819c79662f36069f234"><code>77a9815</code></a> [dist] Update dependency async to v3 (<a href="https://github-redirect.dependabot.com/http-party/node-http-proxy/issues/1359">#1359</a>)</li> <li><a href="https://github.com/http-party/node-http-proxy/commit/c662f9ebcd8d623db374dbc7bef231b2b0af0c3a"><code>c662f9e</code></a> Fix path to local http-proxy in examples. (<a href="https://github-redirect.dependabot.com/http-party/node-http-proxy/issues/1072">#1072</a>)</li> <li>Additional commits viewable in <a href="https://github.com/http-party/node-http-proxy/compare/1.17.0...1.18.1">compare view</a></li> </ul> </details> <br />

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

<details> <summary>Dependabot commands and options</summary> <br />

You can trigger Dependabot actions by commenting on this PR:

• @dependabot rebase will rebase this PR
• @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
• @dependabot merge will merge this PR after your CI passes on it
• @dependabot squash and merge will squash and merge this PR after your CI passes on it
• @dependabot cancel merge will cancel a previously requested merge and block automerging
• @dependabot reopen will reopen this PR if it is closed
• @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
• @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
• @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
• @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
• @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
• @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the Security Alerts page.

</details>

Simple documentation for the pass-through connection parameter added in https://github.com/brianc/node-postgres/pull/2049.

The description is taken from https://www.postgresql.org/docs/current/runtime-config-client.html.

Closes https://github.com/brianc/node-postgres/issues/2381

Hello,

I'm having trouble passing this config into the client: idle_in_transaction_session_timeout implemented here: https://github.com/brianc/node-postgres/pull/2049

However I couldn't figure out where to pass it in given the starting example of

const { Pool } = require('pg');
const pool = new Pool();

pool.query('SQL...')

I looked into the config document and did not see idle_in_transaction_session_timeout as a documented property of the PoolConfig type

It would be nice to have the available options documented here: https://node-postgres.com/api/pool#constructor

Thanks

nice! thanks!

if you want two pools it's pretty straight forward:

const pool1 = new pg.Pool({ database: 'database1' })
const pool2 = new pg.Pool({ database: database2' })

Up to you in your app to decide which pool to use.

As far as serverless + pgbouncer...it should work though I don't run pgbouncer myself in production.

Also is it possible to execute multiple queries in the same statement?

You can only run multiple queries in a single statement if you do not use parameters. This is a postgres protocol restriction & there's no working around it. 99% of the time just do multiple statements w/ async/await

const res1 = await pool.query('SELECT 1')
const res2 = await pool.query('SELECT 2')

If you absolutely need to execute multiple statements in the same query, don't use parameters:

const res = await pool.query('SELECT 1; SELECT 2;).

yo - i can merge this. I'll do so first thing tomorrow (I like to be around all day in case issues come up). Having to deal w/ some personal tragedy this week....our dog of 11 years is getting euthanized this week....so there might be some delays depending on how things go. 😭

Im out of town until Monday. I will look then.

On Fri, Oct 16, 2020 at 6:08 AM Michael Salinger [email protected] wrote:

@brianc https://github.com/brianc Can this be merged?

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/brianc/node-postgres/pull/2372#issuecomment-709981149, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAMHIJYYYNPJM3PU4PL7HLSLASTRANCNFSM4SKRO3RA .

@chyzwar what did you want me to label this pull request with? I think this looks good to merge.

yup that looks correct! I should make some first class getters to return that info, really...but that'll work for now! Good luck with your upgrade & lmk if you have any other questions.

I think dependabot is nice but it would raise PR for versions of packages that no longer support node v8. For example lestest ts-node and mocha no longer suport node v8.

hmm that might be hard to deal with, any suggestions?

it will be nice to turn this to true one day, but this is fine for the initial port.

so you want the main pg package to export DatabaseError? that sounds reasonable

You're joining 3 tables, each with an id field. pg tries to map the column names to an object. In the event of a column name collision it doesn't know what to do. Try something like this:

SELECT task.id as "taskId", room.id as "roomId"

etc - then the names will map into the object without collision. You could also try

client.query({  text: 'SELECT * ....', rowMode: 'array' })

to return an array of fields and map it into an object however you'd like

It's just type exposure through variable, why do you need tests for that?

Otherwise in the future it might get unexposed & then we wouldn't know we're breaking anything until we release a version w/o it exposed. So just a simple check to make sure it's there is fine.

Yeah pg<=8.0 is not compatible with node>=14

I like the amount of tests per the level of lines of library code changed & appreciate your contribution! Since @charmander previously reviewed I'll leave it up to them to make the final call. If you don't hear back from either of us in a week or so LMK & I can take over and drive this across the finish line but want to give charmander some time...we're working for free after all so sometimes its a bit slower but that's how it goes!

Thanks for this - this seems reasonable, but could you write some tests which will both prevent this from regressing & also demonstrate how this would be useful?

@felixfbecker

It would be amazing if Cursor could implement Symbol.asyncIterator so that it can be iterated with for await (const row of cursor) {}.

For this you should probably use pg-query-stream which already supports this. pg-cursor is used by pg-query-stream. I'm definitely cool with promisifying the cursor, but really the cursor is a bit lower level than pg-query-stream and can be useful primarily when you want to read exactly a certain number of rows at a time in chunks to carefully control how you're consuming. pg-query-stream is probably more generally useful for "pipe this query to this other thing" but due to high water marks, back pressure, and internal buffering in node streams you don't have as fine grained control over the exact number of rows read. In practice this is usually fine, and kinda preferred as you usually don't need to think about things at that level...so would recommend pg-query-stream for most use cases, including async iteration.

Greetings,

I am running 30 nodejs instances in AWS talking to a postgreSQL rds instance. Each nodejs instance is using pg and a pg_pool for all database interaction. All calls are using simply pg_pool.query to interact with the pool.

Everything runs along well for several hours - we watch the waitingCount, idleCount and runningCount during execution and the values rise and fall nicely during the run.

As some point in time however things go off the rails. The totalCount rises to the limit of 90, the idleCount remains at zero and the waitingCount continues to rise. (up to and over 100,000).

The netstat command on the ec2 instances show they have 90 sessions established to the database server - however pg_stat_activity on the postgreSQL server indicates no queries are active from the instance.

It appears the connections are idle - however the pool does not seem to be placing them back into the pool for the waiting entries?

NodeJS Version: 4.2.6 pg Version: 6.4.2 pg-pool: 2.0.10 PostgreSQL Version 9.5.15 RDS instance size: db.r4.16xlarge

We are creating a single global pool - and are handling error, remove, connect and aquire messages.
Have the pool configured with an connectionTimeoutMillis: 60000 and idleTimeoutMillis: 120000 Using the bluebird promise.

Am I actually losing connections in the pool?

As an aside - eventually the postgreSQL rds instance eventually runs out of freeableMemory and crashes. AWS claims the crash is caused by excessive load.

@pcbentz - charmander is right you're running pretty old stuff. It's hard to say if the bugs are related to your code, version of node, or older version of PG....a few things I can recommend, some of which you probably already thought of.

Make sure you have a single place in your app to do queries & make sure all the clients are returned to the pool...the most common issue of starved connection pools is not returning the client to the pool. At the very bottom of this page I put a little bit of info on how you could possibly track down unreturned clients.

As an aside - eventually the postgreSQL rds instance eventually runs out of freeableMemory and crashes. AWS claims the crash is caused by excessive load.

If your instance is running out of memory but your sessions connected to the database is remaining steady at 90 then it's likely something fishy going on with your actual database. Connections to postgres backend use a non-trivial amount of ram because postgres uses a threaded model - but it's something like 4 megabytes per connection. In the distant past I've crashed a postgres instance on an 4xl instance in ec2 but we connected like 3,000+ connections to it at once (oops!). Anyways, if you only have 90 connections steadily open to your database and it's running out of freeable memory, something else is going on there...possibly some runaway query or something else on the box.

The final thing you can try doing, if upgrading pg & node isn't an option for you (I run apps in prod too, I know it can be a pain) is using something like https://www.npmjs.com/package/generic-pool and implement your own pool if you suspect the issue is in pg-pool. You can have fine-grained control over acquire and release semantics there & do any sort of instrumentation you need.

Hope this helps!

@pckilgore my pleasure! Let me know if it still happens...was quite a monster to track down but @timotm is the real hero here. Steps to reproduce & deep analysis helped me know where to focus. I regret it took me so long to circle back to this issue.

An attempt to fix #1105

It looks like in rare, random occasions postgres 9.x is sending both ErrorResponse and CommandComplete when a query times out. In an attempt to fix this, I have changed the query to only send a sync on an error message if it hasn't already sent a sync. Judging from the docs this should never happen, and indeed on newer versions of postgres the bug is not reproducible...but lots of folks still run postgres 9.x in production. I'm not super fond of this fix as it feels like it's working around a possible race condition in postgres 9.x so I'd love some additional eyes & ideas on this one. Also, want to run the test suite and make sure everything passes in the whole test matrix on travis.

One option is to put this behind some environment variable like PGSKIPDOUBLESYNC or something and allow people to opt in to this change. I'm worried about a (according to the docs, not possible) circumstance where we receive two messages requiring a sync and we only send one sync back.

First off, thanks for the library! Overall it's been incredibly performant and straightforward to use. I am running into one issue which may be an issue with the library, though it's a bit hard to follow the protocol for me so my usage may be off as well.

I'm running into an uncaught error during the parseComplete handler in client.js (https://github.com/brianc/node-postgres/blob/v6.0.1/lib/client.js#L131).

I am doing a direct connect against the pool and then utilizing pg-query-stream

pool.connect().then(function(client){
  stream = client.query(new QueryStream(query, null));

  stream.on('error', function(streamError){
    stream.close(function(closeError){ 
      // What to do with a close error? At this point stream is already hosed :shrug:
      client.release(streamError);
    }
  });

  stream.on('close', function(){
    client.release();
  }); 
});

At some point during my app's life cycle, I get an uncaught error

TypeError: Cannot read property 'name' of null
    at null.<anonymous> (/var/app/current/node_modules/pg/lib/client.js:139:26)
    at emit (events.js:107:17)
    at Socket.<anonymous> (/var/app/current/node_modules/pg/lib/connection.js:121:12)
    at Socket.emit (events.js:107:17)
    at readableAddChunk (_stream_readable.js:163:16)
    at Socket.Readable.push (_stream_readable.js:126:10)
    at TCP.onread (net.js:540:20)

So obviously activeQuery is becoming null, and I am trying to narrow down why I would receive a parseComplete event when the active query is null.

According to my logs, the stream is not emitting an error, and seems to be closing normally. Therefore it seems like the connection client is getting a readyForQuery or end event, and then right after that getting the parseComplete event. Any idea why this would be happening, or see any issues with my usage? Thanks for any help you can give! I'll keep looking into it as well.

okay...so w/ some help from @sehrope and a postgres maintainer from IRC I've changed the approach somewhat. When discussing the maintainer (I believe their github name is @rhodiumtoad) suggested to pipeline the sync message at the end of the other pipelined messages. Sending sync immediately after execute allows the backend to process all the messages and only respond with either CommandComplete or ErrorResponse. This fixes the issue in the test w/o requiring any hacks or discarding messages. It also speeds up some of my crude benchmarks by over 10% which is a super awesome bonus side effect.

@pvatterott that's good to know, but a bummer as my hunch was around a race condition in older versions of postgres. I've been going back and forth w/ some postgres maintainers & think this might fix it. I can confirm it fixes in 9.x & 10.x of postgres, it might fix it for 12.3 of postgres as well. The change involves pipelining the sync command in with the other extended query commands all at once. It also improves my benchmarks by up to 10% in some cases, which is a nice side effect.

The ssl config is forwarded unmodified directly to node's tls.connect function - anything that function supports is supported here. Anything it doesn't support is out of scope of this library (I try to keep as much magic out as possible) & could probably be handled by another npm module you run your certs through before passing them to the config for node-postgres.

If there's truly a protocol break in older versions might make more sense to scrap the entire connection as who knows what else could be out of sync.

It's not trivial to trash a client and error out any pending and in-flight queries and stop processing other incoming messages w/ the current architecture. I think this might risk introducing more places things could go wrong and unexpected errors or behaviors, but I'll fiddle with it. I am thinking one "safer" (less invasive) fix could be to have an environment variable like PGSKIPDOUBLESYNC=true or something & folks can opt-in to the fix?

Okay here is the matrix of test runs which fail without the patch applied.

Still feels weird working around a potential server FEBE bug. If there's truly a protocol break in older versions might make more sense to scrap the entire connection as who knows what else could be out of sync.

Yeah I'm asking in #postgres on IRC to see if anyone on the postgres development team knows more about this. This isn't a bad idea, i'll look at doing this as well.

ah yeah that's a hold over when i thought it might have been related to the query queue (which is a horrible unfortunate API design mistake I made 10 years ago). So I initially wrote the test kinda abusing that feature thinking it was something I'd done. I just added await back here, thanks!

@charmander @sehrope any thoughts here? All tests are passing which is 👍 but it feels like I'm working around an issue or possible race condition w/in older versions of postgres itself?

I have a PR up for this: #2367. The fix feels dirty to me, but I think what's happening here might be a race condition within postgres itself we're working around.

An attempt to fix #1105

It looks like in rare, random occasions postgres 9.x is sending both ErrorResponse and CommandComplete when a query times out. In an attempt to fix this, I have changed the query to only send a sync on an error message if it hasn't already sent a sync. Judging from the docs this should never happen, and indeed on newer versions of postgres the bug is not reproducible...but lots of folks still run postgres 9.x in production. I'm not super fond of this fix as it feels like it's working around a possible race condition in postgres 9.x so I'd love some additional eyes & ideas on this one. Also, want to run the test suite and make sure everything passes in the whole test matrix on travis.

One option is to put this behind some environment variable like PGSKIPDOUBLESYNC or something and allow people to opt in to this change. I'm worried about a (according to the docs, not possible) circumstance where we receive two messages requiring a sync and we only send one sync back.

Should not load module IMHO.

I agree with you. bad api design from a long forgotten age of the wild west of node.js. I'll try to remove it in [email protected]

oof I need to document that!

Glad you got it sorted out!

what code is triggering this error? Looks like a SASL connection routine issue...we do have test coverage for that, but we haven't added postgres 13 to the matrix yet.

hmmm that's not good - do you have any code to reproduce this or more details? What's the key type error? Also, I don't think typescript should have anything to do w/ the error if its a postgres thing. I can set up postgres 13 locally & start developing against it and add it to the CI matrix. I just ran the test suite against it and everything worked fine on my end, so might be an issue w/ your code.

This is where I went looking for information about what broke in 8. @eschwartz pointed me to the Announcements, which do contain full details but were not a place I thought to look.

This PR includes the most critical breaking change in the "Upgrading" section, with reference to the announcement for the details.

Fixes https://github.com/brianc/node-postgres/issues/2342

I see a nice summary of what changes when upgrading to 7.0, but nothing about 8.0. #2047 has some discussion of what might or might not go into that release, but I don't see any indication there or elsewhere about what the breaking changes are. Are there any, aside form node versions? If so, could a new section of the docs describe those changes?

Amazing! thank you for this!! Sorry I missed it to begin with when I wrote the upgrading instructions in the announcements section! 🤦

awesome! thanks!

Yeah if you can include a test case which reproduces the issue then we can look at the code, suggest any changes, etc...but almost without exception I don't merge code without tests. Having to maintain contributed code for years after its landed and the contributor has moved on is incredibly difficult without test coverage.

Is there another way to prevent parsing? I've got a query with an execution time of 12ms, but with ::text on the jsonb field it takes 143ms.

how big is the jsonb text in bytes? If you can generate a self contained code sample of this being slow I can take a look, but it can be anything from the throughput between your DB and your app server to the size of the json, etc. If you're returning the results as a string then there's no parsing at all other than the mandatory parsing of turning the raw packets of row data into strings to return.

There must be some reason, but I don't get it.

backwards compatibility with older versions of pg. Node postgres has been around quite a bit longer than webpack or babel...some of the API designs are crufty now but have remained because the cost to change them is large...with over a million installs a month any breaking changes must be weighed very carefully. I will consider removing this property in [email protected] but I'm not sure when that will be released. I agree this property isn't super useful and is causing problems while not helping out anything (other than maintaining backwards compatiblity).

if I need to use native driver I can simply; import pg from 'pg-native';

technically you would do import pg from 'pg/native' pg-native is a sub-package of pg that doesn't have the same API as the js client.

Release went out! [email protected]

On Sun, Oct 4, 2020 at 3:52 PM Tom [email protected] wrote:

@garaboncias https://github.com/garaboncias just a heads up, the fix for this was just merged. Keep an eye out for the next release 😁

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/brianc/node-postgres/issues/2340#issuecomment-703314381, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAMHIPQDAUEIHZPC6K4IKTSJDN7VANCNFSM4REZI5IA .

Bumps node-fetch from 2.6.0 to 2.6.1. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/bitinn/node-fetch/releases">node-fetch's releases</a>.</em></p> <blockquote> <h2>v2.6.1</h2> <p><strong>This is an important security release. It is strongly recommended to update as soon as possible.</strong></p> <p>See <a href="https://github.com/node-fetch/node-fetch/blob/master/docs/CHANGELOG.md#v261">CHANGELOG</a> for details.</p> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/node-fetch/node-fetch/blob/master/docs/CHANGELOG.md">node-fetch's changelog</a>.</em></p> <blockquote> <h2>v2.6.1</h2> <p><strong>This is an important security release. It is strongly recommended to update as soon as possible.</strong></p> <ul> <li>Fix: honor the <code>size</code> option after following a redirect.</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/node-fetch/node-fetch/commit/b5e2e41b2b50bf2997720d6125accaf0dd68c0ab"><code>b5e2e41</code></a> update version number</li> <li><a href="https://github.com/node-fetch/node-fetch/commit/2358a6c2563d1730a0cdaccc197c611949f6a334"><code>2358a6c</code></a> Honor the <code>size</code> option after following a redirect and revert data uri support</li> <li><a href="https://github.com/node-fetch/node-fetch/commit/8c197f8982a238b3c345c64b17bfa92e16b4f7c4"><code>8c197f8</code></a> docs: Fix typos and grammatical errors in README.md (<a href="https://github-redirect.dependabot.com/bitinn/node-fetch/issues/686">#686</a>)</li> <li><a href="https://github.com/node-fetch/node-fetch/commit/1e99050f944ac435fce26a9549eadcc2419a968a"><code>1e99050</code></a> fix: Change error message thrown with redirect mode set to error (<a href="https://github-redirect.dependabot.com/bitinn/node-fetch/issues/653">#653</a>)</li> <li><a href="https://github.com/node-fetch/node-fetch/commit/244e6f63d42025465796e3ca4ce813bf2c31fc5b"><code>244e6f6</code></a> docs: Show backers in README</li> <li><a href="https://github.com/node-fetch/node-fetch/commit/6a5d192034a0f438551dffb6d2d8df2c00921d16"><code>6a5d192</code></a> fix: Properly parse meta tag when parameters are reversed (<a href="https://github-redirect.dependabot.com/bitinn/node-fetch/issues/682">#682</a>)</li> <li><a href="https://github.com/node-fetch/node-fetch/commit/47a24a03eb49a49d81b768892aee10074ed54a91"><code>47a24a0</code></a> chore: Add opencollective badge</li> <li><a href="https://github.com/node-fetch/node-fetch/commit/7b136627c537cb24430b0310638c9177a85acee1"><code>7b13662</code></a> chore: Add funding link</li> <li><a href="https://github.com/node-fetch/node-fetch/commit/5535c2ed478d418969ecfd60c16453462de2a53f"><code>5535c2e</code></a> fix: Check for global.fetch before binding it (<a href="https://github-redirect.dependabot.com/bitinn/node-fetch/issues/674">#674</a>)</li> <li><a href="https://github.com/node-fetch/node-fetch/commit/1d5778ad0d910dbd1584fb407a186f5a0bc1ea22"><code>1d5778a</code></a> docs: Add Discord badge</li> <li>Additional commits viewable in <a href="https://github.com/bitinn/node-fetch/compare/v2.6.0...v2.6.1">compare view</a></li> </ul> </details> <details> <summary>Maintainer changes</summary> <p>This version was pushed to npm by <a href="https://www.npmjs.com/~akepinski">akepinski</a>, a new releaser for node-fetch since your current version.</p> </details> <br />

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

<details> <summary>Dependabot commands and options</summary> <br />

You can trigger Dependabot actions by commenting on this PR:

• @dependabot rebase will rebase this PR
• @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
• @dependabot merge will merge this PR after your CI passes on it
• @dependabot squash and merge will squash and merge this PR after your CI passes on it
• @dependabot cancel merge will cancel a previously requested merge and block automerging
• @dependabot reopen will reopen this PR if it is closed
• @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
• @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
• @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
• @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
• @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
• @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the Security Alerts page.

</details>

@ianwalter thank you again! ❤️

@ianwalter thank you again! ❤️

@EdenAzulay

I'm having the same issue when using version 8.3.3 . Do you know if there are any recent changes on the last releases that might cause the same issue?

What issue are you having? can you describe it a bit more? Did you upgrade from an older version without the issue? I don't know of any changes which may have caused the issue. There have been several reports on this thread, but nothing conclusive yet.

@PsyTae

Looks like we are experiencing this same issue. We are using a pool and pool.query() for all our db queries, so if you would like additional info we can provide, we can try anything you suggest as well.

Thanks for the info and it sounds like you cleared up your own issue? Did the issue only show up w/ pg@8 and not pg@7?

As an aside: I have a separate pool in our app for long running queries (we have a few that take several seconds) with longer connection timeouts and so on. It can help.

Yo - sorry I was late to the party here. Left a question on here....could we include a test when both options and statement_timeout is included? Once we got that I can merge & release a new minor ver.

What happens if both options and statement_timeout is included in the config?

When you visit the pg-connection-string module on npmjs.com it links to the wrong place.

thanks!

See example https://github.com/sindresorhus/gulp-chown/blob/bb74168c957b3a94f122aafcecf7ebc87088ec46/package.json#L42-L49

This feature is supported by both npm and yarn.

Wow this is awesome, thank you.

Description

When handling errors from this package it would be better to have the errors publicly exported

Change Summary

• Updated root exports of 'pg-protocol' to include DatabaseError

Ref: #2340

👍 thanks for the PR!

Fixes #1949

If you're using connection strings, you can't use the PGSSLMODE=no-verify envvar because that's only used when config.ssl is not set. I use ?ssl=true&sslmode=no-verify&sslrootcert=/app/data/amazon-rds-ca-cert.pem which does set config.ssl, so we need to also support parsing sslmode from the connection string. I've copied the logic (roughly) from:

https://github.com/brianc/node-postgres/blob/8291b233b81312ce2fbfce12ccd98ceceb53f5b9/packages/pg/lib/connection-parameters.js#L21-L32

I didn't see a CONTRIBUTING file, so I've taken a guess at the steps to make a good PR:

• [x] Added fix
• [x] Noted the GitHub issue it closes
• [x] Added test cases / 100% code coverage
• [x] Matched surrounding code style / passed linting
• [x] Sponsor @brianc :sparkling_heart:

I can't figure out clearly, is sslmode parameter supported for connection strings? As I can see in pg-connection-string, sslmode isn't parsed, but I can see its mention in the codebase.

I use pg-promise in my project, and I have sslmode parameter in the Postgres connection string from Google Cloud. So I can't connect due to the database "dbname?sslmode=disable" does not exist, that's why I'm asking :)

Sorry for being so late on this. Went on a much needed vacation at the start of Sept & came back to a pretty disruptive life event. I just got out of the woods on that on Friday...finally. 🕺

This PR is perfect. I'm going to merge it & release a new minor version right now. ❤️

Also, really appreciate the sponsoring! ❇️

Hi! Amazing library, thanks for all of the hard work.

TLDR: The extended query protocol makes the AWS RDS Proxy entirely useless. Is there any way to turn off the extended query protocol in pg?

I have a serious issue I'm facing in a serverless environment. I'm using AWS Lambda, and directly connecting to postgres has caused horrific performance and stability issues because of the vast numbers of connections being opened and closed. So, the solution from AWS seems to be to use their RDS Proxy. It sits in front of postgres and handles a warm connection pool. I've set it all up properly. Unfortunately, all of my connections are being session pinned, essentially rendering the proxy useless, and it even seems to be performing worse than without the proxy. I've looked at my error logs and found this error repeatedly:

The client session was pinned to the database connection [dbConnection=251482716] for the remainder of the session. The proxy can't reuse this connection until the session ends. Reason: A parse message was detected.

Reason: A parse message was detected.. This seems to come because of the extended query protocol. Apparently the extended query protocol causes pinning, as documented here: https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/rds-proxy.html#rds-proxy-pinning

So my real question is, is there any way to turn off the extended query protocol in pg? If not, does anyone have any insight into how I can properly manage connections to my database in a serverless environment? This is a major issue for us at this time, and I'm not sure how to proceed

But in node-postgres and I imagine other clients, parameterized queries are generally implemented as prepared statements.

There is no difference at the protocol level, actually. Any query with parameters (whether or not you call it a prepared statement) requires a parse/bind/describe/execute series of messages. Either way it "prepares" the query during the parse phase. Sorry the RDS proxy is so picky about parameterized queries...that's quite limiting as doing parameters in queries with string concatenation and client side sanitization is not ideal. Let me know if there's anything you can think of from this libraries side to help...but sounds like an RDS issue for now.

Please feel free to submit a pull request with proper test coverage & I'll be happy to take a look.

I'd like a large result to produce an Error (which can be handled), rather than crashing the entire process with OutOfMemory.

This is similar in nature to limit I/O for HTTP requests:

const express = require('express')
const bodyParser = require('body-parser')

const app = express()
app.use(bodyParser.json({ limit: '5mb' }))

I imagine an option to Connection/Pool like maxResultSize. When the result exceeds the configured number of bytes, an Error is thrown. (I understand this limit would be subject to the choice of binary and text formats...this isn't super precise, just a way to gracefully handle unexpectedly large results.)

This would significantly improve the stability of multi-operation processes using this library.

If it's the total result set, have you considered using a cursor? That would let you operate on the row in chunks without needing to allocate memory for the entire result set. If you have commands that could return an arbitrary / huge number of rows, it's the way to go to ensure you do not blow up the node process memory usage.

Yup, 100% this is the way to handle this! The postgres protocol supports this, and node is obvs build w/ streaming in mind. This library already exists as well. We use pg and regularly stream 500-600 megabytes of results down to clients w/o issue using cursors.

As far a single row taking up more memory than your server...you should fix your data in that case. 🤷

One major difference between a db client & a web server & why body parser accepts a limit is a web server needs to protect you from bad actors sending you gigabytes in a post request and DDoSing you. Generally, you control both your web server & database client, so you are just DDoSing yourself in this case. Even if the postgres client threw an error (and forcibly disconnected) the database server has already gone through the work of shoving hundreds of megabytes into the network socket.

Hey totally open to a PR on this if you wanna try it out! Thanks for letting me know about this. Since it's only used in tests its not suuuper high priority but always nice to keep code hygiene and continuity in mind.