profile
viewpoint
Denys Otrishko lundibundi @metarhia Kyiv, Ukraine

HowProgrammingWorks/Association 1

Associations: Aggregation and Composition

lundibundi/node 1

Node.js JavaScript runtime :sparkles::turtle::rocket::sparkles:

ffluffyhedgehog/yaacrl 0

Yet Another Audio Content Recognition Library

lundibundi/activitywatch 0

Log what you do on your computer. Simple (yet powerful), extensible, no third parties.

lundibundi/aw-core 0

Core library for ActivityWatch

lundibundi/aw-watcher-web 0

Browser watcher for ActivityWatch

lundibundi/aw-webui 0

A webapp for ActivityWatch built in Vue.js

lundibundi/babel 0

🐠 Babel is a compiler for writing next generation JavaScript.

lundibundi/console 0

Early prototype of Console

lundibundi/core-validate-commit 0

Validate commit messages for Node.js core

Pull request review commentnodejs/node

stream: fix broken pipeline tests

 const assert = require('assert'); const http = require('http'); const { promisify } = require('util'); +function mustCall(fn) {+  if (!fn) {+    return common.mustCall();+  }+  return common.mustCall((...args) => {+    new Promise(() => fn(...args));+  });+}

@ronag could you provide a standalone reproduce for this one? I tried (based on 'exceptions inside common.mustCall will not cause failure')

common.mustCall(() => {
  throw new Error('hello');
})();

and

const fun = common.mustCall(() => {
  throw new Error('hello');
});
Promise.resolve(42).then(fun);

and they worked just fine.

ronag

comment created time in 22 minutes

Pull request review commentnodejs/node

src: move BaseObject subclass dtors/ctors out of node_crypto.h

 void SecureContext::Initialize(Environment* env, Local<Object> target) {   env->set_secure_context_constructor_template(t); } +SecureContext::SecureContext(Environment* env, v8::Local<v8::Object> wrap)+    : BaseObject(env, wrap) {+  MakeWeak();+  env->isolate()->AdjustAmountOfExternalAllocatedMemory(kExternalSize);+}++inline void SecureContext::Reset() {+  if (ctx_ != nullptr) {+    env()->isolate()->AdjustAmountOfExternalAllocatedMemory(-kExternalSize);+  }+  ctx_.reset();

Shouldn't this be in the != nullptr check?

jasnell

comment created time in 2 days

pull request commentnodejs/node

fs: validate the input data before opening file

/cc @ZYSzys ^

ZYSzys

comment created time in 3 days

pull request commentnodejs/node

lib: fix few comment typos in fs/watchers.js

I don't think it is needed unless it will help with other conflicts.

lundibundi

comment created time in 3 days

PR opened nodejs/quic

Few code clean ups

<!-- Thank you for your pull request. Please provide a description above and review the requirements below.

Bug fixes and new features should include tests and possibly benchmarks.

Contributors guide: https://github.com/nodejs/node/blob/master/CONTRIBUTING.md -->

Checklist

<!-- Remove items that do not apply. For completed items, change [ ] to [x]. -->

  • [x] make -j4 test (UNIX), or vcbuild test (Windows) passes
  • [x] commit message follows commit guidelines

<!-- Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I have the right to submit it under the open source license indicated in the file; or

(b) The contribution is based upon previous work that, to the best of my knowledge, is covered under an appropriate open source license and I have the right under that license to submit that work with modifications, whether created in whole or in part by me, under the same open source license (unless I am permitted to submit under a different license), as indicated in the file; or

(c) The contribution was provided directly to me by some other person who certified (a), (b) or (c) and I have not modified it.

(d) I understand and agree that this project and the contribution are public and that a record of the contribution (including all personal information I submit with it, including my sign-off) is maintained indefinitely and may be redistributed consistent with this project or the open source license(s) involved. -->

Though this will be easier than making comments in the PR. Split into different commits for easy review, most are just style or nit corrections. All can be simply squashed upon landing.

Also, shouldn't most Local methods in crypto_common actually return MaybeLocal to properly handle errors?

+51 -72

0 comment

6 changed files

pr created time in 4 days

create barnchlundibundi/quic

branch : few-code-clean-ups

created branch time in 4 days

Pull request review commentnodejs/node

vm: implement vm.measureMemory() for per-context memory measurement

 function compileFunction(code, params, options = {}) {   return result.function; } +function measureMemory(options = {}, context) {+  emitExperimentalWarning('vm.measureMemory');+  validateObject(options, 'options');+  let mode = options.mode;+  if (mode === undefined) {+    mode = constants.measureMemory.mode.SUMMARY;+  }+  validateInt32(mode, 'options.mode',+                constants.measureMemory.mode.SUMMARY,+                constants.measureMemory.mode.DETAILED);

I assume they are consecutive numbers so this works but I think explicit check would be better. Also, this doesn't produce a very clear error - it will say that number in range was expected but nothing about vm.constants where to actually get that number.

joyeecheung

comment created time in 4 days

Pull request review commentnodejs/node

vm: implement vm.measureMemory() for per-context memory measurement

 static void WatchdogHasPendingSigint(const FunctionCallbackInfo<Value>& args) {   args.GetReturnValue().Set(ret); } +static void MeasureMemory(const FunctionCallbackInfo<Value>& args) {+  CHECK(args[0]->IsInt32());+  int32_t mode = args[0].As<v8::Int32>()->Value();+  Isolate* isolate = args.GetIsolate();+  Environment* env = Environment::GetCurrent(args);+  Local<Context> context;+  if (args[1]->IsUndefined()) {+    context = isolate->GetCurrentContext();+  } else {+    ContextifyContext* sandbox =

Nit: add CHECK(args[1]->IsObject())?

joyeecheung

comment created time in 4 days

Pull request review commentnodejs/node

vm: implement vm.measureMemory() for per-context memory measurement

+'use strict';+const common = require('../common');+const assert = require('assert');+const vm = require('vm');+const {+  SUMMARY,+  DETAILED+} = vm.constants.measureMemory.mode;++common.expectWarning('ExperimentalWarning',+                     'vm.measureMemory is an experimental feature. ' ++                     'This feature could change at any time');++// Test measuring memory of the current context+{+  vm.measureMemory(undefined)+    .then((result) => {+      assert(result instanceof Object);+    });++  vm.measureMemory({})+    .then((result) => {+      assert(result instanceof Object);+    });++  vm.measureMemory({ mode: SUMMARY })+    .then((result) => {+      assert(result instanceof Object);+    });++  vm.measureMemory({ mode: DETAILED })+    .then((result) => {+      assert(result instanceof Object);+    });

Nit: can be simplified to (also for sandox)

[undefined, {}, { mode: SUMMARY }, { mode: DETAILED }].forEach((options) => {
  vm.measureMemory(options).then((result) => {
    assert(result instanceof Object);
  });
});
joyeecheung

comment created time in 4 days

Pull request review commentnodejs/node

vm: implement vm.measureMemory() for per-context memory measurement

 function compileFunction(code, params, options = {}) {   return result.function; } +function measureMemory(options = {}, context) {+  emitExperimentalWarning('vm.measureMemory');+  validateObject(options, 'options');+  let mode = options.mode;+  if (mode === undefined) {+    mode = constants.measureMemory.mode.SUMMARY;+  }

Nit:

const { mode = constants.measureMemory.mode.SUMMARY } = options;
joyeecheung

comment created time in 4 days

Pull request review commentnodejs/node

vm: implement vm.measureMemory() for per-context memory measurement

 the `process.nextTick()` and `queueMicrotask()` functions. This issue occurs because all contexts share the same microtask and nextTick queues. +## `vm.constants`+<!-- YAML+added: REPLACEME+-->++* {Object} An object containing commonly used constants for the vm module.++### `vm.constants.measureMemory`

Could this perhaps just be replaced with strings of 'summary', 'detailed' which we will then convert to appropriate constants upon call? I think that will be simpler for the end-user without introducing too much trouble on our end.

joyeecheung

comment created time in 4 days

pull request commentnodejs/node

src: fix StringSearch compiler warning

The search_string ones are caused by uninitialized arrays (they are initialized when needed and not upon object creation) in SearchStringBase which can be initialized (i.e. = {0}) with 1-2% performance impact or suppressed with something like #pragma GCC diagnostic ignored "-Wmaybe-uninitialized" (this should also work for clang). Not sure if this is needed though pragma ignore would probably be fine.

From https://github.com/nodejs/node/issues/26733#issuecomment-571995736.

There is also https://github.com/nodejs/node/pull/31532.

sam-github

comment created time in 7 days

PR opened nodejs/node

wasi: clean up options validation

<!-- Thank you for your pull request. Please provide a description above and review the requirements below.

Bug fixes and new features should include tests and possibly benchmarks.

Contributors guide: https://github.com/nodejs/node/blob/master/CONTRIBUTING.md -->

Checklist

<!-- Remove items that do not apply. For completed items, change [ ] to [x]. -->

  • [x] make -j4 test (UNIX), or vcbuild test (Windows) passes
  • [x] commit message follows commit guidelines

<!-- Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I have the right to submit it under the open source license indicated in the file; or

(b) The contribution is based upon previous work that, to the best of my knowledge, is covered under an appropriate open source license and I have the right under that license to submit that work with modifications, whether created in whole or in part by me, under the same open source license (unless I am permitted to submit under a different license), as indicated in the file; or

(c) The contribution was provided directly to me by some other person who certified (a), (b) or (c) and I have not modified it.

(d) I understand and agree that this project and the contribution are public and that a record of the contribution (including all personal information I submit with it, including my sign-off) is maintained indefinitely and may be redistributed consistent with this project or the open source license(s) involved. --> /cc @nodejs/wasi

+20 -28

0 comment

1 changed file

pr created time in 7 days

create barnchlundibundi/node

branch : clean-up-wasi-options-validation

created branch time in 7 days

delete branch lundibundi/node

delete branch : few-comment-typo-fixes-watcher

delete time in 8 days

issue openednodejs/node

workers: worker doesn't write all data to stdout stream with NODE_OPTIONS --require

<!-- Thank you for reporting an issue.

This issue tracker is for bugs and issues found within Node.js core. If you require more general support please file an issue on our help repo. https://github.com/nodejs/help

Please fill in as much of the template below as you're able.

Version: output of node -v Platform: output of uname -a (UNIX), or version and 32 or 64-bit (Windows) Subsystem: if known, please specify affected core module name -->

  • Version: master
  • Platform: Linux (at least)
  • Subsystem: workers

What steps will reproduce the bug?

// main.js
'use strict';

const { Worker } = require('worker_threads');
const assert = require('assert');

process.on('unhandledRejection', (err) => { throw err; });

async function collectStream(readable) {
  readable.setEncoding('utf8');
  let data = '';
  for await (const chunk of readable) {
    data += chunk;
  }
  return data;
}

for (let i = 0; i < 100; i++) {
  const w = new Worker('console.log("B");', {
    env: { NODE_OPTIONS: '--require ./printA.js' },
    eval: true,
    stdout: true
  });
  w.on('exit', () => {
    collectStream(w.stdout).then((data) => {
      assert.strictEqual(data, 'A\nB\n');
    });
  });
}
// printA.js
console.log('A');

<!-- Enter details about your bug, preferably a simple code snippet that can be run using node directly without installing third-party dependencies. -->

How often does it reproduce? Is there a required condition?

Quite often (2/10 runs), under heavy load the frequency is higher.

What is the expected behavior?

Worker prints all data to the stdout stream.

What do you see instead?

Worker only prints part of the written data to the stdout stream.

Example of the test case above if we replace assert with just console.log(JSON.stringify(data)):

 ➔ dev/node/node ./out/Release/node test-worker-stdout-finish.js                                                                                                                                                                             ⇡ master :: ● :: ⬡
"A\nB\n"
"A\nB\n"
"A\nB\n"
"A\nB\n"
"A\nB\n"
"A\nB\n"
"A\nB\n"
"A\n"
"A\n"
"A\nB\n"

Additional information

Weirdly enough this only happens for NODE_OPTIONS --require cases, if the worker is requireing the file by itself or just has multiple console.log statements the bug is not present. This also happens without the eval: true if we just run a file with console.log("B").

/cc @nodejs/workers

created time in 8 days

push eventlundibundi/node

Derek Lewis

commit sha a751389a14ddb3571e3ade22ac472b9aeb912bc0

tools: update Markdown linter to be cross-platform Prior to this commit, the dependencies were not matching the build procedure. This has been corrected and it has the added benefit of being able to be built on Windows as well. * continue using `rollup` rather than `ncc` * do not require `fs-event`s for non-macOS * use `npx` and `shx` for cross-platform building * ensure `lint-md-rollup` runs before `lint-md` PR-URL: https://github.com/nodejs/node/pull/31239 Reviewed-By: Rich Trott <rtrott@gmail.com>

view details

ZYSzys

commit sha 3e9302b2b34ab0309de8eda45c17efc8fd9cc2f5

fs: validate the input data before opening file PR-URL: https://github.com/nodejs/node/pull/31731 Refs: https://github.com/nodejs/node/pull/31030 Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de> Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Denys Otrishko <shishugi@gmail.com>

view details

Denys Otrishko

commit sha 84b8857098b2a846855bb2420f3a1144ef68cce3

src: allow to reuse env options handling PR-URL: https://github.com/nodejs/node/pull/31711 Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>

view details

Denys Otrishko

commit sha d63bcdd9cd81b10b6158f40d44bf6f37e5f54385

worker: properly handle env and NODE_OPTIONS in workers PR-URL: https://github.com/nodejs/node/pull/31711 Fixes: https://github.com/nodejs/node/issues/30627 Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>

view details

Denys Otrishko

commit sha 6693cfb663f2ecd1a6e18d31443e6764135f49a9

src: improve KVStore API This adds `const char*` based APIs to KVStore to avoid multiple string conversions (char -> Utf8 -> Local -> char etc.) when possible.

view details

Denys Otrishko

commit sha d38ed6ce4b67e86764aef5be257c729530a595fc

src: simplify node_worker.cc using new KVStore API

view details

push time in 8 days

issue closednodejs/node

worker_threads and NODE_OPTIONS

  • Version: 12.3.1
  • Platform: all
  • Subsystem: workers
  • Tl;dr: process._preload_modules should be forwarded to the workers

This one is a bit convoluted ... first some context:

  • Yarn 2 keeps the vendors within zip archives instead of unpacking them
  • To make it work, NODE_OPTIONS contains --require /path/to/.pnp.js
  • This .pnp.js extends the fs module to add support for in-zip reading
  • Parcel 2, a vendor (and thus stored within a zip), uses worker_threads if possible

It works fine with child_process.fork, because under the regular CLI the --require scripts are executed before the entry point is resolved (except when using --inspect if I remember correctly), so by the time runMain is called the zip archives can be accessed.

However, with worker_threads it seems that the NODE_OPTIONS --require entries are completely ignored and runMain is called directly, causing the script to be executed outside of the PnP context and thus not being able to access its dependencies or even spawn if stored within a zip archive (which is the case by default, although it can be disabled).

closed time in 8 days

arcanis

issue commentnodejs/node

worker_threads and NODE_OPTIONS

This should now work with or without the execArgv in Worker options. See https://github.com/nodejs/node/pull/31711.

Feel free to reopen if I missed anything.

arcanis

comment created time in 8 days

delete branch lundibundi/node

delete branch : worker-proper-argv-handling

delete time in 8 days

PR opened nodejs/node

src: improve KVStore API

This adds const char* based APIs to KVStore to avoid multiple string conversions (char -> Utf8 -> Local -> char etc.) when possible.

Unfortunately this only changes Get and Query methods as others have a hard dependency on Isolate in RealEnvStore as they need to call DateTimeConfigurationChangeNotification. Not sure how can we move forward with that.

<!-- Thank you for your pull request. Please provide a description above and review the requirements below.

Bug fixes and new features should include tests and possibly benchmarks.

Contributors guide: https://github.com/nodejs/node/blob/master/CONTRIBUTING.md -->

Checklist

<!-- Remove items that do not apply. For completed items, change [ ] to [x]. -->

  • [x] make -j4 test (UNIX), or vcbuild test (Windows) passes
  • [x] commit message follows commit guidelines

/cc @addaleax <!-- Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I have the right to submit it under the open source license indicated in the file; or

(b) The contribution is based upon previous work that, to the best of my knowledge, is covered under an appropriate open source license and I have the right under that license to submit that work with modifications, whether created in whole or in part by me, under the same open source license (unless I am permitted to submit under a different license), as indicated in the file; or

(c) The contribution was provided directly to me by some other person who certified (a), (b) or (c) and I have not modified it.

(d) I understand and agree that this project and the contribution are public and that a record of the contribution (including all personal information I submit with it, including my sign-off) is maintained indefinitely and may be redistributed consistent with this project or the open source license(s) involved. -->

+48 -20

0 comment

2 changed files

pr created time in 8 days

create barnchlundibundi/node

branch : kvstorage-improve-api

created branch time in 8 days

Pull request review commentnodejs/node

doc: add glossary.md

+You may also need to check https://chromium.googlesource.com/chromiumos/docs/+/master/glossary.md.++* LGTM: "Looks good to me", commonly used to approve a code review.+* PTAL: Please take a look.+* RSLGTM: "Rubber-stamp looks good to me". The reviewer approving without doing+  a full code review.+* TSC: Technical Steering Committee. Detailed info see+  [TSC](./GOVERNANCE.md#technical-steering-committee).+* WIP: "Work In Progress" - e.g. a patch that's not finished, but may be worth+  an early look.+* WPT: [web-platform-tests](https://github.com/web-platform-tests/wpt)+* godbolt: [Compiler Explorer](https://godbolt.org/) run compilers interactively

Nit:

* godbolt: [Compiler Explorer](https://godbolt.org/) allows to run compilers interactively
gengjiawen

comment created time in 8 days

Pull request review commentnodejs/node

doc: add glossary.md

+You may also need to check https://chromium.googlesource.com/chromiumos/docs/+/master/glossary.md.

Nit: shouldn't this go at the bottom of the file, as it's meant to be an addition to this file based in the text?

gengjiawen

comment created time in 8 days

Pull request review commentnodejs/node

doc: add glossary.md

+You may also need to check https://chromium.googlesource.com/chromiumos/docs/+/master/glossary.md.++* LGTM: "Looks good to me", commonly used to approve a code review.+* PTAL: Please take a look.+* RSLGTM: "Rubber-stamp looks good to me". The reviewer approving without doing+  a full code review.+* TSC: Technical Steering Committee. Detailed info see

Nit:

* TSC: Technical Steering Committee. Detailed info can be found here

Or

* TSC: Technical Steering Committee. For detailed info see
gengjiawen

comment created time in 8 days

pull request commentnodejs/node

errors: add ERR_INVALID_OPT_TYPE and support options in validators.js

@tniessen I agree that it's not the best solution, but that's what we already have (with ERR_INVALID_ARG_TYPE etc), this PR just expands that to allow handling of option errors in the same way.

The current idea is that if we know a more specific error (e.g. out of range) then we use it, otherwise we just throw a generic invalid value or invalid type. TBH I don't really see a point in specific out of range error in our current approach and IMO we could just always use invalid value and invalid type with specific error messages.

lundibundi

comment created time in 9 days

Pull request review commentnodejs/node

test: remove common.PORT from some pummel tests

 const server = tls.Server(options, common.mustCall(function(socket) {   socket.pipe(mediator); })); -server.listen(common.PORT, common.mustCall(function() {+server.listen(0, common.mustCall(function() {

Nit: while we are at it =)

server.listen(0, common.mustCall(() => {
Trott

comment created time in 9 days

Pull request review commentnodejs/node

src: use symbol to store `AsyncWrap` resource

 function fatalError(e) {   process.exit(1); } +function lookupPublicResource(resource) {+  if (typeof resource !== 'object') return resource;

Nit: I don't think it is possible but pointing out that this doesn't check against resource === null.

addaleax

comment created time in 9 days

issue closednodejs/node

HPE_INVALID_HEADER_TOKEN on https requests

Version: v10.16.3 Platform: alpine:3.10 in docker Subsystem: http_parser: '2.9.2'

HTTP request crashes with HPE_INVALID_HEADER_TOKEN.

Script to reproduce:

https.get('https://www.dezeen.com/2015/08/04/walden-raft-provides-seclusion-french-lake-elise-morin-florent-albinet-henry-david-thoreau-cabin/', (resp) => { }).on("error", (err) => {
  console.log(err);
});

Output:

{ Error: Parse Error
    at TLSSocket.socketOnData (_http_client.js:442:20)
    at TLSSocket.emit (events.js:198:13)
    at TLSSocket.EventEmitter.emit (domain.js:466:23)
    at addChunk (_stream_readable.js:288:12)
    at readableAddChunk (_stream_readable.js:269:11)
    at TLSSocket.Readable.push (_stream_readable.js:224:10)
    at TLSWrap.onStreamRead (internal/stream_base_commons.js:94:17) bytesParsed: 1087, code: 'HPE_INVALID_HEADER_TOKEN' }

I think it's related to http_parser: '2.9.2'. It does not happen with a slightly older build which uses http_parser: '2.8.0'.

Likely related to https://github.com/nodejs/node/issues/27711#issuecomment-554656422.

The mentioned workaround does not work here, as the --http-parser option is not available on node v10.

closed time in 10 days

ejoebstl

issue commentnodejs/node

HPE_INVALID_HEADER_TOKEN on https requests

Closing, see https://github.com/nodejs/node/issues/27711#issuecomment-584621376.

ejoebstl

comment created time in 10 days

issue closednodejs/node

HPE_INVALID_HEADER_TOKEN on http requests

Upgrading to 12.2.0 broke several http calls with a parse error HPE_INVALID_HEADER_TOKEN, all requests were working fine with version 11.10.0 I had before the upgrade.

I tried http-parser-js library to patch http but I still get the same issue process.binding('http_parser').HTTPParser = require('http-parser-js').HTTPParser;

closed time in 10 days

faberyx

issue commentnodejs/node

HPE_INVALID_HEADER_TOKEN on http requests

@omieliekh you are using Node.js v12.9.1 which has neither CLI nor http options, your only option is to use the 4th option (described below) OR upgrade to newer Node.js. @jamiechong I don't see any solution but to use the 1st option below. One more option would be to modify the library you are using, to allow passing http options but that might be infeasible.


Looks like this is 'fixed' and the current solution to this won't change any more. There are 3+1 options available (please note that all of them impose security risks):

  1. It is possible to use --insecure-http-parser in the CLI or NODE_OPTIONS to configure this per-process.
  2. Pass insecureHTTPParser to http.createServer() to configure this on a per-server basis.
  3. Pass insecureHTTPParser to http.request() to configure this on a per-stream basis.
  4. Use --http-parser=legacy CLI options (only available on v12).

For v13: 1st option is available since v13.4.0 while 2nd and 3rd since v13.8.0.

For v12: the first 3 options are available since v12.5.0 and the 4th since the start of v12.

The 4th option has been available since v11.4.0 and is therefore available in v12 but was removed since v13.0.0.

Feel free to reopen if I missed something (collaborators, feel free to just edit this comment).

faberyx

comment created time in 10 days

pull request commentnodejs/node

lib: improve validation utils and refactor vm argument validation

Landed in c405e9b and 2abf0af.

lundibundi

comment created time in 11 days

delete branch lundibundi/node

delete branch : improve-validation-utils

delete time in 11 days

PR closed nodejs/node

lib: improve validation utils and refactor vm argument validation author ready vm

The first commit adds basic validation utilities like validateArray, validateBoolean, validateObject.

The second commit refactors vm.js with the new validators as an example. There are multiple places in the codebase where we manually validate objects and arrays which can be simplified with validators.js.

<!-- Thank you for your pull request. Please provide a description above and review the requirements below.

Bug fixes and new features should include tests and possibly benchmarks.

Contributors guide: https://github.com/nodejs/node/blob/master/CONTRIBUTING.md -->

Checklist

<!-- Remove items that do not apply. For completed items, change [ ] to [x]. -->

  • [x] make -j4 test (UNIX), or vcbuild test (Windows) passes
  • [x] commit message follows commit guidelines

<!-- Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I have the right to submit it under the open source license indicated in the file; or

(b) The contribution is based upon previous work that, to the best of my knowledge, is covered under an appropriate open source license and I have the right under that license to submit that work with modifications, whether created in whole or in part by me, under the same open source license (unless I am permitted to submit under a different license), as indicated in the file; or

(c) The contribution was provided directly to me by some other person who certified (a), (b) or (c) and I have not modified it.

(d) I understand and agree that this project and the contribution are public and that a record of the contribution (including all personal information I submit with it, including my sign-off) is maintained indefinitely and may be redistributed consistent with this project or the open source license(s) involved. --> //cc @nodejs/vm

+144 -101

5 comments

3 changed files

lundibundi

pr closed time in 11 days

push eventnodejs/node

Denys Otrishko

commit sha c405e9b23c6c6dc78f5d2b0747a75ceb61eaf4d9

lib: improve value validation utils Add common validators: `validateArray`, `validateBoolean`, `validateObject` and appropriate tests. PR-URL: https://github.com/nodejs/node/pull/31480 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Gus Caplan <me@gus.host>

view details

Denys Otrishko

commit sha 2abf0afb2b9e2f9e9824e2cbb08bdec27794d039

vm: refactor value validation with internal/validators.js PR-URL: https://github.com/nodejs/node/pull/31480 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Gus Caplan <me@gus.host>

view details

push time in 11 days

issue openednodejs/node

Investigate test-inspector-port-zero

<!-- Thank you for reporting an issue.

This issue tracker is for bugs and issues found within Node.js core. If you require more general support please file an issue on our help repo. https://github.com/nodejs/help

Please fill in as much of the template below as you're able.

Version: output of node -v Platform: output of uname -a (UNIX), or version and 32 or 64-bit (Windows) Subsystem: if known, please specify affected core module name -->

Probably flaky but really rare, seen only a few times. Opening an issue to keep track of it. Perhaps someone knows what could be the issue with it.

  • Version: master
  • Platform: arm
  • Subsystem: inspector/console
Error Message

fail (6)

Stacktrace

internal/console/global.js:44
globalConsole[kBindStreamsLazy](process);
                               ^

TypeError: globalConsole[kBindStreamsLazy] is not a function
    at internal/console/global.js:44:32
    at NativeModule.compileForInternalLoader (internal/bootstrap/loaders.js:276:7)
    at nativeModuleRequire (internal/bootstrap/loaders.js:305:14)
    at createGlobalConsole (internal/bootstrap/node.js:317:5)
    at internal/bootstrap/node.js:120:38

https://ci.nodejs.org/job/node-test-binary-arm-12+/4293/RUN_SUBSET=1,label=pi2-docker/testReport/junit/(root)/test/sequential_test_inspector_port_zero/

/cc @nodejs/testing @addaleax

created time in 11 days

pull request commentnodejs/node

worker_threads: proper env and NODE_OPTIONS handling

Not sure if flaky or a regression: test.sequential/test-inspector-port-zero

Error Message
fail (6)
Stacktrace
internal/console/global.js:44
globalConsole[kBindStreamsLazy](process);
                               ^

TypeError: globalConsole[kBindStreamsLazy] is not a function
    at internal/console/global.js:44:32
    at NativeModule.compileForInternalLoader (internal/bootstrap/loaders.js:276:7)
    at nativeModuleRequire (internal/bootstrap/loaders.js:305:14)
    at createGlobalConsole (internal/bootstrap/node.js:317:5)
    at internal/bootstrap/node.js:120:38

Trying a resume for now.

lundibundi

comment created time in 11 days

push eventlundibundi/node

Denys Otrishko

commit sha 0c87f8cb1359139316b220a46f461a7923f037f8

src: allow to reuse env options handling

view details

Denys Otrishko

commit sha 503003eec63bf3ee1b6f5c7bb2f3e4e8b2c07073

worker_threads: properly handle env and NODE_OPTIONS in workers

view details

push time in 11 days

Pull request review commentnodejs/node

worker_threads: proper env and NODE_OPTIONS handling

 E('ERR_WASI_ALREADY_STARTED', 'WASI instance has already started', Error); E('ERR_WORKER_INVALID_EXEC_ARGV', (errors) =>   `Initiated Worker with invalid execArgv flags: ${errors.join(', ')}`,   Error);+E('ERR_WORKER_INVALID_NODE_OPTIONS', (errors) =>+  `Initiated Worker with invalid NODE_OPTIONS: ${errors.join(', ')}`,+  Error);

(Fwiw, ERR_WORKER_INVALID_OPTIONS sounds like it refers to the options object passed to the Worker constructor…)

That was the idea - part of the options passed is incorrect. tbh we can probably use ERR_INVALID_OPT_VALUE from https://github.com/nodejs/node/pull/31251 :smile:.

Then I'll use the ERR_WORKER_INVALID_EXEC_ARGV in this PR and then open another for the error change.

lundibundi

comment created time in 11 days

Pull request review commentnodejs/node

worker_threads: add support for .cjs extension

 class Worker extends EventEmitter {       filename = path.resolve(filename);        const ext = path.extname(filename);-      if (ext !== '.js' && ext !== '.mjs') {+      if (!/^\.[cm]?js$/.test(ext)) {

Nit: why not just add another && ext !== '.cjs', that would be both simpler to understand and faster to run. I don't see the benefits of using regexp here.

aduh95

comment created time in 11 days

Pull request review commentnodejs/node

worker_threads: proper env and NODE_OPTIONS handling

 void Worker::New(const FunctionCallbackInfo<Value>& args) {    std::string url;   std::shared_ptr<PerIsolateOptions> per_isolate_opts = nullptr;+  std::shared_ptr<KVStore> env_vars = nullptr;    std::vector<std::string> exec_argv_out;-  bool has_explicit_exec_argv = false; -  CHECK_EQ(args.Length(), 3);+  CHECK_EQ(args.Length(), 4);   // Argument might be a string or URL   if (!args[0]->IsNullOrUndefined()) {     Utf8Value value(-        args.GetIsolate(),-        args[0]->ToString(env->context()).FromMaybe(Local<String>()));+        isolate, args[0]->ToString(env->context()).FromMaybe(Local<String>()));     url.append(value.out(), value.length());   } -  if (args[1]->IsArray()) {-    Local<Array> array = args[1].As<Array>();+  if (args[1]->IsNull()) {+    // Means worker.env == process.env.+    env_vars = env->env_vars()->Clone(isolate);+  } else if (args[1]->IsObject()) {+    // User provided env.+    env_vars = KVStore::CreateMapKVStore();+    env_vars->AssignFromObject(isolate->GetCurrentContext(),+                               args[1].As<Object>());+  } else {+    env_vars = env->env_vars();+  }++  if (args[1]->IsObject() || args[2]->IsArray()) {+    per_isolate_opts.reset(new PerIsolateOptions());++    HandleEnvOptions(+        per_isolate_opts->per_env, [isolate, &env_vars](const char* name) {+          MaybeLocal<String> value =+              env_vars->Get(isolate, OneByteString(isolate, name));+          return value.IsEmpty() ? std::string{}

I'm not sure I understand how to distinguish 'no-value' and 'String::NewFromUtf8' failing with exception in KVStore::Get() call as both of them return an empty MaybeLocal?

lundibundi

comment created time in 11 days

Pull request review commentnodejs/node

worker_threads: proper env and NODE_OPTIONS handling

 E('ERR_WASI_ALREADY_STARTED', 'WASI instance has already started', Error); E('ERR_WORKER_INVALID_EXEC_ARGV', (errors) =>   `Initiated Worker with invalid execArgv flags: ${errors.join(', ')}`,   Error);+E('ERR_WORKER_INVALID_NODE_OPTIONS', (errors) =>+  `Initiated Worker with invalid NODE_OPTIONS: ${errors.join(', ')}`,+  Error);

Gave it another thought, imo ERR_WORKER_INVALID_EXEC_ARGV is way too specific and may mislead people into thinking that the error is elsewhere. I think it would be better to add ERR_WORKER_INVALID_OPTIONS and group both ERR_WORKER_INVALID_EXEC_ARGV and ERR_WORKER_INVALID_NODE_OPTIONS under it, though that may be breaking. Or just add and use ERR_WORKER_INVALID_OPTIONS here and then open a semver-major to change that in for ERR_WORKER_INVALID_EXEC_ARGV. wdyt?

lundibundi

comment created time in 11 days

Pull request review commentnodejs/node

worker_threads: proper env and NODE_OPTIONS handling

 void Worker::New(const FunctionCallbackInfo<Value>& args) {    std::string url;   std::shared_ptr<PerIsolateOptions> per_isolate_opts = nullptr;+  std::shared_ptr<KVStore> env_vars = nullptr;    std::vector<std::string> exec_argv_out;-  bool has_explicit_exec_argv = false; -  CHECK_EQ(args.Length(), 3);+  CHECK_EQ(args.Length(), 4);   // Argument might be a string or URL   if (!args[0]->IsNullOrUndefined()) {     Utf8Value value(-        args.GetIsolate(),-        args[0]->ToString(env->context()).FromMaybe(Local<String>()));+        isolate, args[0]->ToString(env->context()).FromMaybe(Local<String>()));     url.append(value.out(), value.length());   } -  if (args[1]->IsArray()) {-    Local<Array> array = args[1].As<Array>();+  if (args[1]->IsNull()) {+    // Means worker.env == process.env.+    env_vars = env->env_vars()->Clone(isolate);+  } else if (args[1]->IsObject()) {+    // User provided env.+    env_vars = KVStore::CreateMapKVStore();+    env_vars->AssignFromObject(isolate->GetCurrentContext(),+                               args[1].As<Object>());+  } else {+    env_vars = env->env_vars();+  }++  if (args[1]->IsObject() || args[2]->IsArray()) {+    per_isolate_opts.reset(new PerIsolateOptions());++    HandleEnvOptions(+        per_isolate_opts->per_env, [isolate, &env_vars](const char* name) {+          MaybeLocal<String> value =+              env_vars->Get(isolate, OneByteString(isolate, name));+          return value.IsEmpty() ? std::string{}+                                 : std::string(*String::Utf8Value(+                                       isolate, value.ToLocalChecked()));

Well, then I think it is fine for now. Later would try to make a PR for std::string based methods for KVStore.

lundibundi

comment created time in 11 days

push eventlundibundi/node

Denys Otrishko

commit sha 8f497798204f9b726eacda844861b69b719008d4

fixup! worker_threads: properly handle env and NODE_OPTIONS in workers

view details

push time in 11 days

Pull request review commentnodejs/node

worker_threads: proper env and NODE_OPTIONS handling

 void Worker::New(const FunctionCallbackInfo<Value>& args) {    std::string url;   std::shared_ptr<PerIsolateOptions> per_isolate_opts = nullptr;+  std::shared_ptr<KVStore> env_vars = nullptr;    std::vector<std::string> exec_argv_out;-  bool has_explicit_exec_argv = false; -  CHECK_EQ(args.Length(), 3);+  CHECK_EQ(args.Length(), 4);   // Argument might be a string or URL   if (!args[0]->IsNullOrUndefined()) {     Utf8Value value(-        args.GetIsolate(),-        args[0]->ToString(env->context()).FromMaybe(Local<String>()));+        isolate, args[0]->ToString(env->context()).FromMaybe(Local<String>()));     url.append(value.out(), value.length());   } -  if (args[1]->IsArray()) {-    Local<Array> array = args[1].As<Array>();+  if (args[1]->IsNull()) {+    // Means worker.env == process.env.+    env_vars = env->env_vars()->Clone(isolate);+  } else if (args[1]->IsObject()) {+    // User provided env.+    env_vars = KVStore::CreateMapKVStore();+    env_vars->AssignFromObject(isolate->GetCurrentContext(),+                               args[1].As<Object>());+  } else {+    env_vars = env->env_vars();+  }++  if (args[1]->IsObject() || args[2]->IsArray()) {+    per_isolate_opts.reset(new PerIsolateOptions());++    HandleEnvOptions(+        per_isolate_opts->per_env, [isolate, &env_vars](const char* name) {+          MaybeLocal<String> value =+              env_vars->Get(isolate, OneByteString(isolate, name));+          return value.IsEmpty() ? std::string{}+                                 : std::string(*String::Utf8Value(+                                       isolate, value.ToLocalChecked()));

Basically converts strings quite a few times to get the value, anyone knows a better way to do this?

lundibundi

comment created time in 11 days

Pull request review commentnodejs/node

worker_threads: proper env and NODE_OPTIONS handling

 void Initialize(Local<Object> target, }  }  // namespace options_parser++void HandleEnvOptions(std::shared_ptr<EnvironmentOptions> env_options) {+  HandleEnvOptions(env_options, [](const char* name) {+    std::string text;+    return credentials::SafeGetenv(name, &text) ? text : "";+  });+}++void HandleEnvOptions(std::shared_ptr<EnvironmentOptions> env_options,+                      std::function<std::string(const char*)> opt_getter) {

std::function<std::string(const char*)> opt_getter is probably not the simplest solution here.

lundibundi

comment created time in 11 days

PR opened nodejs/node

worker_threads: proper env and NODE_OPTIONS handling

Draft for now, as I still want to review the documentation and take another look at the code.

<!-- Thank you for your pull request. Please provide a description above and review the requirements below.

Bug fixes and new features should include tests and possibly benchmarks.

Contributors guide: https://github.com/nodejs/node/blob/master/CONTRIBUTING.md -->

Checklist

<!-- Remove items that do not apply. For completed items, change [ ] to [x]. -->

  • [x] make -j4 test (UNIX), or vcbuild test (Windows) passes
  • [x] tests and/or benchmarks are included
  • [ ] documentation is changed or added
  • [x] commit message follows commit guidelines

<!-- Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I have the right to submit it under the open source license indicated in the file; or

(b) The contribution is based upon previous work that, to the best of my knowledge, is covered under an appropriate open source license and I have the right under that license to submit that work with modifications, whether created in whole or in part by me, under the same open source license (unless I am permitted to submit under a different license), as indicated in the file; or

(c) The contribution was provided directly to me by some other person who certified (a), (b) or (c) and I have not modified it.

(d) I understand and agree that this project and the contribution are public and that a record of the contribution (including all personal information I submit with it, including my sign-off) is maintained indefinitely and may be redistributed consistent with this project or the open source license(s) involved. --> /cc @nodejs/workers @addaleax

+272 -140

0 comment

11 changed files

pr created time in 11 days

create barnchlundibundi/node

branch : worker-proper-argv-handling

created branch time in 11 days

Pull request review commentnodejs/node

test: fix flaky parallel/test-repl-history-navigation test

 const tests = [     env: { NODE_REPL_HISTORY: defaultHistoryPath },     completer(line, callback) {       if (line.endsWith(WAIT)) {-        setTimeout(-          callback,-          common.platformTimeout(40),-          null,-          [[`${WAIT}WOW`], line]-        );+        if (completions++ === 0) {+          callback(null, [[`${WAIT}WOW`], line]);+        } else {+          setTimeout(

Nit: I think this can fit in one line now.

BridgeAR

comment created time in 12 days

pull request commentnodejs/node

errors: add ERR_INVALID_OPT_TYPE and support options in validators.js

ping @BridgeAR @Trott @jasnell.

lundibundi

comment created time in 12 days

pull request commentnodejs/node

lib: improve validation utils and refactor vm argument validation

Started another CI to make sure everything is fine, the previous one is quite old.

lundibundi

comment created time in 12 days

Pull request review commentnodejs/node

fs: fix valid id range on chown, lchown, fchown

 const validateUint32 = hideStackFrames((value, name, positive) => {   } }); +const validateUint32OrNegative1 = hideStackFrames((value, name) => {+  if (!isUint32(value) && value !== -1) {+    if (typeof value !== 'number') {+      throw new ERR_INVALID_ARG_TYPE(name, 'number', value);+    }+    if (!NumberIsInteger(value)) {+      throw new ERR_OUT_OF_RANGE(name, 'an integer', value);+    }+    // 2 ** 32 === 4294967296+    throw new ERR_OUT_OF_RANGE(name, '>= -1 && < 4294967296', value);+  }+});

I agree with @addaleax, the fact that validateInt32 have these parameters doesn't mean that they make sense for validateUint32, it is not possible for Uint to be negative, by the definition of that type (hence the failing isUint32(value) check), therefore it would be strange to have it support such a use case. I think it would be better to just check this manually (the same way we do for optional values by just checking if (value !== undefined) before validation) with

if (uid !== -1) validateUint32(...)

If we want to generalize this, I think validateInteger would be a better fit, by adding min/max to it.

Himself65

comment created time in 12 days

push eventlundibundi/node

Denys Otrishko

commit sha d7cdfca2bcb700f94b24582dfe63481624f00a88

fixup! Update lib/internal/fs/watchers.js Co-Authored-By: Rich Trott <rtrott@gmail.com>

view details

push time in 12 days

push eventlundibundi/node

Denys Otrishko

commit sha 926b24793a1bae5044db2c065094c1479bd68ed0

fixup! Update lib/internal/fs/watchers.js Co-Authored-By: Rich Trott <rtrott@gmail.com>

view details

push time in 12 days

Pull request review commentnodejs/node

lib: fix few comment typos in fs/watchers.js

 function FSWatcher() {       if (this._handle !== null) {         // We don't use this.close() here to avoid firing the close event.         this._handle.close();-        this._handle = null;  // Make the handle garbage collectable+        this._handle = null;  // Make the handle is garbage collectable.

:smile: was probably thinking of the other sentence while changing this. thanks.

lundibundi

comment created time in 12 days

pull request commentnodejs/node

test: change test limit for atime from 2ms to 5ms

@Trott this actually fails for me locally on Arch Linux every now and then (the 2ms), so I'd be in favor of merging this unless we find a better solution.

duncanhealy

comment created time in 12 days

Pull request review commentnodejs/node

benchmark: add "byGroup" config option

 exports.createBenchmark = function(fn, configs, options) { function Benchmark(fn, configs, options) {   // Use the file name as the name of the benchmark   this.name = require.main.filename.slice(__dirname.length + 1);-  // Parse job-specific configuration from the command line arguments-  const parsed_args = this._parseArgs(process.argv.slice(2), configs);-  this.options = parsed_args.cli;-  this.extra_options = parsed_args.extra;-  // The configuration list as a queue of jobs-  this.queue = this._queue(this.options);+  this.queue = [];+  const byGroup = (options && options.byGroup) || false;++  const enqueueConfigsInFirstGroup = (configs) => {+    const firstGroupKey = Object.keys(configs)[0];+    const configsInFirstGroup = configs[firstGroupKey];+    const parsedArgs =+      this._parseArgs(process.argv.slice(2), configsInFirstGroup);+    this.options = parsedArgs.cli;+    this.extra_options = parsedArgs.extra;+    this.queue = this._queue(this.options);+  };++  const enqueueConfigsInGroups = (configs) => {+    for (const groupKey of Object.keys(configs)) {+      const config = configs[groupKey];+      const parsedArgs = this._parseArgs(process.argv.slice(2), config);+      this.options = parsedArgs.cli;+      this.extra_options = parsedArgs.extra;+      // The configuration list as a queue of jobs by merging the existing+      // items with the new items to return a new array+      this.queue = [ ...this.queue, ...this._queue(this.options) ];+    }+  };++  const enqueueConfigs = (configs) => {+    // Parse job-specific configuration from the command line arguments+    const parsedArgs = this._parseArgs(process.argv.slice(2), configs);+    this.options = parsedArgs.cli;+    this.extra_options = parsedArgs.extra;+    // The configuration list as a queue of jobs+    this.queue = this._queue(this.options);+  };++  if (byGroup) {+    if (process.env.NODEJS_BENCHMARK_BYPASS_GROUP) {+      enqueueConfigsInFirstGroup(configs);+    } else {+      enqueueConfigsInGroups(configs);+    }+  } else {+    enqueueConfigs(configs);+  }

I think this can be simplified to

const parseConfig = (config) => {
      const parsedArgs = this._parseArgs(process.argv.slice(2), config);
      this.options = parsedArgs.cli;
      this.extra_options = parsedArgs.extra;
      this.queue = [ ...this.queue, ...this._queue(this.options) ];
}

if (byGroup) {
  let usedConfigs = Object.values(configs);
  if (process.env.NODEJS_BENCHMARK_BYPASS_GROUP)
    usedConfigs = usedConfigs[0];
  usedConfigs.forEach(parseConfig);
} else {
  parseConfig(configs);
}
BeniCheni

comment created time in 12 days

issue commentnodejs/node

Investigate flaky async-hooks/test-statwatcher

This looks like async-hooks bug now, the statwatcher emits exactly 2 events (based on the log from onchangex which is there till the end of the test) but for some reason, we get another before invocation for the 1st file. I actually tried to reproduce this on Windows 10 but to no avail, it didn't fail at even extreme values of -j --repeat.

Trott

comment created time in 12 days

PR opened nodejs/node

lib: fix few comment typos in fs/watchers.js

<!-- Thank you for your pull request. Please provide a description above and review the requirements below.

Bug fixes and new features should include tests and possibly benchmarks.

Contributors guide: https://github.com/nodejs/node/blob/master/CONTRIBUTING.md -->

Checklist

<!-- Remove items that do not apply. For completed items, change [ ] to [x]. -->

  • [x] make -j4 test (UNIX), or vcbuild test (Windows) passes
  • [x] commit message follows commit guidelines

<!-- Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I have the right to submit it under the open source license indicated in the file; or

(b) The contribution is based upon previous work that, to the best of my knowledge, is covered under an appropriate open source license and I have the right under that license to submit that work with modifications, whether created in whole or in part by me, under the same open source license (unless I am permitted to submit under a different license), as indicated in the file; or

(c) The contribution was provided directly to me by some other person who certified (a), (b) or (c) and I have not modified it.

(d) I understand and agree that this project and the contribution are public and that a record of the contribution (including all personal information I submit with it, including my sign-off) is maintained indefinitely and may be redistributed consistent with this project or the open source license(s) involved. -->

+7 -7

0 comment

1 changed file

pr created time in 12 days

create barnchlundibundi/node

branch : few-comment-typo-fixes-watcher

created branch time in 12 days

Pull request review commentnodejs/node

worker: improve MessagePort performance

 MessagePort* MessagePort::New(     return nullptr;   MessagePort* port = new MessagePort(env, context, instance);   CHECK_NOT_NULL(port);+  if (port->async_.data == nullptr) {+    port->Close();

Yeah, half initialized objects is a trade-off in this case.

I’ve updated the code to move the close call into the constructor, and made it private to make it clear that MessagePort::New() is the way to construct MessagePorts.

LGTM.

addaleax

comment created time in 17 days

Pull request review commentnodejs/node

worker: improve MessagePort performance

 MessagePort* MessagePort::New(     return nullptr;   MessagePort* port = new MessagePort(env, context, instance);   CHECK_NOT_NULL(port);+  if (port->async_.data == nullptr) {+    port->Close();

Shouldn't this close happen in the MessagePort as a part of failure-handling when we failed to construct it?

IMO, not blocking but I think it would be better to have some sort of init function (or maybe override operator new if that's possible here?) that could return the status of the initialization than resort to checking properties like this.

addaleax

comment created time in 19 days

PR closed nodejs/node

test: improve flaky http2-stream-destroy-event-order http2 test

This doesn't properly fix the issue, it only significantly reduces (removes?) the flakiness of the test.

The setTimeout must not be needed here. Specifically if the settings frame on the client is 'delayed' and only comes after we have already called req.close(2) then the 'close' event will be emitted before the stream has got a chance to write the rst to the underlying socket (even though FlushRstStream and SendPendingData have been called) and since we are destroying a session here it will never be written resulting in a 'clean' stream close on the server side and therefore a timeout because there will be no error.


Looks like we are not correctly waiting for the rst frame to be sent and I assume the time it takes to read the settings frame delays the write just enough to not get written at all. I will try to investigate this further but at least this should improve our CI so that it won't fail with this test almost every time. This patch reduces the flakiness from 1 failed in 10 runs to 0/2000 on my windows machine.

The debug native logs can be found: http2-destroy-event-order-success.txt http2-destroy-event-order-fail.txt.

/cc @nodejs/testing @jasnell @addaleax <!-- Thank you for your pull request. Please provide a description above and review the requirements below.

Bug fixes and new features should include tests and possibly benchmarks.

Contributors guide: https://github.com/nodejs/node/blob/master/CONTRIBUTING.md -->

Checklist

<!-- Remove items that do not apply. For completed items, change [ ] to [x]. -->

  • [x] make -j4 test (UNIX), or vcbuild test (Windows) passes
  • [x] commit message follows commit guidelines

<!-- Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I have the right to submit it under the open source license indicated in the file; or

(b) The contribution is based upon previous work that, to the best of my knowledge, is covered under an appropriate open source license and I have the right under that license to submit that work with modifications, whether created in whole or in part by me, under the same open source license (unless I am permitted to submit under a different license), as indicated in the file; or

(c) The contribution was provided directly to me by some other person who certified (a), (b) or (c) and I have not modified it.

(d) I understand and agree that this project and the contribution are public and that a record of the contribution (including all personal information I submit with it, including my sign-off) is maintained indefinitely and may be redistributed consistent with this project or the open source license(s) involved. -->

+10 -1

5 comments

1 changed file

lundibundi

pr closed time in 19 days

Pull request review commentnodejs/node

doc: guide - using valgrind to debug memory leaks

+# Investigating Memory Leaks with valgrind++A Node.js process may run out of memory due to excessive consumption of+native memory. Native Memory is memory which is not managed by the+V8 Garbage collector and is allocated either by the Node.js runtime, its+dependencies or native [addons](https://nodejs.org/docs/latest/api/n-api.html).++This guide provides information on how to use valgrind to investigate these+issues.++## valgrind++[Valgrind](https://valgrind.org/docs/manual/quick-start.html) is a+tool available on Linux distributions which can be used to investigate+memory usage including identifying memory leaks (memory which is+allocated and not freed) and other memory related problems+like double freeing memory.++To use valgrind:++* Be patient, running under valgrind slows execution significantly+  due to the checks being performed.+* Reduce your test case to the smallest reproduce. Due to the slowdown it is+  important to run the minimum test case in order to be able to do it in+  a reasonable time.++## Installation++It is an optional package in most cases and must be installed explicitly.+For example on Debian/Ubuntu:++```console+apt-get install valgrind+```++## Invocation+The simplest invocation of valgrind is:++```console+valgrind node test.js+```++with the output being:++```console+user1@minikube1:~/valgrind/node-addon-examples/1_hello_world/napi$ valgrind node test.js+==28993== Memcheck, a memory error detector+==28993== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.+==28993== Using valgrind-3.13.0 and LibVEX; rerun with -h for copyright info+==28993== Command: node test.js+==28993==+==28993== Use of uninitialised value of size 8+==28993==    at 0x12F2279: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0x12F3E9C: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0x12F3C77: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0xC7C9CF: v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0xC7CE87: v8::internal::Execution::Call(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, int, v8::internal::Handle<v8::internal::Object>*) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0xB4CF3A: v8::Function::Call(v8::Local<v8::Context>, v8::Local<v8::Value>, int, v8::Local<v8::Value>*) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==+--28993-- WARNING: unhandled amd64-linux syscall: 332+--28993-- You may be able to write your own handler.+--28993-- Read the file README_MISSING_SYSCALL_OR_IOCTL.+--28993-- Nevertheless we consider this a bug.  Please report+--28993-- it at http://valgrind.org/support/bug_reports.html.+==28993==+==28993== HEAP SUMMARY:+==28993==     in use at exit: 6,140 bytes in 23 blocks+==28993==   total heap usage: 12,888 allocs, 12,865 frees, 13,033,244 bytes allocated+==28993==+==28993== LEAK SUMMARY:+==28993==    definitely lost: 0 bytes in 0 blocks+==28993==    indirectly lost: 0 bytes in 0 blocks+==28993==      possibly lost: 304 bytes in 1 blocks+==28993==    still reachable: 5,836 bytes in 22 blocks+==28993==         suppressed: 0 bytes in 0 blocks+==28993== Rerun with --leak-check=full to see details of leaked memory+==28993==+==28993== For counts of detected and suppressed errors, rerun with: -v+==28993== Use --track-origins=yes to see where uninitialised values come+```++This reports that Node.js is not _completely_ clean as there is some memory+that was allocated but not freed when the process shut down. It is often+impractical/not worth being completely clean in this respect. Modern+operating systems will clean up the memory of the process after the+shutdown while attemping to fre all memory to get a clean
shutdown while attempting to free all memory to get a clean
mhdawson

comment created time in 20 days

Pull request review commentnodejs/node

doc: guide - using valgrind to debug memory leaks

+# Investigating Memory Leaks with valgrind++A Node.js process may run out of memory due to excessive consumption of+native memory. Native Memory is memory which is not managed by the+V8 Garbage collector and is allocated either by the Node.js runtime, its+dependencies or native [addons](https://nodejs.org/docs/latest/api/n-api.html).++This guide provides information on how to use valgrind to investigate these+issues.++## valgrind++[Valgrind](https://valgrind.org/docs/manual/quick-start.html) is a+tool available on Linux distributions which can be used to investigate+memory usage including identifying memory leaks (memory which is+allocated and not freed) and other memory related problems+like double freeing memory.++To use valgrind:++* Be patient, running under valgrind slows execution significantly+  due to the checks being performed.+* Reduce your test case to the smallest reproduce. Due to the slowdown it is+  important to run the minimum test case in order to be able to do it in+  a reasonable time.++## Installation++It is an optional package in most cases and must be installed explicitly.+For example on Debian/Ubuntu:++```console+apt-get install valgrind+```++## Invocation+The simplest invocation of valgrind is:++```console+valgrind node test.js+```++with the output being:++```console+user1@minikube1:~/valgrind/node-addon-examples/1_hello_world/napi$ valgrind node test.js+==28993== Memcheck, a memory error detector+==28993== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.+==28993== Using valgrind-3.13.0 and LibVEX; rerun with -h for copyright info+==28993== Command: node test.js+==28993==+==28993== Use of uninitialised value of size 8+==28993==    at 0x12F2279: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0x12F3E9C: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0x12F3C77: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0xC7C9CF: v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0xC7CE87: v8::internal::Execution::Call(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, int, v8::internal::Handle<v8::internal::Object>*) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0xB4CF3A: v8::Function::Call(v8::Local<v8::Context>, v8::Local<v8::Value>, int, v8::Local<v8::Value>*) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==+--28993-- WARNING: unhandled amd64-linux syscall: 332+--28993-- You may be able to write your own handler.+--28993-- Read the file README_MISSING_SYSCALL_OR_IOCTL.+--28993-- Nevertheless we consider this a bug.  Please report+--28993-- it at http://valgrind.org/support/bug_reports.html.+==28993==+==28993== HEAP SUMMARY:+==28993==     in use at exit: 6,140 bytes in 23 blocks+==28993==   total heap usage: 12,888 allocs, 12,865 frees, 13,033,244 bytes allocated+==28993==+==28993== LEAK SUMMARY:+==28993==    definitely lost: 0 bytes in 0 blocks+==28993==    indirectly lost: 0 bytes in 0 blocks+==28993==      possibly lost: 304 bytes in 1 blocks+==28993==    still reachable: 5,836 bytes in 22 blocks+==28993==         suppressed: 0 bytes in 0 blocks+==28993== Rerun with --leak-check=full to see details of leaked memory+==28993==+==28993== For counts of detected and suppressed errors, rerun with: -v+==28993== Use --track-origins=yes to see where uninitialised values come+```++This reports that Node.js is not _completely_ clean as there is some memory+that was allocated but not freed when the process shut down. It is often+impractical/not worth being completely clean in this respect and Node.js+does a pretty good job only leaving on the order of 6KB that are+not freed on shutdown.++## An obvious memory leak++Leaks can be introduced in native addons and the following is a simple+example leak based on the "Hello world" addon from+[node-addon-examples](https://github.com/nodejs/node-addon-examples).++In this example, a loop which allocates ~1MB of memory and never frees it+has been added:++```C+++void* malloc_holder = nullptr;+napi_value Method(napi_env env, napi_callback_info info) {+  napi_status status;+  napi_value world;+  status = napi_create_string_utf8(env, "world", 5, &world);+  assert(status == napi_ok);++  // NEW LEAK HERE+  for (int i=0; i < 1024; i++) {+    malloc_holder = malloc(1024);+  }++  return world;+}+```++When trying to create a memory leak you need to ensure that+the compiler has not optimized out the code that creates+the leak. For example, assigning the result of `malloc()` to either a+global variable or a variable that is read the compiler will optimize+out the loop with the malloc and valgrind does not report+a problem (since one no longer exists in the code being run).++Running valgrind on this code shows the following:++```console+user1@minikube1:~/valgrind/node-addon-examples/1_hello_world/napi$ valgrind node hello.js+==1504== Memcheck, a memory error detector+==1504== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.+==1504== Using V#algrind-3.13.0 and LibVEX; rerun with -h for copyright info+==1504== Command: node hello.js+==1504==+==1504== Use of uninitialised value of size 8+==1504==    at 0x12F2279: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==1504==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==1504==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==1504==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==1504==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==1504==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==1504==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==1504==    by 0x12F3E9C: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==1504==    by 0x12F3C77: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==1504==    by 0xC7C9CF: v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==1504==    by 0xC7CE87: v8::internal::Execution::Call(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, int, v8::internal::Handle<v8::internal::Object>*) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==1504==    by 0xB4CF3A: v8::Function::Call(v8::Local<v8::Context>, v8::Local<v8::Value>, int, v8::Local<v8::Value>*) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==1504==+--1504-- WARNING: unhandled amd64-linux syscall: 332+--1504-- You may be able to write your own handler.+--1504-- Read the file README_MISSING_SYSCALL_OR_IOCTL.+--1504-- Nevertheless we consider this a bug.  Please report+--1504-- it at http://valgrind.org/support/bug_reports.html.+world+==1504==+==1504== HEAP SUMMARY:+==1504==     in use at exit: 1,008,003 bytes in 1,032 blocks+==1504==   total heap usage: 17,603 allocs, 16,571 frees, 18,306,103 bytes allocated+==1504==+==1504== LEAK SUMMARY:+==1504==    definitely lost: 996,064 bytes in 997 blocks+==1504==    indirectly lost: 0 bytes in 0 blocks+==1504==      possibly lost: 3,304 bytes in 4 blocks+==1504==    still reachable: 8,635 bytes in 31 blocks+==1504==                       of which reachable via heuristic:+==1504==                         multipleinheritance: 48 bytes in 1 blocks+==1504==         suppressed: 0 bytes in 0 blocks+==1504== Rerun with --leak-check=full to see details of leaked memory+==1504==+==1504== For counts of detected and suppressed errors, rerun with: -v+==1504== Use --track-origins=yes to see where uninitialised values come from+==1504== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)+```++Valgrind is reporting a problem as it shows 996,604 bytes as+definitely lost and the question is how to find where that memory was+allocated. The next step is to rerun as suggested in the+output with `--leak-check=full`:++``` bash+user1@minikube1:~/valgrind/node-addon-examples/1_hello_world/napi$ valgrind --leak-check=full node hello.js+==4174== Memcheck, a memory error detector+==4174== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.+==4174== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info+==4174== Command: node hello.js+==4174==+==4174== Use of uninitialised value of size 8+==4174==    at 0x12F2279: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x12F3E9C: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x12F3C77: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0xC7C9CF: v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0xC7CE87: v8::internal::Execution::Call(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, int, v8::internal::Handle<v8::internal::Object>*) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0xB4CF3A: v8::Function::Call(v8::Local<v8::Context>, v8::Local<v8::Value>, int, v8::Local<v8::Value>*) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==+--4174-- WARNING: unhandled amd64-linux syscall: 332+--4174-- You may be able to write your own handler.+--4174-- Read the file README_MISSING_SYSCALL_OR_IOCTL.+--4174-- Nevertheless we consider this a bug.  Please report+--4174-- it at http://valgrind.org/support/bug_reports.html.+world+==4174==+==4174== HEAP SUMMARY:+==4174==     in use at exit: 1,008,003 bytes in 1,032 blocks+==4174==   total heap usage: 17,606 allocs, 16,574 frees, 18,305,977 bytes allocated+==4174==+==4174== 64 bytes in 1 blocks are definitely lost in loss record 17 of 35+==4174==    at 0x4C3017F: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)+==4174==    by 0x9AEAD5: napi_module_register (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x4010732: call_init (dl-init.c:72)+==4174==    by 0x4010732: _dl_init (dl-init.c:119)+==4174==    by 0x40151FE: dl_open_worker (dl-open.c:522)+==4174==    by 0x5D052DE: _dl_catch_exception (dl-error-skeleton.c:196)+==4174==    by 0x40147C9: _dl_open (dl-open.c:605)+==4174==    by 0x4E3CF95: dlopen_doit (dlopen.c:66)+==4174==    by 0x5D052DE: _dl_catch_exception (dl-error-skeleton.c:196)+==4174==    by 0x5D0536E: _dl_catch_error (dl-error-skeleton.c:215)+==4174==    by 0x4E3D734: _dlerror_run (dlerror.c:162)+==4174==    by 0x4E3D050: dlopen@@GLIBC_2.2.5 (dlopen.c:87)+==4174==    by 0x9B29A0: node::binding::DLOpen(v8::FunctionCallbackInfo<v8::Value> const&)::{lambda(node::binding::DLib*)#1}::operator()(node::binding::DLib*) const (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==+==4174== 304 bytes in 1 blocks are possibly lost in loss record 27 of 35+==4174==    at 0x4C31B25: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)+==4174==    by 0x40134A6: allocate_dtv (dl-tls.c:286)+==4174==    by 0x40134A6: _dl_allocate_tls (dl-tls.c:530)+==4174==    by 0x5987227: allocate_stack (allocatestack.c:627)+==4174==    by 0x5987227: pthread_create@@GLIBC_2.2.5 (pthread_create.c:644)+==4174==    by 0xAAF9DC: node::inspector::Agent::Start(std::string const&, node::DebugOptions const&, std::shared_ptr<node::HostPort>, bool) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x9A8BE7: node::Environment::InitializeInspector(std::unique_ptr<node::inspector::ParentInspectorHandle, std::default_delete<node::inspector::ParentInspectorHandle> >) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0xA1C9A5: node::NodeMainInstance::CreateMainEnvironment(int*) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0xA1CB42: node::NodeMainInstance::Run() (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x9ACB67: node::Start(int, char**) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x5BBFB96: (below main) (libc-start.c:310)+==4174==+==4174== 2,000 bytes in 2 blocks are possibly lost in loss record 33 of 35+==4174==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)+==4174==    by 0x9794979: Method(napi_env__*, napi_callback_info__*) (in /home/user1/valgrind/node-addon-examples/1_hello_world/napi/build/Release/hello.node)+==4174==    by 0x98F764: v8impl::(anonymous namespace)::FunctionCallbackWrapper::Invoke(v8::FunctionCallbackInfo<v8::Value> const&) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0xBA6FC8: v8::internal::MaybeHandle<v8::internal::Object> v8::internal::(anonymous namespace)::HandleApiCallHelper<false>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::FunctionTemplateInfo>, v8::internal::Handle<v8::internal::Object>, v8::internal::BuiltinArguments) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0xBA8DB6: v8::internal::Builtin_HandleApiCall(int, unsigned long*, v8::internal::Isolate*) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x1376358: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==+==4174== 997,000 bytes in 997 blocks are definitely lost in loss record 35 of 35+==4174==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)+==4174==    by 0x9794979: Method(napi_env__*, napi_callback_info__*) (in /home/user1/valgrind/node-addon-examples/1_hello_world/napi/build/Release/hello.node)+==4174==    by 0x98F764: v8impl::(anonymous namespace)::FunctionCallbackWrapper::Invoke(v8::FunctionCallbackInfo<v8::Value> const&) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0xBA6FC8: v8::internal::MaybeHandle<v8::internal::Object> v8::internal::(anonymous namespace)::HandleApiCallHelper<false>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::FunctionTemplateInfo>, v8::internal::Handle<v8::internal::Object>, v8::internal::BuiltinArguments) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0xBA8DB6: v8::internal::Builtin_HandleApiCall(int, unsigned long*, v8::internal::Isolate*) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x1376358: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==+==4174== LEAK SUMMARY:+==4174==    definitely lost: 997,064 bytes in 998 blocks+==4174==    indirectly lost: 0 bytes in 0 blocks+==4174==      possibly lost: 2,304 bytes in 3 blocks+==4174==    still reachable: 8,635 bytes in 31 blocks+==4174==                       of which reachable via heuristic:+==4174==                         multipleinheritance: 48 bytes in 1 blocks+==4174==         suppressed: 0 bytes in 0 blocks+==4174== Reachable blocks (those to which a pointer was found) are not shown.+==4174== To see them, rerun with: --leak-check=full --show-leak-kinds=all+==4174==+==4174== For counts of detected and suppressed errors, rerun with: -v+==4174== Use --track-origins=yes to see where uninitialised values come from+==4174== ERROR SUMMARY: 5 errors from 5 contexts (suppressed: 0 from 0)+```++This is the most interesting report:
This is the most interesting part of the report:
mhdawson

comment created time in 21 days

Pull request review commentnodejs/node

doc: guide - using valgrind to debug memory leaks

+# Investigating Memory Leaks with valgrind++A Node.js process may run out of memory due to excessive consumption of+native memory. Native Memory is memory which is not managed by the+V8 Garbage collector and is allocated either by the Node.js runtime, its+dependencies or native [addons](https://nodejs.org/docs/latest/api/n-api.html).++This guide provides information on how to use valgrind to investigate these+issues.++## valgrind++[Valgrind](https://valgrind.org/docs/manual/quick-start.html) is a+tool available on Linux distributions which can be used to investigate+memory usage including identifying memory leaks (memory which is+allocated and not freed) and other memory related problems+like double freeing memory.++To use valgrind:++* Be patient, running under valgrind slows execution significantly+  due to the checks being performed.+* Reduce your test case to the smallest reproduce. Due to the slowdown it is+  important to run the minimum test case in order to be able to do it in+  a reasonable time.++## Installation++It is an optional package in most cases and must be installed explicitly.+For example on Debian/Ubuntu:++```console+apt-get install valgrind+```++## Invocation+The simplest invocation of valgrind is:++```console+valgrind node test.js+```++with the output being:++```console+user1@minikube1:~/valgrind/node-addon-examples/1_hello_world/napi$ valgrind node test.js+==28993== Memcheck, a memory error detector+==28993== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.+==28993== Using valgrind-3.13.0 and LibVEX; rerun with -h for copyright info+==28993== Command: node test.js+==28993==+==28993== Use of uninitialised value of size 8+==28993==    at 0x12F2279: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0x12F3E9C: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0x12F3C77: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0xC7C9CF: v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0xC7CE87: v8::internal::Execution::Call(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, int, v8::internal::Handle<v8::internal::Object>*) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0xB4CF3A: v8::Function::Call(v8::Local<v8::Context>, v8::Local<v8::Value>, int, v8::Local<v8::Value>*) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==+--28993-- WARNING: unhandled amd64-linux syscall: 332+--28993-- You may be able to write your own handler.+--28993-- Read the file README_MISSING_SYSCALL_OR_IOCTL.+--28993-- Nevertheless we consider this a bug.  Please report+--28993-- it at http://valgrind.org/support/bug_reports.html.+==28993==+==28993== HEAP SUMMARY:+==28993==     in use at exit: 6,140 bytes in 23 blocks+==28993==   total heap usage: 12,888 allocs, 12,865 frees, 13,033,244 bytes allocated+==28993==+==28993== LEAK SUMMARY:+==28993==    definitely lost: 0 bytes in 0 blocks+==28993==    indirectly lost: 0 bytes in 0 blocks+==28993==      possibly lost: 304 bytes in 1 blocks+==28993==    still reachable: 5,836 bytes in 22 blocks+==28993==         suppressed: 0 bytes in 0 blocks+==28993== Rerun with --leak-check=full to see details of leaked memory+==28993==+==28993== For counts of detected and suppressed errors, rerun with: -v+==28993== Use --track-origins=yes to see where uninitialised values come+```++This reports that Node.js is not _completely_ clean as there is some memory+that was allocated but not freed when the process shut down. It is often+impractical/not worth being completely clean in this respect and Node.js+does a pretty good job only leaving on the order of 6KB that are+not freed on shutdown.++## An obvious memory leak++Leaks can be introduced in native addons and the following is a simple+example leak based on the "Hello world" addon from+[node-addon-examples](https://github.com/nodejs/node-addon-examples).++In this example, a loop which allocates ~1MB of memory and never frees it+has been added:++```C+++void* malloc_holder = nullptr;+napi_value Method(napi_env env, napi_callback_info info) {+  napi_status status;+  napi_value world;+  status = napi_create_string_utf8(env, "world", 5, &world);+  assert(status == napi_ok);++  // NEW LEAK HERE+  for (int i=0; i < 1024; i++) {+    malloc_holder = malloc(1024);+  }++  return world;+}+```++When trying to create a memory leak you need to ensure that+the compiler has not optimized out the code that creates+the leak. For example, assigning the result of `malloc()` to either a+global variable or a variable that is read the compiler will optimize+out the loop with the malloc and valgrind does not report+a problem (since one no longer exists in the code being run).++Running valgrind on this code shows the following:++```console+user1@minikube1:~/valgrind/node-addon-examples/1_hello_world/napi$ valgrind node hello.js+==1504== Memcheck, a memory error detector+==1504== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.+==1504== Using V#algrind-3.13.0 and LibVEX; rerun with -h for copyright info+==1504== Command: node hello.js+==1504==+==1504== Use of uninitialised value of size 8+==1504==    at 0x12F2279: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==1504==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==1504==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==1504==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==1504==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==1504==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==1504==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==1504==    by 0x12F3E9C: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==1504==    by 0x12F3C77: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==1504==    by 0xC7C9CF: v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==1504==    by 0xC7CE87: v8::internal::Execution::Call(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, int, v8::internal::Handle<v8::internal::Object>*) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==1504==    by 0xB4CF3A: v8::Function::Call(v8::Local<v8::Context>, v8::Local<v8::Value>, int, v8::Local<v8::Value>*) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==1504==+--1504-- WARNING: unhandled amd64-linux syscall: 332+--1504-- You may be able to write your own handler.+--1504-- Read the file README_MISSING_SYSCALL_OR_IOCTL.+--1504-- Nevertheless we consider this a bug.  Please report+--1504-- it at http://valgrind.org/support/bug_reports.html.+world+==1504==+==1504== HEAP SUMMARY:+==1504==     in use at exit: 1,008,003 bytes in 1,032 blocks+==1504==   total heap usage: 17,603 allocs, 16,571 frees, 18,306,103 bytes allocated+==1504==+==1504== LEAK SUMMARY:+==1504==    definitely lost: 996,064 bytes in 997 blocks+==1504==    indirectly lost: 0 bytes in 0 blocks+==1504==      possibly lost: 3,304 bytes in 4 blocks+==1504==    still reachable: 8,635 bytes in 31 blocks+==1504==                       of which reachable via heuristic:+==1504==                         multipleinheritance: 48 bytes in 1 blocks+==1504==         suppressed: 0 bytes in 0 blocks+==1504== Rerun with --leak-check=full to see details of leaked memory+==1504==+==1504== For counts of detected and suppressed errors, rerun with: -v+==1504== Use --track-origins=yes to see where uninitialised values come from+==1504== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)+```++Valgrind is reporting a problem as it shows 996,604 bytes as+definitely lost and the question is how to find where that memory was+allocated. The next step is to rerun as suggested in the+output with `--leak-check=full`:++``` bash+user1@minikube1:~/valgrind/node-addon-examples/1_hello_world/napi$ valgrind --leak-check=full node hello.js+==4174== Memcheck, a memory error detector+==4174== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.+==4174== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info+==4174== Command: node hello.js+==4174==+==4174== Use of uninitialised value of size 8+==4174==    at 0x12F2279: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x12F3E9C: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x12F3C77: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0xC7C9CF: v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0xC7CE87: v8::internal::Execution::Call(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, int, v8::internal::Handle<v8::internal::Object>*) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0xB4CF3A: v8::Function::Call(v8::Local<v8::Context>, v8::Local<v8::Value>, int, v8::Local<v8::Value>*) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==+--4174-- WARNING: unhandled amd64-linux syscall: 332+--4174-- You may be able to write your own handler.+--4174-- Read the file README_MISSING_SYSCALL_OR_IOCTL.+--4174-- Nevertheless we consider this a bug.  Please report+--4174-- it at http://valgrind.org/support/bug_reports.html.+world+==4174==+==4174== HEAP SUMMARY:+==4174==     in use at exit: 1,008,003 bytes in 1,032 blocks+==4174==   total heap usage: 17,606 allocs, 16,574 frees, 18,305,977 bytes allocated+==4174==+==4174== 64 bytes in 1 blocks are definitely lost in loss record 17 of 35+==4174==    at 0x4C3017F: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)+==4174==    by 0x9AEAD5: napi_module_register (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x4010732: call_init (dl-init.c:72)+==4174==    by 0x4010732: _dl_init (dl-init.c:119)+==4174==    by 0x40151FE: dl_open_worker (dl-open.c:522)+==4174==    by 0x5D052DE: _dl_catch_exception (dl-error-skeleton.c:196)+==4174==    by 0x40147C9: _dl_open (dl-open.c:605)+==4174==    by 0x4E3CF95: dlopen_doit (dlopen.c:66)+==4174==    by 0x5D052DE: _dl_catch_exception (dl-error-skeleton.c:196)+==4174==    by 0x5D0536E: _dl_catch_error (dl-error-skeleton.c:215)+==4174==    by 0x4E3D734: _dlerror_run (dlerror.c:162)+==4174==    by 0x4E3D050: dlopen@@GLIBC_2.2.5 (dlopen.c:87)+==4174==    by 0x9B29A0: node::binding::DLOpen(v8::FunctionCallbackInfo<v8::Value> const&)::{lambda(node::binding::DLib*)#1}::operator()(node::binding::DLib*) const (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==+==4174== 304 bytes in 1 blocks are possibly lost in loss record 27 of 35+==4174==    at 0x4C31B25: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)+==4174==    by 0x40134A6: allocate_dtv (dl-tls.c:286)+==4174==    by 0x40134A6: _dl_allocate_tls (dl-tls.c:530)+==4174==    by 0x5987227: allocate_stack (allocatestack.c:627)+==4174==    by 0x5987227: pthread_create@@GLIBC_2.2.5 (pthread_create.c:644)+==4174==    by 0xAAF9DC: node::inspector::Agent::Start(std::string const&, node::DebugOptions const&, std::shared_ptr<node::HostPort>, bool) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x9A8BE7: node::Environment::InitializeInspector(std::unique_ptr<node::inspector::ParentInspectorHandle, std::default_delete<node::inspector::ParentInspectorHandle> >) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0xA1C9A5: node::NodeMainInstance::CreateMainEnvironment(int*) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0xA1CB42: node::NodeMainInstance::Run() (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x9ACB67: node::Start(int, char**) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x5BBFB96: (below main) (libc-start.c:310)+==4174==+==4174== 2,000 bytes in 2 blocks are possibly lost in loss record 33 of 35+==4174==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)+==4174==    by 0x9794979: Method(napi_env__*, napi_callback_info__*) (in /home/user1/valgrind/node-addon-examples/1_hello_world/napi/build/Release/hello.node)+==4174==    by 0x98F764: v8impl::(anonymous namespace)::FunctionCallbackWrapper::Invoke(v8::FunctionCallbackInfo<v8::Value> const&) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0xBA6FC8: v8::internal::MaybeHandle<v8::internal::Object> v8::internal::(anonymous namespace)::HandleApiCallHelper<false>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::FunctionTemplateInfo>, v8::internal::Handle<v8::internal::Object>, v8::internal::BuiltinArguments) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0xBA8DB6: v8::internal::Builtin_HandleApiCall(int, unsigned long*, v8::internal::Isolate*) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x1376358: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==+==4174== 997,000 bytes in 997 blocks are definitely lost in loss record 35 of 35+==4174==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)+==4174==    by 0x9794979: Method(napi_env__*, napi_callback_info__*) (in /home/user1/valgrind/node-addon-examples/1_hello_world/napi/build/Release/hello.node)+==4174==    by 0x98F764: v8impl::(anonymous namespace)::FunctionCallbackWrapper::Invoke(v8::FunctionCallbackInfo<v8::Value> const&) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0xBA6FC8: v8::internal::MaybeHandle<v8::internal::Object> v8::internal::(anonymous namespace)::HandleApiCallHelper<false>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::FunctionTemplateInfo>, v8::internal::Handle<v8::internal::Object>, v8::internal::BuiltinArguments) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0xBA8DB6: v8::internal::Builtin_HandleApiCall(int, unsigned long*, v8::internal::Isolate*) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x1376358: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==+==4174== LEAK SUMMARY:+==4174==    definitely lost: 997,064 bytes in 998 blocks+==4174==    indirectly lost: 0 bytes in 0 blocks+==4174==      possibly lost: 2,304 bytes in 3 blocks+==4174==    still reachable: 8,635 bytes in 31 blocks+==4174==                       of which reachable via heuristic:+==4174==                         multipleinheritance: 48 bytes in 1 blocks+==4174==         suppressed: 0 bytes in 0 blocks+==4174== Reachable blocks (those to which a pointer was found) are not shown.+==4174== To see them, rerun with: --leak-check=full --show-leak-kinds=all+==4174==+==4174== For counts of detected and suppressed errors, rerun with: -v+==4174== Use --track-origins=yes to see where uninitialised values come from+==4174== ERROR SUMMARY: 5 errors from 5 contexts (suppressed: 0 from 0)+```++This is the most interesting report:++```console+==4174== 997,000 bytes in 997 blocks are definitely lost in loss record 35 of 35+==4174==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)+==4174==    by 0x9794979: Method(napi_env__*, napi_callback_info__*) (in /home/user1/valgrind/node-addon-examples/1_hello_world/napi/build/Release/hello.node)+==4174==    by 0x98F764: v8impl::(anonymous namespace)::FunctionCallbackWrapper::Invoke(v8::FunctionCallbackInfo<v8::Value> const&) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0xBA6FC8: v8::internal::MaybeHandle<v8::internal::Object> v8::internal::(anonymous namespace)::HandleApiCallHelper<false>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::FunctionTemplateInfo>, v8::internal::Handle<v8::internal::Object>, v8::internal::BuiltinArguments) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0xBA8DB6: v8::internal::Builtin_HandleApiCall(int, unsigned long*, v8::internal::Isolate*) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x1376358: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+```++From the stack trace we can tell that the leak came from a native addon:++```console+==4174==    by 0x9794979: Method(napi_env__*, napi_callback_info__*) (in /home/user1/valgrind/node-addon-examples/1_hello_world/napi/build/Release/hello.node)+```++What we can't tell is where in the native addon the memory is being+allocated. This is because by default the addon is compiled without+the debug symbols which valgrind needs to be able to provide more+information.++## Enabling debug symbols to get more information++Leaks may be either in addons or Node.js itself. The sections which+follow cover the steps needed to enable debug symbols to get more info.++### Native addons++To enable debug symbols for all of your addons that are compiled on+install use:++```console+npm install --debug+```++Any options which are not consumed by npm are passed on to node-gyp and this+results in the addons being compiled with the debug option.++If the native addon contains pre-built binaries you will need to force+a rebuild.+

```console
npm install --debug
npm rebuild
```

mhdawson

comment created time in 21 days

Pull request review commentnodejs/node

doc: guide - using valgrind to debug memory leaks

+# Investigating Memory Leaks with valgrind++A Node.js process may run out of memory due to excessive consumption of+native memory. Native Memory is memory which is not managed by the+V8 Garbage collector and is allocated either by the Node.js runtime, its+dependencies or native [addons](https://nodejs.org/docs/latest/api/n-api.html).++This guide provides information on how to use valgrind to investigate these+issues.++## valgrind++[Valgrind](https://valgrind.org/docs/manual/quick-start.html) is a+tool available on Linux distributions which can be used to investigate+memory usage including identifying memory leaks (memory which is+allocated and not freed) and other memory related problems+like double freeing memory.++To use valgrind:++* Be patient, running under valgrind slows execution significantly+  due to the checks being performed.+* Reduce your test case to the smallest reproduce. Due to the slowdown it is+  important to run the minimum test case in order to be able to do it in+  a reasonable time.++## Installation++It is an optional package in most cases and must be installed explicitly.+For example on Debian/Ubuntu:++```console+apt-get install valgrind+```++## Invocation+The simplest invocation of valgrind is:++```console+valgrind node test.js+```++with the output being:++```console+user1@minikube1:~/valgrind/node-addon-examples/1_hello_world/napi$ valgrind node test.js+==28993== Memcheck, a memory error detector+==28993== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.+==28993== Using valgrind-3.13.0 and LibVEX; rerun with -h for copyright info+==28993== Command: node test.js+==28993==+==28993== Use of uninitialised value of size 8+==28993==    at 0x12F2279: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0x12F3E9C: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0x12F3C77: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0xC7C9CF: v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0xC7CE87: v8::internal::Execution::Call(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, int, v8::internal::Handle<v8::internal::Object>*) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0xB4CF3A: v8::Function::Call(v8::Local<v8::Context>, v8::Local<v8::Value>, int, v8::Local<v8::Value>*) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==+--28993-- WARNING: unhandled amd64-linux syscall: 332+--28993-- You may be able to write your own handler.+--28993-- Read the file README_MISSING_SYSCALL_OR_IOCTL.+--28993-- Nevertheless we consider this a bug.  Please report+--28993-- it at http://valgrind.org/support/bug_reports.html.+==28993==+==28993== HEAP SUMMARY:+==28993==     in use at exit: 6,140 bytes in 23 blocks+==28993==   total heap usage: 12,888 allocs, 12,865 frees, 13,033,244 bytes allocated+==28993==+==28993== LEAK SUMMARY:+==28993==    definitely lost: 0 bytes in 0 blocks+==28993==    indirectly lost: 0 bytes in 0 blocks+==28993==      possibly lost: 304 bytes in 1 blocks+==28993==    still reachable: 5,836 bytes in 22 blocks+==28993==         suppressed: 0 bytes in 0 blocks+==28993== Rerun with --leak-check=full to see details of leaked memory+==28993==+==28993== For counts of detected and suppressed errors, rerun with: -v+==28993== Use --track-origins=yes to see where uninitialised values come+```++This reports that Node.js is not _completely_ clean as there is some memory+that was allocated but not freed when the process shut down. It is often+impractical/not worth being completely clean in this respect and Node.js+does a pretty good job only leaving on the order of 6KB that are+not freed on shutdown.++## An obvious memory leak++Leaks can be introduced in native addons and the following is a simple+example leak based on the "Hello world" addon from+[node-addon-examples](https://github.com/nodejs/node-addon-examples).++In this example, a loop which allocates ~1MB of memory and never frees it+has been added:++```C+++void* malloc_holder = nullptr;+napi_value Method(napi_env env, napi_callback_info info) {+  napi_status status;+  napi_value world;+  status = napi_create_string_utf8(env, "world", 5, &world);+  assert(status == napi_ok);++  // NEW LEAK HERE+  for (int i=0; i < 1024; i++) {+    malloc_holder = malloc(1024);+  }++  return world;+}+```++When trying to create a memory leak you need to ensure that+the compiler has not optimized out the code that creates+the leak. For example, assigning the result of `malloc()` to either a+global variable or a variable that is read the compiler will optimize+out the loop with the malloc and valgrind does not report

Perhaps, to clear it up a bit:

For example, by assigning the result of the allocation (`malloc()`) to either a
global variable or a variable that will be read afterwards the compiler will not optimize
it out along with the malloc and Valgrind will properly report the memory leak. If `malloc_holder`
in the example above is made into a local variable then the compiler may freely remove
it along with the allocations (since it is not used) and Valgrind will not find any leaks since they
will no longer exist in the code being run.
mhdawson

comment created time in 21 days

Pull request review commentnodejs/node

doc: guide - using valgrind to debug memory leaks

+# Investigating Memory Leaks with valgrind++A Node.js process may run out of memory due to excessive consumption of+native memory. Native Memory is memory which is not managed by the+V8 Garbage collector and is allocated either by the Node.js runtime, its+dependencies or native [addons](https://nodejs.org/docs/latest/api/n-api.html).++This guide provides information on how to use valgrind to investigate these+issues.++## valgrind++[Valgrind](https://valgrind.org/docs/manual/quick-start.html) is a+tool available on Linux distributions which can be used to investigate+memory usage including identifying memory leaks (memory which is+allocated and not freed) and other memory related problems+like double freeing memory.++To use valgrind:++* Be patient, running under valgrind slows execution significantly+  due to the checks being performed.+* Reduce your test case to the smallest reproduce. Due to the slowdown it is+  important to run the minimum test case in order to be able to do it in+  a reasonable time.++## Installation++It is an optional package in most cases and must be installed explicitly.+For example on Debian/Ubuntu:++```console+apt-get install valgrind+```++## Invocation+The simplest invocation of valgrind is:++```console+valgrind node test.js+```++with the output being:++```console+user1@minikube1:~/valgrind/node-addon-examples/1_hello_world/napi$ valgrind node test.js+==28993== Memcheck, a memory error detector+==28993== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.+==28993== Using valgrind-3.13.0 and LibVEX; rerun with -h for copyright info+==28993== Command: node test.js+==28993==+==28993== Use of uninitialised value of size 8+==28993==    at 0x12F2279: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0x12F3E9C: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0x12F3C77: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0xC7C9CF: v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0xC7CE87: v8::internal::Execution::Call(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, int, v8::internal::Handle<v8::internal::Object>*) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0xB4CF3A: v8::Function::Call(v8::Local<v8::Context>, v8::Local<v8::Value>, int, v8::Local<v8::Value>*) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==+--28993-- WARNING: unhandled amd64-linux syscall: 332+--28993-- You may be able to write your own handler.+--28993-- Read the file README_MISSING_SYSCALL_OR_IOCTL.+--28993-- Nevertheless we consider this a bug.  Please report+--28993-- it at http://valgrind.org/support/bug_reports.html.+==28993==+==28993== HEAP SUMMARY:+==28993==     in use at exit: 6,140 bytes in 23 blocks+==28993==   total heap usage: 12,888 allocs, 12,865 frees, 13,033,244 bytes allocated+==28993==+==28993== LEAK SUMMARY:+==28993==    definitely lost: 0 bytes in 0 blocks+==28993==    indirectly lost: 0 bytes in 0 blocks+==28993==      possibly lost: 304 bytes in 1 blocks+==28993==    still reachable: 5,836 bytes in 22 blocks+==28993==         suppressed: 0 bytes in 0 blocks+==28993== Rerun with --leak-check=full to see details of leaked memory+==28993==+==28993== For counts of detected and suppressed errors, rerun with: -v+==28993== Use --track-origins=yes to see where uninitialised values come+```++This reports that Node.js is not _completely_ clean as there is some memory+that was allocated but not freed when the process shut down. It is often+impractical/not worth being completely clean in this respect and Node.js+does a pretty good job only leaving on the order of 6KB that are+not freed on shutdown.++## An obvious memory leak++Leaks can be introduced in native addons and the following is a simple+example leak based on the "Hello world" addon from+[node-addon-examples](https://github.com/nodejs/node-addon-examples).++In this example, a loop which allocates ~1MB of memory and never frees it+has been added:++```C+++void* malloc_holder = nullptr;+napi_value Method(napi_env env, napi_callback_info info) {+  napi_status status;+  napi_value world;+  status = napi_create_string_utf8(env, "world", 5, &world);+  assert(status == napi_ok);++  // NEW LEAK HERE+  for (int i=0; i < 1024; i++) {+    malloc_holder = malloc(1024);+  }++  return world;+}+```++When trying to create a memory leak you need to ensure that+the compiler has not optimized out the code that creates+the leak. For example, assigning the result of `malloc()` to either a+global variable or a variable that is read the compiler will optimize+out the loop with the malloc and valgrind does not report+a problem (since one no longer exists in the code being run).++Running valgrind on this code shows the following:++```console+user1@minikube1:~/valgrind/node-addon-examples/1_hello_world/napi$ valgrind node hello.js+==1504== Memcheck, a memory error detector+==1504== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.+==1504== Using V#algrind-3.13.0 and LibVEX; rerun with -h for copyright info+==1504== Command: node hello.js+==1504==+==1504== Use of uninitialised value of size 8+==1504==    at 0x12F2279: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==1504==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==1504==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==1504==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==1504==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==1504==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==1504==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==1504==    by 0x12F3E9C: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==1504==    by 0x12F3C77: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==1504==    by 0xC7C9CF: v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==1504==    by 0xC7CE87: v8::internal::Execution::Call(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, int, v8::internal::Handle<v8::internal::Object>*) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==1504==    by 0xB4CF3A: v8::Function::Call(v8::Local<v8::Context>, v8::Local<v8::Value>, int, v8::Local<v8::Value>*) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==1504==+--1504-- WARNING: unhandled amd64-linux syscall: 332+--1504-- You may be able to write your own handler.+--1504-- Read the file README_MISSING_SYSCALL_OR_IOCTL.+--1504-- Nevertheless we consider this a bug.  Please report+--1504-- it at http://valgrind.org/support/bug_reports.html.+world+==1504==+==1504== HEAP SUMMARY:+==1504==     in use at exit: 1,008,003 bytes in 1,032 blocks+==1504==   total heap usage: 17,603 allocs, 16,571 frees, 18,306,103 bytes allocated+==1504==+==1504== LEAK SUMMARY:+==1504==    definitely lost: 996,064 bytes in 997 blocks+==1504==    indirectly lost: 0 bytes in 0 blocks+==1504==      possibly lost: 3,304 bytes in 4 blocks+==1504==    still reachable: 8,635 bytes in 31 blocks+==1504==                       of which reachable via heuristic:+==1504==                         multipleinheritance: 48 bytes in 1 blocks+==1504==         suppressed: 0 bytes in 0 blocks+==1504== Rerun with --leak-check=full to see details of leaked memory+==1504==+==1504== For counts of detected and suppressed errors, rerun with: -v+==1504== Use --track-origins=yes to see where uninitialised values come from+==1504== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)+```++Valgrind is reporting a problem as it shows 996,604 bytes as+definitely lost and the question is how to find where that memory was+allocated. The next step is to rerun as suggested in the+output with `--leak-check=full`:++``` bash+user1@minikube1:~/valgrind/node-addon-examples/1_hello_world/napi$ valgrind --leak-check=full node hello.js+==4174== Memcheck, a memory error detector+==4174== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.+==4174== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info+==4174== Command: node hello.js+==4174==+==4174== Use of uninitialised value of size 8+==4174==    at 0x12F2279: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x12F3E9C: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x12F3C77: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0xC7C9CF: v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0xC7CE87: v8::internal::Execution::Call(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, int, v8::internal::Handle<v8::internal::Object>*) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0xB4CF3A: v8::Function::Call(v8::Local<v8::Context>, v8::Local<v8::Value>, int, v8::Local<v8::Value>*) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==+--4174-- WARNING: unhandled amd64-linux syscall: 332+--4174-- You may be able to write your own handler.+--4174-- Read the file README_MISSING_SYSCALL_OR_IOCTL.+--4174-- Nevertheless we consider this a bug.  Please report+--4174-- it at http://valgrind.org/support/bug_reports.html.+world+==4174==+==4174== HEAP SUMMARY:+==4174==     in use at exit: 1,008,003 bytes in 1,032 blocks+==4174==   total heap usage: 17,606 allocs, 16,574 frees, 18,305,977 bytes allocated+==4174==+==4174== 64 bytes in 1 blocks are definitely lost in loss record 17 of 35+==4174==    at 0x4C3017F: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)+==4174==    by 0x9AEAD5: napi_module_register (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x4010732: call_init (dl-init.c:72)+==4174==    by 0x4010732: _dl_init (dl-init.c:119)+==4174==    by 0x40151FE: dl_open_worker (dl-open.c:522)+==4174==    by 0x5D052DE: _dl_catch_exception (dl-error-skeleton.c:196)+==4174==    by 0x40147C9: _dl_open (dl-open.c:605)+==4174==    by 0x4E3CF95: dlopen_doit (dlopen.c:66)+==4174==    by 0x5D052DE: _dl_catch_exception (dl-error-skeleton.c:196)+==4174==    by 0x5D0536E: _dl_catch_error (dl-error-skeleton.c:215)+==4174==    by 0x4E3D734: _dlerror_run (dlerror.c:162)+==4174==    by 0x4E3D050: dlopen@@GLIBC_2.2.5 (dlopen.c:87)+==4174==    by 0x9B29A0: node::binding::DLOpen(v8::FunctionCallbackInfo<v8::Value> const&)::{lambda(node::binding::DLib*)#1}::operator()(node::binding::DLib*) const (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==+==4174== 304 bytes in 1 blocks are possibly lost in loss record 27 of 35+==4174==    at 0x4C31B25: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)+==4174==    by 0x40134A6: allocate_dtv (dl-tls.c:286)+==4174==    by 0x40134A6: _dl_allocate_tls (dl-tls.c:530)+==4174==    by 0x5987227: allocate_stack (allocatestack.c:627)+==4174==    by 0x5987227: pthread_create@@GLIBC_2.2.5 (pthread_create.c:644)+==4174==    by 0xAAF9DC: node::inspector::Agent::Start(std::string const&, node::DebugOptions const&, std::shared_ptr<node::HostPort>, bool) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x9A8BE7: node::Environment::InitializeInspector(std::unique_ptr<node::inspector::ParentInspectorHandle, std::default_delete<node::inspector::ParentInspectorHandle> >) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0xA1C9A5: node::NodeMainInstance::CreateMainEnvironment(int*) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0xA1CB42: node::NodeMainInstance::Run() (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x9ACB67: node::Start(int, char**) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x5BBFB96: (below main) (libc-start.c:310)+==4174==+==4174== 2,000 bytes in 2 blocks are possibly lost in loss record 33 of 35+==4174==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)+==4174==    by 0x9794979: Method(napi_env__*, napi_callback_info__*) (in /home/user1/valgrind/node-addon-examples/1_hello_world/napi/build/Release/hello.node)+==4174==    by 0x98F764: v8impl::(anonymous namespace)::FunctionCallbackWrapper::Invoke(v8::FunctionCallbackInfo<v8::Value> const&) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0xBA6FC8: v8::internal::MaybeHandle<v8::internal::Object> v8::internal::(anonymous namespace)::HandleApiCallHelper<false>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::FunctionTemplateInfo>, v8::internal::Handle<v8::internal::Object>, v8::internal::BuiltinArguments) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0xBA8DB6: v8::internal::Builtin_HandleApiCall(int, unsigned long*, v8::internal::Isolate*) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x1376358: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==+==4174== 997,000 bytes in 997 blocks are definitely lost in loss record 35 of 35+==4174==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)+==4174==    by 0x9794979: Method(napi_env__*, napi_callback_info__*) (in /home/user1/valgrind/node-addon-examples/1_hello_world/napi/build/Release/hello.node)+==4174==    by 0x98F764: v8impl::(anonymous namespace)::FunctionCallbackWrapper::Invoke(v8::FunctionCallbackInfo<v8::Value> const&) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0xBA6FC8: v8::internal::MaybeHandle<v8::internal::Object> v8::internal::(anonymous namespace)::HandleApiCallHelper<false>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::FunctionTemplateInfo>, v8::internal::Handle<v8::internal::Object>, v8::internal::BuiltinArguments) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0xBA8DB6: v8::internal::Builtin_HandleApiCall(int, unsigned long*, v8::internal::Isolate*) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x1376358: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==+==4174== LEAK SUMMARY:+==4174==    definitely lost: 997,064 bytes in 998 blocks+==4174==    indirectly lost: 0 bytes in 0 blocks+==4174==      possibly lost: 2,304 bytes in 3 blocks+==4174==    still reachable: 8,635 bytes in 31 blocks+==4174==                       of which reachable via heuristic:+==4174==                         multipleinheritance: 48 bytes in 1 blocks+==4174==         suppressed: 0 bytes in 0 blocks+==4174== Reachable blocks (those to which a pointer was found) are not shown.+==4174== To see them, rerun with: --leak-check=full --show-leak-kinds=all+==4174==+==4174== For counts of detected and suppressed errors, rerun with: -v+==4174== Use --track-origins=yes to see where uninitialised values come from+==4174== ERROR SUMMARY: 5 errors from 5 contexts (suppressed: 0 from 0)+```++This is the most interesting report:++```console+==4174== 997,000 bytes in 997 blocks are definitely lost in loss record 35 of 35+==4174==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)+==4174==    by 0x9794979: Method(napi_env__*, napi_callback_info__*) (in /home/user1/valgrind/node-addon-examples/1_hello_world/napi/build/Release/hello.node)+==4174==    by 0x98F764: v8impl::(anonymous namespace)::FunctionCallbackWrapper::Invoke(v8::FunctionCallbackInfo<v8::Value> const&) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0xBA6FC8: v8::internal::MaybeHandle<v8::internal::Object> v8::internal::(anonymous namespace)::HandleApiCallHelper<false>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::FunctionTemplateInfo>, v8::internal::Handle<v8::internal::Object>, v8::internal::BuiltinArguments) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0xBA8DB6: v8::internal::Builtin_HandleApiCall(int, unsigned long*, v8::internal::Isolate*) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x1376358: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+```++From the stack trace we can tell that the leak came from a native addon:++```console+==4174==    by 0x9794979: Method(napi_env__*, napi_callback_info__*) (in /home/user1/valgrind/node-addon-examples/1_hello_world/napi/build/Release/hello.node)+```++What we can't tell is where in the native addon the memory is being+allocated. This is because by default the addon is compiled without+the debug symbols which valgrind needs to be able to provide more+information.++## Enabling debug symbols to get more information++Leaks may be either in addons or Node.js itself. The sections which+follow cover the steps needed to enable debug symbols to get more info.++### Native addons++To enable debug symbols for all of your addons that are compiled on+install use:++```console+npm install --debug+```++Any options which are not consumed by npm are passed on to node-gyp and this+results in the addons being compiled with the debug option.++If the native addon contains pre-built binaries you will need to force+a rebuild.++The next step is to run valgrind after the rebuild. This time the information+for the leaking location includes the name of the source file and the+line number:++```console+==18481== 997,000 bytes in 997 blocks are definitely lost in loss record 35 of 35+==18481==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)+>>>>> ==18481==    by 0x9794989: Method(napi_env__*, napi_callback_info__*) (hello.cc:13)  <<<<<+==18481==    by 0x98F764: v8impl::(anonymous namespace)::FunctionCallbackWrapper::Invoke(v8::FunctionCallbackInfo<v8::Value> const&) (in /home/user1/val  grind/node-v12.14.1-linux-x64/bin/node)+==18481==    by 0xBA6FC8: v8::internal::MaybeHandle<v8::internal::Object> v8::internal::(anonymous namespace)::HandleApiCallHelper<false>(v8::internal::  Isolate*, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::FunctionTem  plateInfo>, v8::internal::Handle<v8::internal::Object>, v8::internal::BuiltinArguments) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==18481==    by 0xBA8DB6: v8::internal::Builtin_HandleApiCall(int, unsigned long*, v8::internal::Isolate*) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==18481==    by 0x1376358: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==18481==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==18481==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==18481==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==18481==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==18481==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==18481==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+```++This new output shows us exactly where the leak is occurring in the file `hello.cc`:++```C+++  6 void* malloc_holder = nullptr;+  7 napi_value Method(napi_env env, napi_callback_info info) {+  8   napi_status status;+  9   napi_value world;+ 10   status = napi_create_string_utf8(env, "world", 5, &world);+ 11   assert(status == napi_ok);+ 12   for (int i=0; i< 1000; i++) {+ 13     malloc_holder = malloc(1000);  // <<<<<< This is where we are allocating the memory that is not freed+ 14   }+ 15   return world;+ 16 }+```++### Node.js binary++If the leak is not in an addon and is instead in the Node.js binary itself,+you may need to compile node yourself and turn on debug symbols. Looking at+this entry reported by valgrind, with a release binary we see:++```console+ ==4174== 304 bytes in 1 blocks are possibly lost in loss record 27 of 35+==4174==    at 0x4C31B25: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)+==4174==    by 0x40134A6: allocate_dtv (dl-tls.c:286)+==4174==    by 0x40134A6: _dl_allocate_tls (dl-tls.c:530)+==4174==    by 0x5987227: allocate_stack (allocatestack.c:627)+==4174==    by 0x5987227: pthread_create@@GLIBC_2.2.5 (pthread_create.c:644)+==4174==    by 0xAAF9DC: node::inspector::Agent::Start(std::string const&, node::DebugOptions const&, std::shared_ptr<node::HostPort>, bool) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x9A8BE7: node::Environment::InitializeInspector(std::unique_ptr<node::inspector::ParentInspectorHandle, std::default_delete<node::inspector::ParentInspectorHandle> >) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0xA1C9A5: node::NodeMainInstance::CreateMainEnvironment(int*) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0xA1CB42: node::NodeMainInstance::Run() (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x9ACB67: node::Start(int, char**) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==4174==    by 0x5BBFB96: (below main) (libc-start.c:310)+```++This gives us some information of where to look (`node::inspector::Agent::Start`)+but not where in that function. We get more information than you might expect+(or see by default with addons) because the Node.js binary exports many of+its symbols using `-rdynamic` so that they can be used by addons. If the stack+gives you enough information to track down where the leak is, that's great,+otherwise the next step is to compile a debug build of Node.js.++To get additional information with valgrind:++* Check out the Node.js source corresponding to the release that you+  want to debug. For example:+```console+git clone https://github.com/nodejs/node.git+git checkout v12.14.1+```+* Compile with debug enabled (for additional info see+[building a debug build](https://github.com/nodejs/node/blob/v12.14.1/BUILDING.md#building-a-debug-build)).+For example, on *nix:+```console+./configure --debug+make  -j4
make -j4
mhdawson

comment created time in 21 days

Pull request review commentnodejs/node

doc: guide - using valgrind to debug memory leaks

+# Investigating Memory Leaks with valgrind++A Node.js process may run out of memory due to excessive consumption of+native memory. Native Memory is memory which is not managed by the+V8 Garbage collector and is allocated either by the Node.js runtime, its+dependencies or native [addons](https://nodejs.org/docs/latest/api/n-api.html).++This guide provides information on how to use valgrind to investigate these+issues.++## valgrind++[Valgrind](https://valgrind.org/docs/manual/quick-start.html) is a+tool available on Linux distributions which can be used to investigate+memory usage including identifying memory leaks (memory which is+allocated and not freed) and other memory related problems+like double freeing memory.++To use valgrind:++* Be patient, running under valgrind slows execution significantly+  due to the checks being performed.+* Reduce your test case to the smallest reproduce. Due to the slowdown it is+  important to run the minimum test case in order to be able to do it in+  a reasonable time.++## Installation++It is an optional package in most cases and must be installed explicitly.+For example on Debian/Ubuntu:++```console+apt-get install valgrind+```++## Invocation+The simplest invocation of valgrind is:++```console+valgrind node test.js+```++with the output being:++```console+user1@minikube1:~/valgrind/node-addon-examples/1_hello_world/napi$ valgrind node test.js+==28993== Memcheck, a memory error detector+==28993== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.+==28993== Using valgrind-3.13.0 and LibVEX; rerun with -h for copyright info+==28993== Command: node test.js+==28993==+==28993== Use of uninitialised value of size 8+==28993==    at 0x12F2279: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0x12F3E9C: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0x12F3C77: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0xC7C9CF: v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0xC7CE87: v8::internal::Execution::Call(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, int, v8::internal::Handle<v8::internal::Object>*) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0xB4CF3A: v8::Function::Call(v8::Local<v8::Context>, v8::Local<v8::Value>, int, v8::Local<v8::Value>*) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==+--28993-- WARNING: unhandled amd64-linux syscall: 332+--28993-- You may be able to write your own handler.+--28993-- Read the file README_MISSING_SYSCALL_OR_IOCTL.+--28993-- Nevertheless we consider this a bug.  Please report+--28993-- it at http://valgrind.org/support/bug_reports.html.+==28993==+==28993== HEAP SUMMARY:+==28993==     in use at exit: 6,140 bytes in 23 blocks+==28993==   total heap usage: 12,888 allocs, 12,865 frees, 13,033,244 bytes allocated+==28993==+==28993== LEAK SUMMARY:+==28993==    definitely lost: 0 bytes in 0 blocks+==28993==    indirectly lost: 0 bytes in 0 blocks+==28993==      possibly lost: 304 bytes in 1 blocks+==28993==    still reachable: 5,836 bytes in 22 blocks+==28993==         suppressed: 0 bytes in 0 blocks+==28993== Rerun with --leak-check=full to see details of leaked memory+==28993==+==28993== For counts of detected and suppressed errors, rerun with: -v+==28993== Use --track-origins=yes to see where uninitialised values come+```++This reports that Node.js is not _completely_ clean as there is some memory+that was allocated but not freed when the process shut down. It is often+impractical/not worth being completely clean in this respect and Node.js

Could we add the 'why' around It is often impractical/not worth being completely clean in this respect so that people not familiar with the topic will at least get a pointer of where to look? Maybe something like most modern operating systems will clean up the memory of the process after the shutdown while doing this manually for _all_ of the cases may have a negative impact on the code complexity and shutdown times. Nevertheless, this does _not_ mean that it's fine to leave not freed memory, just that in some cases it's a necessary evil. (my wording could obviously be better)

mhdawson

comment created time in 21 days

Pull request review commentnodejs/node

doc: guide - using valgrind to debug memory leaks

+# Investigating Memory Leaks with valgrind++A Node.js process may run out of memory due to excessive consumption of+native memory. Native Memory is memory which is not managed by the+V8 Garbage collector and is allocated either by the Node.js runtime, its+dependencies or native [addons](https://nodejs.org/docs/latest/api/n-api.html).++This guide provides information on how to use valgrind to investigate these+issues.++## valgrind++[Valgrind](https://valgrind.org/docs/manual/quick-start.html) is a+tool available on Linux distributions which can be used to investigate+memory usage including identifying memory leaks (memory which is+allocated and not freed) and other memory related problems+like double freeing memory.++To use valgrind:++* Be patient, running under valgrind slows execution significantly+  due to the checks being performed.+* Reduce your test case to the smallest reproduce. Due to the slowdown it is+  important to run the minimum test case in order to be able to do it in+  a reasonable time.++## Installation++It is an optional package in most cases and must be installed explicitly.+For example on Debian/Ubuntu:++```console+apt-get install valgrind+```++## Invocation+The simplest invocation of valgrind is:++```console+valgrind node test.js+```++with the output being:++```console+user1@minikube1:~/valgrind/node-addon-examples/1_hello_world/napi$ valgrind node test.js+==28993== Memcheck, a memory error detector+==28993== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.+==28993== Using valgrind-3.13.0 and LibVEX; rerun with -h for copyright info+==28993== Command: node test.js+==28993==+==28993== Use of uninitialised value of size 8+==28993==    at 0x12F2279: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0x12F68A3: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0x12F3E9C: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0x12F3C77: ??? (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0xC7C9CF: v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0xC7CE87: v8::internal::Execution::Call(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, int, v8::internal::Handle<v8::internal::Object>*) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==    by 0xB4CF3A: v8::Function::Call(v8::Local<v8::Context>, v8::Local<v8::Value>, int, v8::Local<v8::Value>*) (in /home/user1/valgrind/node-v12.14.1-linux-x64/bin/node)+==28993==+--28993-- WARNING: unhandled amd64-linux syscall: 332+--28993-- You may be able to write your own handler.+--28993-- Read the file README_MISSING_SYSCALL_OR_IOCTL.+--28993-- Nevertheless we consider this a bug.  Please report+--28993-- it at http://valgrind.org/support/bug_reports.html.+==28993==+==28993== HEAP SUMMARY:+==28993==     in use at exit: 6,140 bytes in 23 blocks+==28993==   total heap usage: 12,888 allocs, 12,865 frees, 13,033,244 bytes allocated+==28993==+==28993== LEAK SUMMARY:+==28993==    definitely lost: 0 bytes in 0 blocks+==28993==    indirectly lost: 0 bytes in 0 blocks+==28993==      possibly lost: 304 bytes in 1 blocks+==28993==    still reachable: 5,836 bytes in 22 blocks+==28993==         suppressed: 0 bytes in 0 blocks+==28993== Rerun with --leak-check=full to see details of leaked memory+==28993==+==28993== For counts of detected and suppressed errors, rerun with: -v+==28993== Use --track-origins=yes to see where uninitialised values come+```++This reports that Node.js is not _completely_ clean as there is some memory+that was allocated but not freed when the process shut down. It is often+impractical/not worth being completely clean in this respect and Node.js+does a pretty good job only leaving on the order of 6KB that are+not freed on shutdown.++## An obvious memory leak++Leaks can be introduced in native addons and the following is a simple+example leak based on the "Hello world" addon from+[node-addon-examples](https://github.com/nodejs/node-addon-examples).++In this example, a loop which allocates ~1MB of memory and never frees it+has been added:++```C+++void* malloc_holder = nullptr;+napi_value Method(napi_env env, napi_callback_info info) {+  napi_status status;+  napi_value world;+  status = napi_create_string_utf8(env, "world", 5, &world);+  assert(status == napi_ok);++  // NEW LEAK HERE+  for (int i=0; i < 1024; i++) {+    malloc_holder = malloc(1024);+  }++  return world;+}+```++When trying to create a memory leak you need to ensure that+the compiler has not optimized out the code that creates+the leak. For example, assigning the result of `malloc()` to either a+global variable or a variable that is read the compiler will optimize+out the loop with the malloc and valgrind does not report+a problem (since one no longer exists in the code being run).
a problem (since it no longer exists in the code being run).
mhdawson

comment created time in 21 days

Pull request review commentnodejs/node

stream: fix finished w/ 'close' before 'finish'

 function isWritable(stream) {     !!stream._writableState; } +function isWritableFinished(stream) {+  if (stream.writableFinished) return true;+  const wState = stream._writableState;+  if (wState && wState.finished) return true;+  if (wState && wState.ended && wState.length === 0 &&+    !wState.errored) return true;+  return false;

Nit:

if (!wState) return false;
return wState.finished ||
  wState.ended && wState.length === 0 && !wState.errored;
ronag

comment created time in 21 days

pull request commentnodejs/node

test: improve flaky http2-stream-destroy-event-order

:laughing: got hit by 3 other flaky tests parallel/test-stream-pipeline-http2, async-hooks/test-statwatcher, test-inspector-connect-main-thread. Unfortunately, none of them failed for me on Windows at even extreme -j and --repeat numbers =(. Resuming the CI.

lundibundi

comment created time in 21 days

push eventlundibundi/node

Denys Otrishko

commit sha 6343f0cfce820a2317d08a1194ccb0f77225fe54

fixup! Update doc/api/modules.md Co-Authored-By: Rich Trott <rtrott@gmail.com>

view details

push time in 21 days

PR opened nodejs/node

test: improve flaky http2-stream-destroy-event-order

This doesn't properly fix the issue, it only significantly reduces (removes?) the flakiness of the test.

The setTimeout must not be needed here. Specifically if the settings frame on the client is 'delayed' and only comes after we have already called req.close(2) then the 'close' event will be emitted before the stream has got a chance to write the rst to the underlying socket (even though FlushRstStream and SendPendingData have been called) and since we are destroying a session here it will never be written resulting in a 'clean' stream close on the server side and therefore a timeout because there will be no error.


Looks like we are not correctly waiting for the rst frame to be sent and I assume the time it takes to read the settings frame delays the write just enough to not get written at all. I will try to investigate this further but at least this should improve our CI so that it won't fail with this test almost every time. This patch reduces the flakiness from 1 failed in 10 runs to 0/2000 on my windows machine.

The debug native logs can be found: http2-destroy-event-order-success.txt http2-destroy-event-order-fail.txt.

/cc @nodejs/testing @jasnell @addaleax <!-- Thank you for your pull request. Please provide a description above and review the requirements below.

Bug fixes and new features should include tests and possibly benchmarks.

Contributors guide: https://github.com/nodejs/node/blob/master/CONTRIBUTING.md -->

Checklist

<!-- Remove items that do not apply. For completed items, change [ ] to [x]. -->

  • [x] make -j4 test (UNIX), or vcbuild test (Windows) passes
  • [x] commit message follows commit guidelines

<!-- Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I have the right to submit it under the open source license indicated in the file; or

(b) The contribution is based upon previous work that, to the best of my knowledge, is covered under an appropriate open source license and I have the right under that license to submit that work with modifications, whether created in whole or in part by me, under the same open source license (unless I am permitted to submit under a different license), as indicated in the file; or

(c) The contribution was provided directly to me by some other person who certified (a), (b) or (c) and I have not modified it.

(d) I understand and agree that this project and the contribution are public and that a record of the contribution (including all personal information I submit with it, including my sign-off) is maintained indefinitely and may be redistributed consistent with this project or the open source license(s) involved. -->

+10 -1

0 comment

1 changed file

pr created time in 21 days

create barnchlundibundi/node

branch : plug-flaky-destroy-event-order

created branch time in 21 days

PR opened nodejs/node

benchmark: clean up config resolution in multiple benchmarks

This removes 'to Number' casting in multiple benchmarks (which is handled by the benchmark runner) and cleans up some var usage in changed benchmarks.

<!-- Thank you for your pull request. Please provide a description above and review the requirements below.

Bug fixes and new features should include tests and possibly benchmarks.

Contributors guide: https://github.com/nodejs/node/blob/master/CONTRIBUTING.md -->

Checklist

<!-- Remove items that do not apply. For completed items, change [ ] to [x]. -->

  • [x] make -j4 test (UNIX), or vcbuild test (Windows) passes
  • [x] commit message follows commit guidelines

<!-- Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I have the right to submit it under the open source license indicated in the file; or

(b) The contribution is based upon previous work that, to the best of my knowledge, is covered under an appropriate open source license and I have the right under that license to submit that work with modifications, whether created in whole or in part by me, under the same open source license (unless I am permitted to submit under a different license), as indicated in the file; or

(c) The contribution was provided directly to me by some other person who certified (a), (b) or (c) and I have not modified it.

(d) I understand and agree that this project and the contribution are public and that a record of the contribution (including all personal information I submit with it, including my sign-off) is maintained indefinitely and may be redistributed consistent with this project or the open source license(s) involved. -->

+13 -26

0 comment

7 changed files

pr created time in 22 days

create barnchlundibundi/node

branch : bench-clean-up

created branch time in 22 days

Pull request review commentnodejs/node

worker: add ability to take heap snapshot from parent thread

+'use strict';+const {+  Symbol+} = primordials;+const {+  kUpdateTimer,+  onStreamRead,+} = require('internal/stream_base_commons');+const { owner_symbol } = require('internal/async_hooks').symbols;+const { Readable } = require('stream');++const kHandle = Symbol('kHandle');++class HeapSnapshotStream extends Readable {+  constructor(handle) {+    super({ autoDestroy: true });

Yeah, if we consider backporting then it would be better to leave it as is. Just a nit anyways.

addaleax

comment created time in 22 days

issue closednodejs/node

NodeJs 12.11.1 NGHTTP2_ENHANCE_YOUR_CALM

  • Version: 12.11.1
  • Platform: Docker 12.11-alpine (on Ubuntu)

This is a regression (or a feature?) compared to 12.7

When performing a consequent amount of request on an http2 connection, using 12.11 I reach this error: Error [ERR_HTTP2_STREAM_ERROR]: Stream closed with error code NGHTTP2_ENHANCE_YOUR_CALM at ClientHttp2Stream._destroy (internal/http2/core.js:2098:13) at ClientHttp2Stream.destroy (internal/streams/destroy.js:37:8) at ClientHttp2Stream.[kMaybeDestroy] (internal/http2/core.js:2114:12) at Http2Stream.onStreamClose (internal/http2/core.js:491:26)

This doesn't happen on 12.7 under the same condition with the same code. This happens when the amount of concurrent stream is almost at the limit agreed between the 2 peers (128 in my case). This happens when the amount of request is high but the number of concurrent stream is lower than the limit (~10 concurrent stream, with a limit of 128)

I believe this was handled correctly in 12.7 by keeping the request in the queue, but in 12.11 it seems to push it over the limit and the peer returns the NGHTTP2_ENHANCE_YOUR_CALM error.

closed time in 22 days

crystalin

issue commentnodejs/node

NodeJs 12.11.1 NGHTTP2_ENHANCE_YOUR_CALM

The 12.7 doesn't include the commit that supposedly introduced the bug (https://github.com/nodejs/node/pull/29122 which was released in 12.8.1) while 12.11 has it, could you update to the latest v12 and check if the problem still persists? Closing for now as this is very similar to the other reported errors related to NGHTTP2_ENHANCE_YOUR_CALM, feel free to reopen.

Should be fixed in all release lines since 13.3.0, 12.14.1 and 10.18.1 via https://github.com/nodejs/node/pull/30684.

crystalin

comment created time in 22 days

issue closednodejs/node

Nodejs v13.0.1 HTTP2 NGHTTP2_ENHANCE_YOUR_CALM & NGHTTP2_INTERNAL_ERROR

  • Version: 13.0.1
  • Platform: Linux

I've tried to create a simple client-server app using http/2, that downloads directories. On September 1st I posted that nodejs crashes at some point. My previous bug report: https://github.com/nodejs/node/issues/29393

Now, nodejs doesn't crash. But I still get NGHTTP2_ENHANCE_YOUR_CALM & NGHTTP2_INTERNAL_ERROR at some point.

Here's my code to reproduce, hope it helps: https://github.com/Rantoledo/http2_nodejs_client_server_example2.git

BTW, I've tried to set MaxSessionMemory to 10, 100, 1000 and even 10000. I don't think it made any difference.

closed time in 22 days

Rantoledo

issue commentnodejs/node

Nodejs v13.0.1 HTTP2 NGHTTP2_ENHANCE_YOUR_CALM & NGHTTP2_INTERNAL_ERROR

Should be fixed in all release lines since 13.3.0, 12.14.1 and 10.18.1 via https://github.com/nodejs/node/pull/30684.

Feel free to reopen if the issue is still present while using 'fixed' Node.js version.

Rantoledo

comment created time in 22 days

Pull request review commentnodejs/node

worker: add ability to take heap snapshot from parent thread

 E('ERR_WASI_ALREADY_STARTED', 'WASI instance has already started', Error); E('ERR_WORKER_INVALID_EXEC_ARGV', (errors) =>   `Initiated Worker with invalid execArgv flags: ${errors.join(', ')}`,   Error);+E('ERR_WORKER_NOT_RUNNING', 'Worker instance not running', Error);

Nit:

E('ERR_WORKER_NOT_RUNNING', 'Worker instance is not running', Error);
addaleax

comment created time in 22 days

Pull request review commentnodejs/node

worker: add ability to take heap snapshot from parent thread

+'use strict';+const {+  Symbol+} = primordials;+const {+  kUpdateTimer,+  onStreamRead,+} = require('internal/stream_base_commons');+const { owner_symbol } = require('internal/async_hooks').symbols;+const { Readable } = require('stream');++const kHandle = Symbol('kHandle');++class HeapSnapshotStream extends Readable {+  constructor(handle) {+    super({ autoDestroy: true });

Nit: autoDestroy: true is the current default.

addaleax

comment created time in 23 days

Pull request review commentnodejs/node

benchmark: add MessagePort benchmark

+'use strict';++const common = require('../common.js');+const { MessageChannel } = require('worker_threads');+const bench = common.createBenchmark(main, {+  payload: ['string', 'object'],+  n: [1e6]+});++function main(conf) {++  const n = +conf.n;

So based on the code in benchmark/common.js and specifically _parseArgs it should detect that n is a number in the config and accordingly cast the parameter to a number, so this line shouldn't be needed.

addaleax

comment created time in 22 days

Pull request review commentnodejs/node

benchmark: add MessagePort benchmark

+'use strict';++const common = require('../common.js');+const { MessageChannel } = require('worker_threads');+const bench = common.createBenchmark(main, {

Oh, yes, sorry. Kind of forgot that our benchmark test are for a benchmark group in general.

addaleax

comment created time in 22 days

push eventlundibundi/node

Denys Otrishko

commit sha 74b9e3b9d39314515e38bcc7d0123f9af55965fd

fixup! src: fix console debug output on Windows

view details

push time in 22 days

issue closednodejs/node

investigate flaky test-http2-large-writes-session-memory-leak on Windows CI

https://ci.nodejs.org/job/node-test-binary-windows-js-suites/634/RUN_SUBSET=3,nodes=win2012r2-COMPILED_BY-vs2019-x86/console

test-rackspace-win2012r2_vs2019-x64-2

00:24:35 not ok 295 parallel/test-http2-large-writes-session-memory-leak
00:24:35   ---
00:24:35   duration_ms: 0.266
00:24:35   severity: fail
00:24:35   exitcode: 1
00:24:35   stack: |-
00:24:35     events.js:297
00:24:35           throw er; // Unhandled 'error' event
00:24:35           ^
00:24:35     
00:24:35     Error [ERR_HTTP2_STREAM_ERROR]: Stream closed with error code NGHTTP2_ENHANCE_YOUR_CALM
00:24:35         at ClientHttp2Stream._destroy (internal/http2/core.js:2117:13)
00:24:35         at ClientHttp2Stream.destroy (internal/streams/destroy.js:39:8)
00:24:35         at ClientHttp2Stream.[kMaybeDestroy] (internal/http2/core.js:2133:12)
00:24:35         at Http2Stream.onStreamClose (internal/http2/core.js:500:26)
00:24:35     Emitted 'error' event on ClientHttp2Stream instance at:
00:24:35         at emitErrorNT (internal/streams/destroy.js:94:8)
00:24:35         at emitErrorCloseNT (internal/streams/destroy.js:66:3)
00:24:35         at processTicksAndRejections (internal/process/task_queues.js:84:21) {
00:24:35       code: 'ERR_HTTP2_STREAM_ERROR'
00:24:35     }
00:24:35   ...

@nodejs/http2 @nodejs/platform-windows

closed time in 22 days

Trott

issue commentnodejs/node

investigate flaky test-http2-large-writes-session-memory-leak on Windows CI

Well, stress CI is green, perhaps the machine should be under heavy load (from other tests) for this to actually fail. Anyway, this doesn't seem to fail often (at all anymore?) so closing this for now.

Trott

comment created time in 22 days

PR opened nodejs/node

Reviewers
src: fix console debug output on Windows

The FWrite function on Windows assumed that MultiByteToWideChar automatically null-terminates the resulting string, but it will only do so if the size of the source was passed as -1 or null character was explicitly counted in the size. The FWrite uses std::string and its .size() method which doesn't count the null character even though the .data() method adds it to the resulting string.

https://docs.microsoft.com/en-us/windows/win32/api/stringapiset/nf-stringapiset-multibytetowidechar#remarks

MultiByteToWideChar does not null-terminate an output string if the input string length is explicitly specified without a terminating null character. To null-terminate an output string for this function, the application should pass in -1 or explicitly count the terminating null character for the input string.


Discovered while debugging http2 tests on windows, all native logs were in one line. Hopefully I didn't miss another thing in the MultiByteToWideChar doc, it's quite confusing.

/cc @addaleax <!-- Thank you for your pull request. Please provide a description above and review the requirements below.

Bug fixes and new features should include tests and possibly benchmarks.

Contributors guide: https://github.com/nodejs/node/blob/master/CONTRIBUTING.md -->

Checklist

<!-- Remove items that do not apply. For completed items, change [ ] to [x]. -->

  • [x] make -j4 test (UNIX), or vcbuild test (Windows) passes
  • [x] commit message follows commit guidelines

<!-- Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I have the right to submit it under the open source license indicated in the file; or

(b) The contribution is based upon previous work that, to the best of my knowledge, is covered under an appropriate open source license and I have the right under that license to submit that work with modifications, whether created in whole or in part by me, under the same open source license (unless I am permitted to submit under a different license), as indicated in the file; or

(c) The contribution was provided directly to me by some other person who certified (a), (b) or (c) and I have not modified it.

(d) I understand and agree that this project and the contribution are public and that a record of the contribution (including all personal information I submit with it, including my sign-off) is maintained indefinitely and may be redistributed consistent with this project or the open source license(s) involved. -->

+1 -3

0 comment

1 changed file

pr created time in 22 days

create barnchlundibundi/node

branch : fix-debug-windows

created branch time in 22 days

Pull request review commentnodejs/node

doc: clarify require() OS independence

 Used to import modules, `JSON`, and local files. Modules can be imported from `node_modules`. Local modules and JSON files can be imported using a relative path (e.g. `./`, `./foo`, `./bar/baz`, `../foo`) that will be resolved against the directory named by [`__dirname`][] (if defined) or-the current working directory.+the current working directory. The relative paths of POSIX style are resolved+in an OS independent fashion, meaning that the examples above will work on+Windows in the same way they would on Unix systems.  ```js-// Importing a local module:+// Importing a local module with a path relative to the `__dirname` or current+// working directory (on Windows this would resolve to .\\path\\myLocalModule):

Done.

lundibundi

comment created time in 22 days

push eventlundibundi/node

Denys Otrishko

commit sha d4f19c6aca51b6c85564626503d7dddfe7adf071

fixup! doc: clarify require() OS independence

view details

push time in 22 days

Pull request review commentnodejs/node

doc: clarify require() OS independence

 Used to import modules, `JSON`, and local files. Modules can be imported from `node_modules`. Local modules and JSON files can be imported using a relative path (e.g. `./`, `./foo`, `./bar/baz`, `../foo`) that will be resolved against the directory named by [`__dirname`][] (if defined) or-the current working directory.+the current working directory. The relative paths of POSIX style are resolved+in an OS independent fashion, meaning that the examples above will work on+Windows in the same way they would on Unix systems.  ```js-// Importing a local module:+// Importing a local module with a path relative to the `__dirname` or current+// working directory (on Windows this would resolve to .\\path\\myLocalModule):

That what I was debating on in my head I guess it makes more sense since it's a comment.

lundibundi

comment created time in 23 days

issue commentnodejs/node

investigate flaky test-http2-large-writes-session-memory-leak on Windows CI

Didn't fail on my local Windows vs2017 machine with -j 32 --repeat 1000, trying out CI https://ci.nodejs.org/job/node-stress-single-test/38/

Trott

comment created time in 23 days

Pull request review commentnodejs/node

benchmark: add MessagePort benchmark

+'use strict';++const common = require('../common.js');+const { MessageChannel } = require('worker_threads');+const bench = common.createBenchmark(main, {+  payload: ['string', 'object'],+  n: [1e6]+});++function main(conf) {++  const n = +conf.n;

I don't think we usually cast n to a number, why is this needed here?

addaleax

comment created time in 23 days

Pull request review commentnodejs/node

benchmark: add MessagePort benchmark

+'use strict';++const common = require('../common.js');+const { MessageChannel } = require('worker_threads');+const bench = common.createBenchmark(main, {

I think this is missing a test in test/benchmark?

addaleax

comment created time in 23 days

pull request commentnodejs/node

lib: improve validation utils and refactor vm argument validation

ping @nodejs/vm @BridgeAR (I guess there is no proper group for this one too).

lundibundi

comment created time in 23 days

PR opened nodejs/node

doc: clarify require() OS independence

<!-- Thank you for your pull request. Please provide a description above and review the requirements below.

Bug fixes and new features should include tests and possibly benchmarks.

Contributors guide: https://github.com/nodejs/node/blob/master/CONTRIBUTING.md -->

Checklist

<!-- Remove items that do not apply. For completed items, change [ ] to [x]. -->

  • [x] make -j4 test (UNIX), or vcbuild test (Windows) passes
  • [x] documentation is changed or added
  • [x] commit message follows commit guidelines

<!-- Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I have the right to submit it under the open source license indicated in the file; or

(b) The contribution is based upon previous work that, to the best of my knowledge, is covered under an appropriate open source license and I have the right under that license to submit that work with modifications, whether created in whole or in part by me, under the same open source license (unless I am permitted to submit under a different license), as indicated in the file; or

(c) The contribution was provided directly to me by some other person who certified (a), (b) or (c) and I have not modified it.

(d) I understand and agree that this project and the contribution are public and that a record of the contribution (including all personal information I submit with it, including my sign-off) is maintained indefinitely and may be redistributed consistent with this project or the open source license(s) involved. --> /cc @nodejs/documentation

+5 -2

0 comment

1 changed file

pr created time in 23 days

more