profile
viewpoint

nodejs/nan 2763

Native Abstractions for Node.js

trevnorris/cbuffer 171

JavaScript Circular Buffer

nodejs/benchmarking 151

Node.js Benchmarking Working Group

indutny/caine 142

Friendly butler

trevnorris/buffer-dispose 46

Incredibly unsafe way to free a Buffer

indutny/hash-cracker 20

V8 Hash Seed timing attack

nodesource/event-loop-demo 11

Example server demonstrating effect of processing load and response times

indutny/talks 8

Slides from talks

trevnorris/defactor 7

A Deferred Factory Object

dshaw/node-error-handling 5

Node Error Handing Mini Summit

pull request commentlibuv/libuv

Add API for measuring event loop idle time

ping @bnoordhuis

trevnorris

comment created time in 8 days

push eventtrevnorris/libuv

Ben Noordhuis

commit sha 7ccea60b1f576de2e65e8891b76165b8048aaac8

doc: fix formatting Fixes: https://github.com/libuv/libuv/issues/2831 PR-URL: https://github.com/libuv/libuv/pull/2833 Reviewed-By: Bartosz Sosnowski <bartosz@janeasystems.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com>

view details

Anna Henningsen

commit sha 55dede5b6f730882d79f994138f0f23a7b28b119

unix: fix memory leak when uv_loop_init() fails `uv_signal_init()` leads to the allocation of an IO watcher, so if the loop initialization fails at a later point, the `loop->watchers` list needs to be released. PR-URL: https://github.com/libuv/libuv/pull/2837 Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Santiago Gimeno <santiago.gimeno@gmail.com>

view details

Trevor Norris

commit sha 8145d85dd1366c2ff812f69a58ea99152583fe17

zos: fix potential event loop stall This is a port of 70002c80 to z/OS to fix the same potential issue that could effectively enter an infinite loop (but not a busy loop) under certain conditions when polling for events.

view details

Trevor Norris

commit sha 00e9954864afc91874e029c22c4ef5e23454c881

include: add internal fields struct to uv_loop_t Add struct `uv__loop_internal_fields_t` as a location for future additions to `uv_loop_t` while also maintaining v1.x compatibility. Currently `uv__loop_internal_fields_t` only contains the `flags` field. The reason for adding the `flags` field is because the same field was never added to `UV_LOOP_PRIVATE_FIELDS` in Windows as it was in Unix. The idea for creating a struct and attaching it to `uv_loop_t` for future API enhancements was taken from a comment by bnoordhuis in: https://github.com/libuv/libuv/issues/2506#issuecomment-540050665 Also add `internal_fields` to `uv_loop_t` to store the pointer to `uv__loop_internal_fields_t`. This naming makes more sense than just using `active_reqs.unused[1]`. To maintain ABI compatibility, shrink the `unused` array.

view details

Trevor Norris

commit sha f4e8732d963402d63738c8c83735170c31bd84bd

unix,win: add API to measure event loop idle time The API addition `uv_metrics_idle_time()` is a thread safe call that allows the user to retrieve the amount of time the event loop has spent in the kernel's event provider (i.e. poll). It was done this way to allow retrieving this value without needing to interrupt the execution of the event loop. This option can be enabled by passing `UV_METRICS_IDLE_TIME` to `uv_loop_configure()`. One important aspect of this change is, when enabled, to always first call the event provider with a `timeout == 0`. This allows libuv to know whether any events were waiting in the event queue when the event provider was called. The importance of this is because libuv is tracking the amount of "idle time", not "poll time". Thus the provider entry time is not recorded when `timeout == 0` (the event provider never idles in this case). While this does add a small amount of overhead, when enabled, but the overhead decreases when the event loop has a heavier load. This is because poll events will be waiting when the event provider is called. Thus never actually recording the provider entry time. Checking if `uv_loop_t` is configured with `UV_METRICS_IDLE_TIME` always happens in `uv__metrics_set_provider_entry_time()` and `uv__metrics_update_idle_time()`. Making the conditional logic wrapping each call simpler and allows for instrumentation to always hook into those two function calls. Rather than placing the fields directly on `uv__loop_internal_fields_t` add the struct `uv__loop_metrics_t` as a location for future metrics API additions. Tests and additional documentation has been included.

view details

push time in 16 days

issue closednodejs/node

JSON is not in the primordials list

  • Version: v12.16.3 v14.2.0
  • Platform: All
  • Subsystem: internals

The JSON object does not appear in the primordials list, but make jslint will display a no-restricted-globals error.

closed time in 20 days

trevnorris

issue commentnodejs/node

JSON is not in the primordials list

And that would explain why my search for git grep "\<JSON\>" -- lib/ didn't help. Well that solves this issue. Thanks.

On a side note; lib/internal/v8_prof_polyfill.js still uses JSON.stringify() in a template literal (possibly why lint doesn't catch it).

Also, might be nice to update lib/.eslintrc.yaml with better instructions on what to use. Instead of

Use const { JSON } = primordials; instead of the global.

trevnorris

comment created time in 20 days

issue openednodejs/node

JSON is not in the primordials list

  • Version: v12.16.3 v14.2.0
  • Platform: All
  • Subsystem: internals

The JSON object does not appear in the primordials list, but make jslint will display a no-restricted-globals error.

created time in 20 days

pull request commentlibuv/libuv

Add API for measuring event loop idle time

@bnoordhuis I'd like to get some more feedback from you on this PR and how it can be improved and accepted.

In https://github.com/nodejs/node/pull/33138#issuecomment-621714442 you mentioned using hooks around poll. In my PR I've intentionally had uv__metrics_set_provider_entry_time() run around poll, but also on the condition that timeout != 0. Reason for that is because whenever timeout == 0 then poll is never idle.

Likewise it will only run just before the first event is about to be processed. This is to get around the fact that polling may be interrupted, but the file descriptor will only be disarmed and poll is re-entered. As explained the the comment:

File descriptor that we've stopped watching, disarm it.

The placement and conditions here would be the most idle for any callback hooks you think would be worth adding.

trevnorris

comment created time in a month

push eventtrevnorris/libuv

erw7

commit sha 6b3df0577daa496361ac5ff00c63a749071e7a75

build,cmake: fix compilation on old MinGW _WIN32_WINNT specifies the minimum version of the operating system supported by the code, so change it to the minimum version supported by libuv. If _WIN32_WINNT=0x0600, PSAPI_VERSION=1 is set and it is necessary to link psapi, so it adds psapi unconditionally to the linked library. Fixes: https://github.com/libuv/libuv/issues/2742 Refs: https://docs.microsoft.com/en-us/cpp/porting/modifying-winver-and-win32-winnt?view=vs-2019 PR-URL: https://github.com/libuv/libuv/pull/2748 Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl> Reviewed-By: Jameson Nash <vtjnash@gmail.com>

view details

Bartosz Sosnowski

commit sha e1180a641730e1306f71b8e4919e62a01edd1fe9

build: remove unnessesary MSVC warnings Removes warnings W4100, W4127, W4201, W4206, W4210, W4232, W4456, W4457, W4459, W4706 and W4996 when building with MSVC. PR-URL: https://github.com/libuv/libuv/pull/2777 Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>

view details

Ben Noordhuis

commit sha aa7b62efd976eacf9c55150631a08ceec91a5cd1

win: make uv_udp_init_ex() accept UV_UDP_RECVMMSG Commit 5736658b ("udp: add flag to enable recvmmsg(2) explicitly") added the flag but didn't update the validation logic in src/win/udp.c. This commit moves the validation logic to src/uv-common.c. The flag is now accepted as a no-op on Windows. Fixes: https://github.com/libuv/libuv/issues/2806 PR-URL: https://github.com/libuv/libuv/pull/2809 Reviewed-By: Bartosz Sosnowski <bartosz@janeasystems.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Santiago Gimeno <santiago.gimeno@gmail.com>

view details

Ben Noordhuis

commit sha 4d3779c08c54b142680c9394f19346735b696c99

unix: simplify uv__udp_init_ex() PR-URL: https://github.com/libuv/libuv/pull/2809 Reviewed-By: Bartosz Sosnowski <bartosz@janeasystems.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Santiago Gimeno <santiago.gimeno@gmail.com>

view details

Bartosz Sosnowski

commit sha da7e50bbd843a9278c57fc1f90cb4cbbd5fba169

win: remove MAX_PATH limitations Since Windows 10 1607 some WinApi functions no longer have a MAX_PATH limit on the filenames length. This removes this hard-coded path length limit from various places in libuv, switching to dynamically allocating string buffers. Fixes: https://github.com/libuv/libuv/issues/2331 PR-URL: https://github.com/libuv/libuv/pull/2788 Reviewed-By: Colin Ihrig <cjihrig@gmail.com>

view details

Bartosz Sosnowski

commit sha be0e957f1802064cb7eca850f6a39900289b4da3

build, win: add long path aware manifest Adds manifest file that makes the test runner work with long filenames when those are enabled in the system. PR-URL: https://github.com/libuv/libuv/pull/2789 Reviewed-By: Colin Ihrig <cjihrig@gmail.com>

view details

Ben Noordhuis

commit sha 25368e2db1debb5ffa99041ac543e6d8539e461e

doc: check/idle/prepare functions always succeed Make the documentation reflect that the init/start/stop functions for check/idle/prepare handles always succeed. PR-URL: https://github.com/libuv/libuv/pull/2803 Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Santiago Gimeno <santiago.gimeno@gmail.com>

view details

Ben Noordhuis

commit sha 75c8850c91ffa93fded1c8e2a8e725e00124e4be

darwin: fix build with non-apple compilers The header files for ApplicationServices and CoreFoundation contain C language extensions that Apple's compiler understands but gcc does not, notably blocks: https://en.wikipedia.org/wiki/Blocks_(C_language_extension) Work around that by defining the types inline and stop including the headers. It's inelegant but the alternatives are worse. Fixes: https://github.com/libuv/libuv/issues/2805 PR-URL: https://github.com/libuv/libuv/pull/2811 Reviewed-By: Colin Ihrig <cjihrig@gmail.com>

view details

Ben Noordhuis

commit sha db96a61c26a88e18fd09a45d78e78add7c25a6f9

win: support environment variables > 32767 chars The Windows documentation states these should not be possible but several people have reported that they do in fact happen. Try with a smallish stack-allocated buffer first and switch to a heap-allocated buffer if the first one is too small. Fixes: https://github.com/libuv/libuv/issues/2587 PR-URL: https://github.com/libuv/libuv/pull/2589 Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Saúl Ibarra Corretgé <saghul@gmail.com>

view details

Ben Noordhuis

commit sha 43b456dcb375319ad0e45931ddddd88fd3dd7ee1

unix: fully initialize struct msghdr Commit 3d713663 ("freebsd,linux: add recvmmsg() + sendmmsg() udp implementation") forgot to zero some of the members of the msghdr struct. It seems to work by accident, not by design. PR-URL: https://github.com/libuv/libuv/pull/2819 Refs: https://github.com/libuv/libuv/pull/2818 Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Richard Lau <riclau@uk.ibm.com> Reviewed-By: Santiago Gimeno <santiago.gimeno@gmail.com>

view details

twosee

commit sha 06b2f13bad0db043a10952f2a16258c236ff451f

doc: add uv_replace_allocator thread safety warning PR-URL: https://github.com/libuv/libuv/pull/2815/ Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>

view details

Michal Artazov

commit sha 2bbf7d5c8cd070cc8541698fe72136328bc18eae

unix: fix int overflow when copying large files I was getting some weird results when copying a 5GB file on Alpine Linux on Raspberry Pi. Out of the 5GB only 1.1GB would get copied and the process would finish without an error. After some digging I found out that there's a problem that some data types have smaller number of bytes on Alpine Linux on Raspberry Pi than on other platforms apparently. When getting the size of the file in bytes, stat holds the size in off_t data type, like this: struct stat { ... off_t st_size; /* total size, in bytes */ ... }; In my case, off_t has 8 bytes which is enough to hold a value up to some exabytes. The problem is that it gets assigned to bytes_to_send variable, which is size_t. In my case, size_t is only 4 bytes, which is only good for about 4GB. If the file is any larger, there's an overflow when assigning it from stat to bytes_to_send. That's easy to fix, I just changed the data type of bytes_to_send to off_t. However there's more. The other 2 variables - in_offset and bytes_written also have to be able to hold the size of the entire file, therefore it makes sense to change them to off_t as well. The last problem is that bytes_to_send is passed down to uv_fs_sendfile() which converts it to size_t again. I could go and change the types everywhere across the whole codebase to off_t but that could break other things, so it seams a bit too much. A much better solution is to have a proxy variable bytes_chunk that is size_t type and copy as much bytes as possible at a time that can fit into size_t. That way it will work the same as before on other platforms, where size_t is more than 4 bytes. PR-URL: https://github.com/libuv/libuv/pull/2758 Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>

view details

Bartosz Sosnowski

commit sha 457285827cfcc6bbe33abe9bd211e7cee332b3c5

fs: report original error Exposes the original system error of the filesystem syscalls. Adds a new uv_fs_get_system_error which returns orignal errno on Linux or GetLastError on Windows. Ref: https://github.com/libuv/libuv/issues/2348 PR-URL: https://github.com/libuv/libuv/pull/2810 Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl> Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Santiago Gimeno <santiago.gimeno@gmail.com>

view details

Bartosz Sosnowski

commit sha e7ebae26247d2fee0a04547eb7f9aa8f78d4a642

win, fs: add IO_REPARSE_TAG_APPEXECLINK support Adds support for IO_REPARSE_TAG_APPEXECLINK reparse points, used by Windows Store. Ref: https://github.com/nodejs/node/issues/33024 PR-URL: https://github.com/libuv/libuv/pull/2812 Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl> Reviewed-By: Richard Lau <riclau@uk.ibm.com>

view details

Trevor Norris

commit sha be491b3f4562c5066dfc917022ac96f1052bb10d

zos: fix potential event loop stall This is a port of 70002c80 to z/OS to fix the same potential issue that could effectively enter an infinite loop (but not a busy loop) under certain conditions when polling for events.

view details

Trevor Norris

commit sha 378a47ae69d451df5c2d0d41341b0ea2c105a39c

include: add internal fields struct to uv_loop_t Add struct `uv__loop_internal_fields_t` as a location for future additions to `uv_loop_t` while also maintaining v1.x compatibility. Currently `uv__loop_internal_fields_t` only contains the `flags` field. The reason for adding the `flags` field is because the same field was never added to `UV_LOOP_PRIVATE_FIELDS` in Windows as it was in Unix. The idea for creating a struct and attaching it to `uv_loop_t` for future API enhancements was taken from a comment by bnoordhuis in: https://github.com/libuv/libuv/issues/2506#issuecomment-540050665 Also add `internal_fields` to `uv_loop_t` to store the pointer to `uv__loop_internal_fields_t`. This naming makes more sense than just using `active_reqs.unused[1]`. To maintain ABI compatibility, shrink the `unused` array.

view details

Trevor Norris

commit sha 1e0496fbd038f66641f2579d13406916ec1b67ae

unix,win: add API to measure event loop idle time The API addition `uv_metrics_idle_time()` is a thread safe call that allows the user to retrieve the amount of time the event loop has spent in the kernel's event provider (i.e. poll). It was done this way to allow retrieving this value without needing to interrupt the execution of the event loop. This option can be enabled by passing `UV_METRICS_IDLE_TIME` to `uv_loop_configure()`. One important aspect of this change is, when enabled, to always first call the event provider with a `timeout == 0`. This allows libuv to know whether any events were waiting in the event queue when the event provider was called. The importance of this is because libuv is tracking the amount of "idle time", not "poll time". Thus the provider entry time is not recorded when `timeout == 0` (the event provider never idles in this case). While this does add a small amount of overhead, when enabled, but the overhead decreases when the event loop has a heavier load. This is because poll events will be waiting when the event provider is called. Thus never actually recording the provider entry time. Checking if `uv_loop_t` is configured with `UV_METRICS_IDLE_TIME` always happens in `uv__metrics_set_provider_entry_time()` and `uv__metrics_update_idle_time()`. Making the conditional logic wrapping each call simpler and allows for instrumentation to always hook into those two function calls. Rather than placing the fields directly on `uv__loop_internal_fields_t` add the struct `uv__loop_metrics_t` as a location for future metrics API additions. Tests and additional documentation has been included.

view details

push time in a month

issue commentnodejs/node

[discuss] event loop idle metrics

@addaleax

That means that the Atomics.wait() call effectively fulfills the function of the event loop wait, and should count as idle time.

The metric should only measure idle time in the event provider. Specifically the amount of time idle while it waits for an event to be received (reason for this is explained in https://github.com/libuv/libuv/pull/2725#issuecomment-594752268). Looking at AtomicsWaitEvent, it seems like Atomic.wait() could be considered a type of event provider. Though if it's working in conjunction with uv_run() it'd probably be better to not include it.

Well, I guess that actually depends on whether the Worker's event loop has any handles from which it expects to receive an event. If not then Atomic.wait() would be considered the event provider. And since signal handlers can't be added from a Worker thread, there wouldn't be another way to interrupt the event provider. So I can see what you're getting at.

That being said, measuring the amount of time spent in Atomic.wait() would be a great step towards calculating total CPU idle time per thread. Since there isn't a platform agnostic way to do getrusage(RUSAGE_THREAD, ..., being able to include all other types of idle time is immensely helpful for analysis.

So for usability maybe just provide both times as separate values? IMO that would be the most helpful.

jasnell

comment created time in a month

issue commentnodejs/node

[discuss] event loop idle metrics

I assume active above is actually the wall clock time in the same unit as idle time.

Yes. That is the case.

jasnell

comment created time in a month

issue commentnodejs/node

[discuss] event loop idle metrics

If it's similar to my example above, to prevent yet another one-liner module we might just as well add it. So maybe it would operate something like so:

// This simply logs out the loop utilization for a three second duration
setTimeout((last) => {
  console.log(performance.loopUtilization(last));
}, 3000, performance.idleTime());
jasnell

comment created time in a month

issue commentnodejs/node

[discuss] event loop idle metrics

@Flarna Instead of reporting "utilization", if idleTime() returned { idle: n, active: m }, where active is the cumulative time spent in uv_run(), then it would be easy enough for users to implement their own.

A simple example:

function loopUtilization(last, current = performance.idleTime()) {
  return (current.idle - last.idle) / (current.active - last.active);
}

Though if active is included then maybe it shouldn't be called idleTime() anymore. :P

jasnell

comment created time in a month

issue commentnodejs/node

[discuss] event loop idle metrics

@Flarna It's a bit in depth, but you should read my post on https://github.com/libuv/libuv/pull/2725#issuecomment-594752268 about the differences.

TL;DR "idle time" is measured using uv_hrtime(), which I believe defaults to monotonic, and represents the amount of time the event loop was idle, not the amount of time the event loop was spent in the event provider (e.g. poll). "event loop utilization" is "idle time" divided by the amount of time the event loop has been running (also measured using uv_hrtime()).

jasnell

comment created time in a month

pull request commentlibuv/libuv

Add API for measuring event loop idle time

@jasnell Since uv_loop_configure() doesn't provide a way to disable a configuration I didn't bother to handle a situation where it would be disabled after being enabled. So unless libuv adds something like uv_loop_unconfigure() it will probably only be allowed to stay on once enabled.

trevnorris

comment created time in a month

pull request commentlibuv/libuv

Add API for measuring event loop idle time

Ping @libuv/collaborators. Any further thoughts on this?

trevnorris

comment created time in a month

push eventtrevnorris/libuv

James Ross

commit sha 37a8affbd319209fd0d0d96ee4e3002c8ecc5076

linux: make cpu_times consistently be milliseconds On Linux, cpu_times have been calculated differently to all other platforms for a while. Other platforms (FreeBSD, Mac, Windows) are all in milliseconds, but Linux has been returning values ten times larger. libuv has not previously documented what unit cpu_times uses, even though NodeJS did - as milliseconds. Here we're both documenting that the cpu_times are indeed in milliseconds, and fixing the inconsistency on Linux. Fixes: https://github.com/libuv/libuv/issues/2773 PR-URL: https://github.com/libuv/libuv/pull/2796 Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>

view details

Ben Noordhuis

commit sha 887cf62c32c3ec2a2f2fc4787a4c17c77a0c6295

win: DRY uv_poll_start() and uv_poll_stop() PR-URL: https://github.com/libuv/libuv/pull/2746 Reviewed-By: Bartosz Sosnowski <bartosz@janeasystems.com> Reviewed-By: erw7 <erw7.github@gmail.com>

view details

Ben Noordhuis

commit sha b29612fe59f664d9b370dceda1060b1dc1deaff0

win: DRY uv_poll_close() PR-URL: https://github.com/libuv/libuv/pull/2746 Reviewed-By: Bartosz Sosnowski <bartosz@janeasystems.com> Reviewed-By: erw7 <erw7.github@gmail.com>

view details

Ben Noordhuis

commit sha 72fe3543feb23ae555e08628b70a3fae4da5706c

unix,win: add uv_library_shutdown() Make it possible to explicitly tell libuv to release any resources it's still holding onto (memory, threads, file descriptors, etc.) Before this commit, cleanup was performed in various destructors. This commit centralizes the cleanup logic, enabling the addition of `uv_library_shutdown()`, but maintains the current observable behavior of cleaning up when libuv is unloaded by means of `dlclose(3)`. Fixes: https://github.com/libuv/libuv/issues/2763 PR-URL: https://github.com/libuv/libuv/pull/2764 Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Richard Lau <riclau@uk.ibm.com> Reviewed-By: Saúl Ibarra Corretgé <saghul@gmail.com>

view details

Ben Noordhuis

commit sha a9d9d4ea1b628368170fea18b042483f850e48a1

unix: yield cpu when spinlocking on async handle Fixes: https://github.com/libuv/libuv/issues/2769 PR-URL: https://github.com/libuv/libuv/pull/2772 Reviewed-By: Anna Henningsen <anna@addaleax.net>

view details

Colin Finck

commit sha aa93eb25839d662dc2503c164bb93756c238f660

win: remove dep on GetQueuedCompletionStatusEx Libuv already works without that API since commit 153ea114ff but still had it as a hard requirement in the import table. This code uses the `pGetQueuedCompletionStatusEx` function pointer instead, hence it also works on systems that don't export `GetQueuedCompletionStatusEx`. This simple fix improves compatibility of libuv with ReactOS and Windows XP (latter using Vista+ compatibility libraries like https://github.com/MyTDT-Mysoft/DllCompat) PR-URL: https://github.com/libuv/libuv/pull/2800 Reviewed-By: Bartosz Sosnowski <bartosz@janeasystems.com> Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>

view details

Shohei YOSHIDA

commit sha 99f8ffcc4e66fe089fbeba372dadd5940632c060

doc: correct source lines PR-URL: https://github.com/libuv/libuv/pull/2802 Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>

view details

twosee

commit sha 13886c52930ad8b14d594ad4ac9c90da7e084cb5

build,android: fix typo PR-URL: https://github.com/libuv/libuv/pull/2798 Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl> Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Richard Lau <riclau@uk.ibm.com>

view details

Philip Chimento

commit sha a87bb3c5f609504f6e6741d40f24ed2a67af24db

doc: uv_cancel() handles uv_random_t requests This seems to be the case from reading the code of uv_cancel(). Also fixes a broken link due to a markup typo. PR-URL: https://github.com/libuv/libuv/pull/2797 Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl> Reviewed-By: Colin Ihrig <cjihrig@gmail.com>

view details

Philip Chimento

commit sha 002fe6e5f2d084fc12fee3d6e5a3fdffe491b7e8

doc: fix unescaped character This was causing a warning during the documentation build. PR-URL: https://github.com/libuv/libuv/pull/2797 Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl> Reviewed-By: Colin Ihrig <cjihrig@gmail.com>

view details

Trevor Norris

commit sha 07dad65b824629fb1e0971ca8223deca8ec9f5e2

zos: fix potential event loop stall This is a port of 70002c80 to z/OS to fix the same potential issue that could effectively enter an infinite loop (but not a busy loop) under certain conditions when polling for events.

view details

Trevor Norris

commit sha 29c91dc4dd68f511b133c5e6cfa9139550034245

include: add internal fields struct to uv_loop_t Add struct `uv__loop_internal_fields_t` as a location for future additions to `uv_loop_t` while also maintaining v1.x compatibility. Currently `uv__loop_internal_fields_t` only contains the `flags` field. The reason for adding the `flags` field is because the same field was never added to `UV_LOOP_PRIVATE_FIELDS` in Windows as it was in Unix. The idea for creating a struct and attaching it to `uv_loop_t` for future API enhancements was taken from a comment by bnoordhuis in: https://github.com/libuv/libuv/issues/2506#issuecomment-540050665 Also add `internal_fields` to `uv_loop_t` to store the pointer to `uv__loop_internal_fields_t`. This naming makes more sense than just using `active_reqs.unused[1]`. To maintain ABI compatibility, shrink the `unused` array.

view details

Trevor Norris

commit sha 04fe4183c8ffcc73718ba17e62650669a7317762

unix,win: add API to measure event loop idle time The API addition `uv_metrics_idle_time()` is a thread safe call that allows the user to retrieve the amount of time the event loop has spent in the kernel's event provider (i.e. poll). It was done this way to allow retrieving this value without needing to interrupt the execution of the event loop. This option can be enabled by passing `UV_METRICS_IDLE_TIME` to `uv_loop_configure()`. One important aspect of this change is, when enabled, to always first call the event provider with a `timeout == 0`. This allows libuv to know whether any events were waiting in the event queue when the event provider was called. The importance of this is because libuv is tracking the amount of "idle time", not "poll time". Thus the provider entry time is not recorded when `timeout == 0` (the event provider never idles in this case). While this does add a small amount of overhead, when enabled, but the overhead decreases when the event loop has a heavier load. This is because poll events will be waiting when the event provider is called. Thus never actually recording the provider entry time. Checking if `uv_loop_t` is configured with `UV_METRICS_IDLE_TIME` always happens in `uv__metrics_set_provider_entry_time()` and `uv__metrics_update_idle_time()`. Making the conditional logic wrapping each call simpler and allows for instrumentation to always hook into those two function calls. Rather than placing the fields directly on `uv__loop_internal_fields_t` add the struct `uv__loop_metrics_t` as a location for future metrics API additions. Tests and additional documentation has been included.

view details

push time in a month

push eventtrevnorris/cbuffer

Trevor Norris

commit sha 5ea1ab9c1f2c3cf24d200a5cf555236b67338644

Fix couple test failures and crappy whitespace

view details

push time in a month

created tagtrevnorris/cbuffer

tag2.1.0

JavaScript Circular Buffer

created time in a month

push eventtrevnorris/cbuffer

Trevor Norris

commit sha 0c0c0de3b9c88744e537bece8a4e3723fe0a190f

Release v2.1.0

view details

push time in a month

push eventjstat/jstat.github.com

Trevor Norris

commit sha 62a0de4a5fabddf72d65cf3b77ea2eaeb161bca3

Update to v1.9.3

view details

push time in a month

created tagjstat/jstat

tag1.9.3

JavaScript Statistical Library

created time in a month

delete tag jstat/jstat

delete tag : 1.9.3

delete time in a month

push eventjstat/jstat

Trevor Norris

commit sha 718a06c5467fa1b1e8c24097b9783acfec5f760f

Release v1.9.3 * Have gammafn() handle large numbers correctly * Improve performance of rank()

view details

push time in a month

created tagjstat/jstat

tag1.9.3

JavaScript Statistical Library

created time in a month

push eventjstat/jstat

Trevor Norris

commit sha 35aef22d3df71e3a148bed3e391f03b0a8057182

Release v1.9.3 * Have gammafn() handle large numbers correctly * Improve performance of rank()

view details

push time in a month

PR closed jstat/jstat

Array ranking efficiency

Make array ranking more efficient for longer arrays with more distinct numbers.

For a longer array with a larger variety of numbers, indexOf and lastIndexOf take longer and are called more frequently. Eliminate the need to call indexOf and lastIndexOf by using the frequency of each number in the array.

Sample times taken Array of length 100, with random numbers between 1 and 100

  • previous rank: 0.072021484375ms
  • new rank: 0.054931640625ms

Array of length 100,000, ascending distinct numbers from 0 through 99,999

  • previous rank: 14851.73291015625ms
  • new rank: 33.7861328125ms

Array of length 100,000, with random numbers between 1 and 100

  • previous rank: 14704.958984375ms
  • new rank: 37.547119140625ms

Note that previous rank is slightly faster when all numbers in the array are the same: Array of length 1 million, all 1s

  • previous rank: 53.232177734375ms
  • new rank: 162.830078125ms
+26 -12

5 comments

1 changed file

suzukia

pr closed time in a month

pull request commentjstat/jstat

Array ranking efficiency

@suzukia Thanks for the patch. It landed in f0dfb003f9705d38cabd72454a8e518a5f144873.

suzukia

comment created time in a month

push eventjstat/jstat

Ami Suzuki

commit sha f0dfb003f9705d38cabd72454a8e518a5f144873

Make array ranking more efficient PR-URL: https://github.com/jstat/jstat/pull/255 Reviewed-by: Trevor Norris <trev.norris@gmail.com>

view details

push time in a month

issue commentjstat/jstat

Gamma Function has nonsensical behavior with large arguments

Seems I made the change but never pushed it. The fix landed in 25d3cf40f.

mscotch1

comment created time in a month

push eventjstat/jstat

Trevor Norris

commit sha 25d3cf40f30fcdebfb13e4c4f712142684f84239

Have gammafn() correctly handle large numbers Fixes: https://github.com/jstat/jstat/issues/250

view details

push time in a month

push eventtrevnorris/libuv

Sk Sajidul Kadir

commit sha bd4292385f5b0a1be85d6c7dd7b51d4b22a4289c

fs: add uv_fs_lutime() PR-URL: https://github.com/libuv/libuv/pull/2723 Reviewed-By: Bartosz Sosnowski <bartosz@janeasystems.com> Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl> Reviewed-By: Colin Ihrig <cjihrig@gmail.com>

view details

David Carlier

commit sha a2689ab2500ceb44ead14de554303dd209d42898

unix: implement cpu_relax() for arm Emits yield instruction in ARM mode (mainly needs to pass -march=...). PR-URL: https://github.com/libuv/libuv/pull/2751 Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl> Reviewed-By: Richard Lau <riclau@uk.ibm.com>

view details

twosee

commit sha fbafdbf71ab27ee3c55ffeba1540062cb6d47729

linux: fix uv__accept4() Refs: https://github.com/libuv/libuv/pull/2665 PR-URL: https://github.com/libuv/libuv/pull/2786 Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl> Reviewed-By: Santiago Gimeno <santiago.gimeno@gmail.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com>

view details

erw7

commit sha ad618647b996f1dc73e15632e81925b03bd19190

win: handle file paths in uv_fs_statfs() GetDiskFreeSpaceW() doesn't handle regular file paths well. This commit switches to using GetFullPathNameW(). Fixes: https://github.com/libuv/libuv/pull/2683 Refs: https://github.com/libuv/libuv/pull/2691 PR-URL: https://github.com/libuv/libuv/pull/2695 Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Jameson Nash <vtjnash@gmail.com>

view details

Rikard Falkeborn

commit sha b44d5ee1b418420243793289c17308c658ce1458

unix: fix uv_os_environ() null pointer check Check the pointer to the allocated memory, not the pointer to the pointer of the allocated memory. Previously, a failed allocation of *envitems would lead to a NULL pointer dereference. PR-URL: https://github.com/libuv/libuv/pull/2778 Reviewed-By: Richard Lau <riclau@uk.ibm.com> Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Bartosz Sosnowski <bartosz@janeasystems.com>

view details

Rikard Falkeborn

commit sha cd11c2b1ea2e5103a779040d7ffa40f6a3008e6b

win: fix uv_os_environ() null pointer check Check the pointer to the allocated memory, not the pointer to the pointer of the allocated memory. Previously, a failed allocation of *envitems would lead to a NULL pointer dereference. PR-URL: https://github.com/libuv/libuv/pull/2778 Reviewed-By: Richard Lau <riclau@uk.ibm.com> Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Bartosz Sosnowski <bartosz@janeasystems.com>

view details

Brad King

commit sha 815b0766eddf1ef6ebc9481c432b59d0477eb74d

unix: fix compilation on macOS 32-bit architectures In commit 2475296c (build: make code compilable for iOS on Xcode, 2020-01-18, v1.35.0~47) we added a `defined(TARGET_OS_IPHONE)` preprocessor condition, but `TARGET_OS_IPHONE` is always defined on Apple to either 0 or 1. On 32-bit macOS architectures this leads to an undefined symbol reference to `_close$NOCANCEL`. Fix the preprocessor condition to use just `TARGET_OS_IPHONE`. Refs: https://github.com/libuv/libuv/pull/2639 PR-URL: https://github.com/libuv/libuv/pull/2776 Reviewed-By: Colin Ihrig <cjihrig@gmail.com>

view details

Ben Noordhuis

commit sha 3e5d26144085bd709a0de24dd744bee8142bae5f

win: replace alloca() with stack-based array `required_vars_value_len` has a fixed number of elements. There is no need to use alloca() to allocate it. Refs: https://github.com/libuv/libuv/pull/2781 PR-URL: https://github.com/libuv/libuv/pull/2783 Reviewed-By: Santiago Gimeno <santiago.gimeno@gmail.com> Reviewed-By: Richard Lau <riclau@uk.ibm.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com>

view details

cjihrig

commit sha 533b738838ad8407032e14b6772b29ef9af63cfa

2020.04.16, Version 1.36.0 (Stable) Changes since version 1.35.0: * build: add aix-common.c for AIX cmake build (Jesse Gorzinski) * zos: explicitly mark message queue events (Irek Fakhrutdinov) * zos: move mq check out of loop to save cpu cycles (Irek Fakhrutdinov) * zos: add checks to ensure behavior of epoll_wait (Irek Fakhrutdinov) * src: add uv__reallocf() (Ben Noordhuis) * build: ibmi support for cmake (Jesse Gorzinski) * build: fix gyp build for Android API >= 28 (Lin Zhang) * udp: return recvmmsg-ed datagrams in order (Saúl Ibarra Corretgé) * zos,test: fix spawn_empty_env for shared library build (Richard Lau) * zos: fix non-Release builds (Richard Lau) * zos: fix return value on expired nanosleep() call (Richard Lau) * build: fix z/OS cmake build (Richard Lau) * test: add a bunch of ASSERT macros (Santiago Gimeno) * test: remove unused extern declaration (Ben Noordhuis) * test: canonicalize argv[0] in exepath test (Ben Noordhuis) * test: simplify platform_init() (Ben Noordhuis) * ibmi: Fix isatty EBADF handling and refactor (Kevin Adler) * test: Test EBADF tty handling (Kevin Adler) * build: make cmake build benchmarks (Ben Noordhuis) * win: use RtlGenRandom from advapi32.dll directly (Ben Noordhuis) * android: fix OOB write in uv_interface_addresses() (Lin Zhang) * test: pass test when hostname is single character (毛毛) * ibmi: set the highest process priority to -10 (Xu Meng) * build: remove support for gyp (Ben Noordhuis) * doc: add note to README on cross-compiling (Ben Noordhuis) * fs: add uv_fs_lutime() (Sk Sajidul Kadir) * unix: implement cpu_relax() for arm (David Carlier) * linux: fix uv__accept4() (twosee) * win: handle file paths in uv_fs_statfs() (erw7) * unix: fix uv_os_environ() null pointer check (Rikard Falkeborn) * win: fix uv_os_environ() null pointer check (Rikard Falkeborn) * unix: fix compilation on macOS 32-bit architectures (Brad King) * win: replace alloca() with stack-based array (Ben Noordhuis)

view details

cjihrig

commit sha 40e8804287b9f97322f5faeea76fbb06aeb0229d

Add SHA to ChangeLog

view details

cjihrig

commit sha 80c18ffaf555716119210d5e66dfa8a58f2f846d

Now working on version 1.36.1 Fixes: https://github.com/libuv/libuv/issues/2787

view details

Yash Ladha

commit sha 0fd993195f7fea121d8d9d75c2d9d802435b3110

timer: remove redundant check in heap compare `timer_less_than()` function is basically a comparator function that returns true or false. In the end of the function we were checking for the comparison of id, but the later if is redundant as we are anyways in the end are returning `0`. That extra check can thus be safely removed. PR-URL: https://github.com/libuv/libuv/pull/2785 Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl> Reviewed-By: Colin Ihrig <cjihrig@gmail.com>

view details

Saúl Ibarra Corretgé

commit sha 5736658bee0eede05caa55d66b9ad337e3b77266

udp: add flag to enable recvmmsg(2) explicitly Instead of implicitly enabling it by checking the supplied buffer size to alloc_cb, have a dedicated flag that must be set on `uv_udp_init_ex`. Fixes: https://github.com/libuv/libuv/issues/2791 Closes: https://github.com/libuv/libuv/pull/2792 PR-URL: https://github.com/libuv/libuv/pull/2799 Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl> Reviewed-By: Colin Ihrig <cjihrig@gmail.com>

view details

cjihrig

commit sha 02a9e1be252b623ee032a3137c0b0c94afbe6809

2020.04.20, Version 1.37.0 (Stable) Changes since version 1.36.0: * timer: remove redundant check in heap compare (Yash Ladha) * udp: add flag to enable recvmmsg(2) explicitly (Saúl Ibarra Corretgé)

view details

cjihrig

commit sha 87af8359056d3d15fc57197f2baf5733f80ef204

Add SHA to ChangeLog

view details

cjihrig

commit sha 0e00e14003b2c0f36eb3d3a90767355717309df9

Now working on version 1.37.1 Fixes: https://github.com/libuv/libuv/issues/2801

view details

Xu Meng

commit sha f52786b4badf82e1bfad06b52ade0397b61cd14e

test: skip poll_duplex and poll_unidirectional on PASE Issuing a shutdown() on IBM i PASE with parameter SHUT_WR also sends a normal close sequence to the partner program. This leads to timing issues and ECONNRESET failures in the test poll_duplex and poll_unidirectional. https://www.ibm.com/support/knowledgecenter/en/ssw_ibm_i_74/apis/shutdn.htm PR-URL: https://github.com/libuv/libuv/pull/2782 Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Richard Lau <riclau@uk.ibm.com>

view details

Trevor Norris

commit sha f8c79a0bfb22ac7962fb71fa988517d73aa84a42

zos: fix potential event loop stall This is a port of 70002c80 to z/OS to fix the same potential issue that could effectively enter an infinite loop (but not a busy loop) under certain conditions when polling for events.

view details

Trevor Norris

commit sha bac2b7dfcaba8be6e96a1fcc3bd48a19003d7114

include: add internal fields struct to uv_loop_t Add struct `uv__loop_internal_fields_t` as a location for future additions to `uv_loop_t` while also maintaining v1.x compatibility. Currently `uv__loop_internal_fields_t` only contains the `flags` field. The reason for adding the `flags` field is because the same field was never added to `UV_LOOP_PRIVATE_FIELDS` in Windows as it was in Unix. The idea for creating a struct and attaching it to `uv_loop_t` for future API enhancements was taken from a comment by bnoordhuis in: https://github.com/libuv/libuv/issues/2506#issuecomment-540050665 Also add `internal_fields` to `uv_loop_t` to store the pointer to `uv__loop_internal_fields_t`. This naming makes more sense than just using `active_reqs.unused[1]`. To maintain ABI compatibility, shrink the `unused` array.

view details

Trevor Norris

commit sha 9a2a10931e64e36706bb15f1e6673ab3aad9c98b

unix,win: add API to measure event loop idle time The API addition `uv_metrics_idle_time()` is a thread safe call that allows the user to retrieve the amount of time the event loop has spent in the kernel's event provider (i.e. poll). It was done this way to allow retrieving this value without needing to interrupt the execution of the event loop. This option can be enabled by passing `UV_METRICS_IDLE_TIME` to `uv_loop_configure()`. One important aspect of this change is, when enabled, to always first call the event provider with a `timeout == 0`. This allows libuv to know whether any events were waiting in the event queue when the event provider was called. The importance of this is because libuv is tracking the amount of "idle time", not "poll time". Thus the provider entry time is not recorded when `timeout == 0` (the event provider never idles in this case). While this does add a small amount of overhead, when enabled, but the overhead decreases when the event loop has a heavier load. This is because poll events will be waiting when the event provider is called. Thus never actually recording the provider entry time. Checking if `uv_loop_t` is configured with `UV_METRICS_IDLE_TIME` always happens in `uv__metrics_set_provider_entry_time()` and `uv__metrics_update_idle_time()`. Making the conditional logic wrapping each call simpler and allows for instrumentation to always hook into those two function calls. Rather than placing the fields directly on `uv__loop_internal_fields_t` add the struct `uv__loop_metrics_t` as a location for future metrics API additions. Tests and additional documentation has been included.

view details

push time in a month

pull request commentlibuv/libuv

Add API for measuring event loop idle time

@jasnell I do believe this information should be exposed in Node. I figured performance.idleTime() would be an appropriate API, and the implementation is simple if Node just wants to return the same value as uv_metrics_idle_time(). In addition, one reason I made this API thread safe is for easier Worker thread support. If this lands, I'll go into greater detail in a Node PR.

trevnorris

comment created time in a month

push eventtrevnorris/libuv

Ben Noordhuis

commit sha 53f3c687fc288708721a5a3d9563febda1b9d2c1

build: remove support for gyp Upstream GYP has been unmaintained for some time now. The only big downstream user is Node.js and they already maintain GYP build files for their other dependencies. This commit removes vcbuild.bat because: 1. it's entirely built around GYP, and 2. duplicates a lot of functionality that cmake provides out of the box This commit also removes appveyor.yml. It hasn't been used after we stopped providing prebuilt binaries for Windows about two years ago. As if that isn't enough, this commit also removes the Android build scripts. They were wrappers around GYP. Use CMake. PR-URL: https://github.com/libuv/libuv/pull/2682 Refs: https://github.com/libuv/libuv/pull/2681 Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Fedor Indutny <fedor.indutny@gmail.com> Reviewed-By: Richard Lau <riclau@uk.ibm.com> Reviewed-By: Santiago Gimeno <santiago.gimeno@gmail.com> Reviewed-By: Saúl Ibarra Corretgé <saghul@gmail.com>

view details

Ben Noordhuis

commit sha dc7c874660526e4ed70c7c7579b974283c9ad6e0

doc: add note to README on cross-compiling PR-URL: https://github.com/libuv/libuv/pull/2682 Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Fedor Indutny <fedor.indutny@gmail.com> Reviewed-By: Richard Lau <riclau@uk.ibm.com> Reviewed-By: Santiago Gimeno <santiago.gimeno@gmail.com> Reviewed-By: Saúl Ibarra Corretgé <saghul@gmail.com>

view details

Trevor Norris

commit sha ea23685fd20bd0df2df67c068d22ba3122382fdc

zos: fix potential event loop stall This is a port of 70002c80 to z/OS to fix the same potential issue that could effectively enter an infinite loop (but not a busy loop) under certain conditions when polling for events.

view details

Trevor Norris

commit sha e85e347c8ef9f79526e8d7d01b3068d3b2969c83

include: add internal fields struct to uv_loop_t Add struct `uv__loop_internal_fields_t` as a location for future additions to `uv_loop_t` while also maintaining v1.x compatibility. Currently `uv__loop_internal_fields_t` only contains the `flags` field. The reason for adding the `flags` field is because the same field was never added to `UV_LOOP_PRIVATE_FIELDS` in Windows as it was in Unix. The idea for creating a struct and attaching it to `uv_loop_t` for future API enhancements was taken from a comment by bnoordhuis in: https://github.com/libuv/libuv/issues/2506#issuecomment-540050665 Also add `internal_fields` to `uv_loop_t` to store the pointer to `uv__loop_internal_fields_t`. This naming makes more sense than just using `active_reqs.unused[1]`. To maintain ABI compatibility, shrink the `unused` array.

view details

Trevor Norris

commit sha c2cb5d63fce06d2a397a8134ecd53c63cf3a8f1e

unix,win: add API to measure event loop idle time The API addition `uv_metrics_idle_time()` is a thread safe call that allows the user to retrieve the amount of time the event loop has spent in the kernel's event provider (i.e. poll). It was done this way to allow retrieving this value without needing to interrupt the execution of the event loop. This option can be enabled by passing `UV_METRICS_IDLE_TIME` to `uv_loop_configure()`. One important aspect of this change is, when enabled, to always first call the event provider with a `timeout == 0`. This allows libuv to know whether any events were waiting in the event queue when the event provider was called. The importance of this is because libuv is tracking the amount of "idle time", not "poll time". Thus the provider entry time is not recorded when `timeout == 0` (the event provider never idles in this case). While this does add a small amount of overhead, when enabled, but the overhead decreases when the event loop has a heavier load. This is because poll events will be waiting when the event provider is called. Thus never actually recording the provider entry time. Checking if `uv_loop_t` is configured with `UV_METRICS_IDLE_TIME` always happens in `uv__metrics_set_provider_entry_time()` and `uv__metrics_update_idle_time()`. Making the conditional logic wrapping each call simpler and allows for instrumentation to always hook into those two function calls. Rather than placing the fields directly on `uv__loop_internal_fields_t` add the struct `uv__loop_metrics_t` as a location for future metrics API additions. Tests and additional documentation has been included.

view details

push time in 2 months

pull request commentlibuv/libuv

Add API for measuring event loop idle time

with composability you'll get questions from people on how to ensure they're really the last or first callback :-/

@bnoordhuis I forgot to mention that another reason for my API decision is because I wanted to remove the effect of users' observation. The timing is sensitive and as soon as multiple users are allowed to run callbacks the collected metrics all interfere w/ each other. In which case the cat always ends up dead.

I'm not opposed to changing the implementation, and will be happy to do so if we can find one that 1) has the necessary precision and 2) doesn't cause multiple users to interfere with each other.

trevnorris

comment created time in 2 months

push eventtrevnorris/libuv

Ben Noordhuis

commit sha 87cbf546d8d4db769e7524304713af39322c83fd

test: remove unused extern declaration PR-URL: https://github.com/libuv/libuv/pull/2755 Reviewed-By: Richard Lau <riclau@uk.ibm.com>

view details

Ben Noordhuis

commit sha fa1da1854946bf9517ca8c0e76dff8e41bb35963

test: canonicalize argv[0] in exepath test Commit ff29322b ("test: canonicalize test runner path") from 2014 changed the test runner to call `realpath(3)` on `argv[0]` in order to fix the `get_currentexe` test failing with the autotools build when the executable path contained symbolic links but that is now causing the `spawn_reads_child_path` test to fail on z/os with the cmake build. Fix that by only doing path canonicalization in the `get_currentexe` test, not always. An auxiliary fix is applied to the `process_title_threadsafe` test because it assumed that setting the process title to a long string, then reading it back produces in the original string. On some platforms however the maximum size of the process title is limited to the size of the `argv` vector. Because the test runner used absolute paths until now, the argv vector was bigger than it is with relative paths, big enough to let this bad assumption go unnoticed until now. Minor fixes are applied to tests that assumed 1024 for the maximum path size because this commit makes visible that some of the CI matrix systems support much longer paths. PR-URL: https://github.com/libuv/libuv/pull/2755 Refs: https://github.com/libuv/libuv/pull/2737#issuecomment-602800431 Refs: https://github.com/libuv/libuv/pull/2754#issuecomment-604015785 Reviewed-By: Richard Lau <riclau@uk.ibm.com>

view details

Ben Noordhuis

commit sha 0ed9692a80b2f310ad52060a8cb400f2a963d09c

test: simplify platform_init() PR-URL: https://github.com/libuv/libuv/pull/2755 Reviewed-By: Richard Lau <riclau@uk.ibm.com>

view details

Kevin Adler

commit sha 5f1ac8ee360813e91821adfe845123e27e2c43e2

ibmi: Fix isatty EBADF handling and refactor In e14c56b, support was added to implement true isatty() support when running in the IBM i PASE environment, but it did not handle EBADF properly. This commit fixes the EBADF handling, but because the handling was a bit more complicated than previously, it was moved to a separate isatty function to keep the mainline code simpler. This also documents why we can't just use isatty on PASE a little more completely. PR-URL: https://github.com/libuv/libuv/pull/2753 Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl> Reviewed-By: Richard Lau <riclau@uk.ibm.com>

view details

Kevin Adler

commit sha 4878e8284f5f983968de188d6d740bd3f6ac0e2f

test: Test EBADF tty handling PR-URL: https://github.com/libuv/libuv/pull/2753 Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl> Reviewed-By: Richard Lau <riclau@uk.ibm.com>

view details

Ben Noordhuis

commit sha e15a3c45d86af05b8cfc2f7d984df3536df3d3f0

build: make cmake build benchmarks PR-URL: https://github.com/libuv/libuv/pull/2771 Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Richard Lau <riclau@uk.ibm.com>

view details

Ben Noordhuis

commit sha 335e8a6d128646e5a19d39dfc677f5a5a555f7cc

win: use RtlGenRandom from advapi32.dll directly At least two people have reported that `LoadLibrary("advapi32.dll")` fails in some configurations. Libuv already links against advapi32.dll so let's sidestep the issue by linking to `RtlGenRandom()` directly instead of looking it up at runtime. Fixes: https://github.com/libuv/libuv/issues/2759 PR-URL: https://github.com/libuv/libuv/pull/2762 Reviewed-By: Bartosz Sosnowski <bartosz@janeasystems.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Jameson Nash <vtjnash@gmail.com>

view details

Lin Zhang

commit sha 7656e981b6633635e72bacc447bc6d3ab88ce0cd

android: fix OOB write in uv_interface_addresses() PR-URL: https://github.com/libuv/libuv/pull/2767 Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl> Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: David Carlier <devnexen@gmail.com> Reviewed-By: Richard Lau <riclau@uk.ibm.com>

view details

毛毛

commit sha 0e8ff951f3ed85039c67024733bc3dded84010e6

test: pass test when hostname is single character PR-URL: https://github.com/libuv/libuv/pull/2741 Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl> Reviewed-By: Colin Ihrig <cjihrig@gmail.com>

view details

Xu Meng

commit sha d21f5aeab0a4b6193f13640471cde405579f9c69

ibmi: set the highest process priority to -10 On IBMi PASE, the highest process priority is -10. PR-URL: https://github.com/libuv/libuv/pull/2642 Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl> Reviewed-By: Richard Lau <riclau@uk.ibm.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com>

view details

Trevor Norris

commit sha a13b96f43bd725154d553528e0239f073b3422ca

zos: fix potential event loop stall This is a port of 70002c80 to z/OS to fix the same potential issue that could effectively enter an infinite loop (but not a busy loop) under certain conditions when polling for events.

view details

Trevor Norris

commit sha faf602846cce3e68d2804387ef6fb7904fb71b55

include: add internal fields struct to uv_loop_t Add struct `uv__loop_internal_fields_t` as a location for future additions to `uv_loop_t` while also maintaining v1.x compatibility. Currently `uv__loop_internal_fields_t` only contains the `flags` field. The reason for adding the `flags` field is because the same field was never added to `UV_LOOP_PRIVATE_FIELDS` in Windows as it was in Unix. The idea for creating a struct and attaching it to `uv_loop_t` for future API enhancements was taken from a comment by bnoordhuis in: https://github.com/libuv/libuv/issues/2506#issuecomment-540050665 Also add `internal_fields` to `uv_loop_t` to store the pointer to `uv__loop_internal_fields_t`. This naming makes more sense than just using `active_reqs.unused[1]`. To maintain ABI compatibility, shrink the `unused` array.

view details

Trevor Norris

commit sha a3c2387982c9fbe2fb001bfdbac6d07536a32d0e

unix,win: add API to measure event loop idle time The API addition `uv_metrics_idle_time()` is a thread safe call that allows the user to retrieve the amount of time the event loop has spent in the kernel's event provider (i.e. poll). It was done this way to allow retrieving this value without needing to interrupt the execution of the event loop. This option can be enabled by passing `UV_METRICS_IDLE_TIME` to `uv_loop_configure()`. One important aspect of this change is, when enabled, to always first call the event provider with a `timeout == 0`. This allows libuv to know whether any events were waiting in the event queue when the event provider was called. The importance of this is because libuv is tracking the amount of "idle time", not "poll time". Thus the provider entry time is not recorded when `timeout == 0` (the event provider never idles in this case). While this does add a small amount of overhead, when enabled, but the overhead decreases when the event loop has a heavier load. This is because poll events will be waiting when the event provider is called. Thus never actually recording the provider entry time. Checking if `uv_loop_t` is configured with `UV_METRICS_IDLE_TIME` always happens in `uv__metrics_set_provider_entry_time()` and `uv__metrics_update_idle_time()`. Making the conditional logic wrapping each call simpler and allows for instrumentation to always hook into those two function calls. Rather than placing the fields directly on `uv__loop_internal_fields_t` add the struct `uv__loop_metrics_t` as a location for future metrics API additions. Tests and additional documentation has been included.

view details

push time in 2 months

pull request commentlibuv/libuv

Add API for measuring event loop idle time

@bnoordhuis Really has been a long time. :)

The goal is to obtain the precise time libuv spends in epoll_pwait/kevent/etc.?

More specifically it's an attempt to measure the amount of idle time in the event provider (e.g. epoll_pwait, etc.). Below is a timeline of the event loop making event provider requests and time spent processing those requests; and when events are received by the event queue.

<pre style="line-height:1;"><code> ╭──────╮ │ idle │ ╓───┬───────────┬─────────┼──────┼───╖ event ║ │ │ ┝━━━━━━┥ ║ loop ║ │ │ ┝━━━━━━┥ ║ ╟─┬─┴───┬───┬───┴─────────┴──────┼───╢ event ║ │ │ │ │ ║ queue ║ │ │ │ │ ║ ╙─┴─────┴───┴────────────────────┴───╜</code></pre>

Here we can see that regardless of when events are placed in the event queue they are not received by the event loop until after it has completed processing all previous events. In these cases the event loop was never actually idle during the event provider request. Since the purposed of the metrics is to measure the idle time we need to be able to determine if the event provider request was ever actually idle during its execution.

In order to make this measurement more precise, part of this PR always makes the initial event provider request (i.e. call to epoll_pwait) with a timeout == 0 (I'm doing it this way because there isn't another method to get the necessary information). Since the event provider request returns immediately we can surmise whether a subsequent request will contain some idle time. Afterwards it'll reset timeout and resume execution as normal. Potentially making the event provider request again if timeout > 0 and no events were waiting in the event queue.

I'd say that can be simplified to two simple callbacks, either global or per-loop. Metrics collection shouldn't be libuv's job, just the means to enable it.

I don't disagree with you, but I also couldn't find a simpler method to collect metrics with the precision necessary to support the research I explained in https://github.com/libuv/libuv/pull/2725#issuecomment-594752268 using a simple callback API. For example, if all time in the event provider was considered idle time then idle time is always > 0. Even if there are always waiting to be processed in the event queue.

trevnorris

comment created time in 2 months

push eventtrevnorris/libuv

Trevor Norris

commit sha 1ad8ff8a9b1893f414dd5a6883d613a456d39bfa

unix,win: add API to measure event loop idle time The API addition `uv_metrics_idle_time()` is a thread safe call that allows the user to retrieve the amount of time the event loop has spent in the kernel's event provider (i.e. poll). It was done this way to allow retrieving this value without needing to interrupt the execution of the event loop. This option can be enabled by passing `UV_METRICS_IDLE_TIME` to `uv_loop_configure()`. One important aspect of this change is, when enabled, to always first call the event provider with a `timeout == 0`. This allows libuv to know whether any events were waiting in the event queue when the event provider was called. The importance of this is because libuv is tracking the amount of "idle time", not "poll time". Thus the provider entry time is not recorded when `timeout == 0` (the event provider never idles in this case). While this does add a small amount of overhead, when enabled, but the overhead decreases when the event loop has a heavier load. This is because poll events will be waiting when the event provider is called. Thus never actually recording the provider entry time. Checking if `uv_loop_t` is configured with `UV_METRICS_IDLE_TIME` always happens in `uv__metrics_set_provider_entry_time()` and `uv__metrics_update_idle_time()`. Making the conditional logic wrapping each call simpler and allows for instrumentation to always hook into those two function calls. Rather than placing the fields directly on `uv__loop_internal_fields_t` add the struct `uv__loop_metrics_t` as a location for future metrics API additions. Tests and additional documentation has been included.

view details

push time in 2 months

push eventtrevnorris/libuv

Trevor Norris

commit sha 6b63e12c0dfad5aeaefc55dc3ed1b7f626f6760b

unix,win: add API to measure event loop idle time The API addition `uv_metrics_idle_time()` is a thread safe call that allows the user to retrieve the amount of time the event loop has spent in the kernel's event provider (i.e. poll). It was done this way to allow retrieving this value without needing to interrupt the execution of the event loop. This option can be enabled by passing `UV_METRICS_IDLE_TIME` to `uv_loop_configure()`. One important aspect of this change is, when enabled, to always first call the event provider with a `timeout == 0`. This allows libuv to know whether any events were waiting in the event queue when the event provider was called. The importance of this is because libuv is tracking the amount of "idle time", not "poll time". Thus the provider entry time is not recorded when `timeout == 0` (the event provider never idles in this case). While this does add a small amount of overhead, when enabled, but the overhead decreases when the event loop has a heavier load. This is because poll events will be waiting when the event provider is called. Thus never actually recording the provider entry time. Checking if `uv_loop_t` is configured with `UV_METRICS_IDLE_TIME` always happens in `uv__metrics_set_provider_entry_time()` and `uv__metrics_update_idle_time()`. Making the conditional logic wrapping each call simpler and allows for instrumentation to always hook into those two function calls. Rather than placing the fields directly on `uv__loop_internal_fields_t` add the struct `uv__loop_metrics_t` as a location for future metrics API additions. Tests and additional documentation has been included.

view details

push time in 2 months

push eventtrevnorris/libuv

Santiago Gimeno

commit sha 53caf044e39476c4888f21a79e425391027cdd58

test: add a bunch of ASSERT macros To make the debugging of test issues easier. The following integer macros are added: `ASSERT_EQ(a, b)`, `ASSERT_GE(a, b)`, `ASSERT_GT(a, b)`, `ASSERT_LE(a, b)`, `ASSERT_LT(a, b)` and `ASSERT_NE(a, b)`. And its corresponding unsigned integer macros: `ASSERT_UINT64_EQ(a, b)`, `ASSERT_UINT64_GE(a, b)`, `ASSERT_UINT64_GT(a, b)`, `ASSERT_UINT64_LE(a, b)`, `ASSERT_UINT64_LT(a, b)` and `ASSERT_UINT64_NE(a, b)`. Also these macros for `NULL` and pointer checks: `ASSERT_NULL(a)`, `ASSERT_NOT_NULL(a)`, `ASSERT_PTR_EQ(a, b)` and `ASSERT_PTR_NE(a, b)`. And finally these macros for strings and buffers: `ASSERT_STR_EQ(a, b)`/`ASSERT_STR_NEQ(a, b)` that use the `strcmp()` call. `ASSERT_MEM_EQ(a, b)`/`ASSERT_MEM_NEQ(a, b)` and `ASSERT_MEM_HEX_EQ(a, b)`/`ASSERT_MEM_HEX_NEQ(a, b)` that use the `memcmp()` call. The former, prints the data in string format and the latter in hex format. These macros are used in the following way: ```c ASSERT_EQ(UV_EINVAL, uv_loop_close(&loop)); ``` With a sample output that would be as follows: ``` Assertion failed in test/test-loop-close.c on line 44: `UV_EINVAL == uv_loop_close(&loop)` (-22 == -16) ``` To view multiples examples if their use, the `test-ipc.c` file has been modified to use these macros. The `_ISOC99_SOURCE` is defined to support `inttypes.h` in `z/OS`. PR-URL: https://github.com/libuv/libuv/pull/2739 Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl> Reviewed-By: Richard Lau <riclau@uk.ibm.com>

view details

Trevor Norris

commit sha 3ac51fac69b722560f9ac74f9147b70a7eb24b18

zos: fix potential event loop stall This is a port of 70002c80 to z/OS to fix the same potential issue that could effectively enter an infinite loop (but not a busy loop) under certain conditions when polling for events.

view details

Trevor Norris

commit sha 6ba7e8f31effeeb4c8913db928b1ff2ff33e564f

include: add internal fields struct to uv_loop_t Add struct `uv__loop_internal_fields_t` as a location for future additions to `uv_loop_t` while also maintaining v1.x compatibility. Currently `uv__loop_internal_fields_t` only contains the `flags` field. The reason for adding the `flags` field is because the same field was never added to `UV_LOOP_PRIVATE_FIELDS` in Windows as it was in Unix. The idea for creating a struct and attaching it to `uv_loop_t` for future API enhancements was taken from a comment by bnoordhuis in: https://github.com/libuv/libuv/issues/2506#issuecomment-540050665 Also add `internal_fields` to `uv_loop_t` to store the pointer to `uv__loop_internal_fields_t`. This naming makes more sense than just using `active_reqs.unused[1]`. To maintain ABI compatibility, shrink the `unused` array.

view details

Trevor Norris

commit sha d3df37e34d2292458ba258461194889f6b6d7440

unix,win: add API to measure event loop idle time The API addition `uv_metrics_idle_time()` is a thread safe call that allows the user to retrieve the amount of time the event loop has spent in the kernel's event provider (i.e. poll). It was done this way to allow retrieving this value without needing to interrupt the execution of the event loop. This option can be enabled by passing `UV_METRICS_IDLE_TIME` to `uv_loop_configure()`. One important aspect of this change is, when enabled, to always first call the event provider with a `timeout == 0`. This allows libuv to know whether any events were waiting in the event queue when the event provider was called. The importance of this is because libuv is tracking the amount of "idle time", not "poll time". Thus the provider entry time is not recorded when `timeout == 0` (the event provider never idles in this case). While this does add a small amount of overhead, when enabled, but the overhead decreases when the event loop has a heavier load. This is because poll events will be waiting when the event provider is called. Thus never actually recording the provider entry time. Checking if `uv_loop_t` is configured with `UV_METRICS_IDLE_TIME` always happens in `uv__metrics_set_provider_entry_time()` and `uv__metrics_update_idle_time()`. Making the conditional logic wrapping each call simpler and allows for instrumentation to always hook into those two function calls. Rather than placing the fields directly on `uv__loop_internal_fields_t` add the struct `uv__loop_metrics_t` as a location for future metrics API additions. Tests and additional documentation has been included.

view details

push time in 2 months

pull request commentlibuv/libuv

Add API for measuring event loop idle time

I've reordered the commits. Now the addition of uv__loop_internal_fields_t has been placed before the metrics patch. I believe this will make bisecting and potential reverts easier. Also renamed a few variables to make things more clear.

@santigimeno Mind kicking off another CI to make sure I didn't accidentally mess something up?

trevnorris

comment created time in 2 months

push eventtrevnorris/libuv

Trevor Norris

commit sha 6cbfa53eb9b5a53075f1217bd278b203ca7ace8a

include: add internal fields struct to uv_loop_t Add struct `uv__loop_internal_fields_t` as a location for future additions to `uv_loop_t` while also maintaining v1.x compatibility. Currently `uv__loop_internal_fields_t` only contains the `flags` field. The reason for adding the `flags` field is because the same field was never added to `UV_LOOP_PRIVATE_FIELDS` in Windows as it was in Unix. The idea for creating a struct and attaching it to `uv_loop_t` for future API enhancements was taken from a comment by bnoordhuis in: https://github.com/libuv/libuv/issues/2506#issuecomment-540050665 Also add `internal_fields` to `uv_loop_t` to store the pointer to `uv__loop_internal_fields_t`. This naming makes more sense than just using `active_reqs.unused[1]`. To maintain ABI compatibility, shrink the `unused` array.

view details

Trevor Norris

commit sha 212e3a70954cf49ccd22a0b11d9ab00bbdf25301

unix,win: add API to measure event loop idle time The API addition `uv_metrics_idle_time()` is a thread safe call that allows the user to retrieve the amount of time the event loop has spent in the kernel's event provider (i.e. poll). It was done this way to allow retrieving this value without needing to interrupt the execution of the event loop. This option can be enabled by passing `UV_METRICS_IDLE_TIME` to `uv_loop_configure()`. One important aspect of this change is, when enabled, to always first call the event provider with a `timeout == 0`. This allows libuv to know whether any events were waiting in the event queue when the event provider was called. The importance of this is because libuv is tracking the amount of "idle time", not "poll time". Thus the provider entry time is not recorded when `timeout == 0` (the event provider never idles in this case). While this does add a small amount of overhead, when enabled, but the overhead decreases when the event loop has a heavier load. This is because poll events will be waiting when the event provider is called. Thus never actually recording the provider entry time. Checking if `uv_loop_t` is configured with `UV_METRICS_IDLE_TIME` always happens in `uv__metrics_set_provider_entry_time()` and `uv__metrics_update_idle_time()`. Making the conditional logic wrapping each call simpler and allows for instrumentation to always hook into those two function calls. Rather than placing the fields directly on `uv__loop_internal_fields_t` add the struct `uv__loop_metrics_t` as a location for future metrics API additions. Tests and additional documentation has been included.

view details

push time in 2 months

pull request commenttrevnorris/cbuffer

Setup Continuous Integration

A bit delayed, but thanks for the contribution!

AdrieanKhisbe

comment created time in 2 months

push eventtrevnorris/cbuffer

Adrien Becchis

commit sha eadf3ce12b0b3e8c8fb651f80bcfaa1ce70488a8

Setup Continuous Integration Changes: * Add travis.yml * Add CI badges to README * Add licence TAG to README * Improved formating of README and add ring mention * Add keywords to package.json for better searchability on npm PR-URL: https://github.com/trevnorris/cbuffer/pull/27

view details

push time in 2 months

pull request commenttrevnorris/cbuffer

Added map and join methods. Also made negative indices work for get

@dfellis It's been a while. Didn't realize people still used this library. Thanks for the PR. It's now been squashed and merged.

dfellis

comment created time in 2 months

push eventtrevnorris/cbuffer

David Ellis

commit sha 3c4aa55a1002331b4b3fdff52cc8e53938920434

Add map and join, and negative indices support * Added map and join methods. Also made negative indices work for get * Switch from spaces to tabs to match project PR-URL: https://github.com/trevnorris/cbuffer/pull/26

view details

push time in 2 months

PR merged trevnorris/cbuffer

Added map and join methods. Also made negative indices work for get

Hi, thanks for getting this project started for a circular buffer with an Array interface!

For my own usage, I needed map and join, so at first I just did a toArray() and did them, but would prefer eliminating the extra temporary object, so I've implemented the parts I need and wrote tests for them.

While writing the tests I noticed that the get function doesn't allow negative indexing, which seems useful for Circular Buffers in a way that it wouldn't be for a normal Array, so I also added a negative-capable modulus function and used it there.

+87 -2

3 comments

3 changed files

dfellis

pr closed time in 2 months

pull request commentjstat/jstat

Array ranking efficiency

@suzukia This is excellent. Thanks for the contribution. Can I ask what code you used to perform the benchmarks, and what you used to test it (e.g. chrome, node.js, etc.)?

suzukia

comment created time in 2 months

issue commentjstat/jstat

T Statistics issue

@vincerubinetti I created https://github.com/jstat/jstat/issues/256 from your comment. Will write up a fix based on your feedback.

cherylcaprina

comment created time in 2 months

issue openedjstat/jstat

Fix Welch's ttest results

I was using this code to replace the ttest library in the project I'm working on. But when I started comparing it to results from R, I noticed some numeric/floating point errors.

Using the same a and b test data from the above post, here are the Welch's ttest results, from the above algorithm, from R, and from the ttest package, respectively:

0.473787465132 0.472242366784 0.472242366784

So, to the authors, you might want to look into this more closely before you make a PR, and to anyone else stumbling across this thread, it's probably not safe to use the above code.

EDIT I seem to have found the error. Removing the round on t_df_upper/t_df_lower makes the pValue results match that of R.

Originally posted by @vincerubinetti in https://github.com/jstat/jstat/issues/189#issuecomment-590549351

created time in 2 months

issue commentjstat/jstat

Looking for equivalent to statsmodels.tsa.stattools.adfuller

Sorry. I just maintain the code and am not a statistics expert. Will require input from the community on this one.

runvnc

comment created time in 2 months

pull request commentlibuv/libuv

Add API for measuring event loop idle time

Ping @saghul or @bnoordhuis

trevnorris

comment created time in 2 months

push eventtrevnorris/libuv

Lin Zhang

commit sha 055e89f637fcd767096a3b258243c08fc5c93ddf

build: fix gyp build for Android API >= 28 PR-URL: https://github.com/libuv/libuv/pull/2749 Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>

view details

Saúl Ibarra Corretgé

commit sha d9cd7d437d6bcea56355b6e0ef215aa64687d7a1

udp: return recvmmsg-ed datagrams in order When recvmmsg support was added it returned the datagrams in reverse received order, which may impact some applications. To restore the previous behavior, we call recv_cb one last time with nread == 0 and addr == NULL so applications can free the buffer. PR-URL: https://github.com/libuv/libuv/pull/2736 Reviewed-By: Santiago Gimeno <santiago.gimeno@gmail.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com>

view details

Richard Lau

commit sha 1099d298d424546b74db11bf6fb7a51b7954fab6

zos,test: fix spawn_empty_env for shared library build z/OS uses LIBPATH instead of LD_LIBRARY_PATH for the search path for shared libraries. PR-URL: https://github.com/libuv/libuv/pull/2737 Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>

view details

Richard Lau

commit sha a9c58e72a34e957fbff6b5020e1a29e32d269115

zos: fix non-Release builds z/OS doesn't support POSIX semaphores. On Release builds the code paths that reference POSIX semaphore functions (e.g. sem_init()) are optimized away so linking succeeds but on a non-Release build the references to the unavailable functions result in unresolved symbol errors. Stub the unavailable POSIX semaphore functions on z/OS as so the code can link on non-Release builds. PR-URL: https://github.com/libuv/libuv/pull/2737 Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>

view details

Richard Lau

commit sha 875a4fe653f93073c14b6eb1a1f8d363583a54b9

zos: fix return value on expired nanosleep() call In the emulated nanosleep() function on z/OS an EAGAIN error from BPX1CTW/BPX4CTW indicates that the timeout has expired. In that case return 0 and not -1. PR-URL: https://github.com/libuv/libuv/pull/2737 Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>

view details

Richard Lau

commit sha c10431165abf0752525dfcb6172bf11a74b9ae62

build: fix z/OS cmake build - Correct CMAKE_SYSTEM_NAME. - Exclude pthread lib on z/OS. - Remove deleted src/unix/pthread-barrier.c. - Set LIBPATH for shared library test. PR-URL: https://github.com/libuv/libuv/pull/2737 Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>

view details

Trevor Norris

commit sha 1a252a051fedea43d4861c0b43deffca2ac28937

zos: fix potential event loop stall This is a port of 70002c80 to z/OS to fix the same potential issue that could effectively enter an infinite loop (but not a busy loop) under certain conditions when polling for events.

view details

Trevor Norris

commit sha cd848f64f3ab3617e15749c26637f047fc9fa650

unix,win: add API to measure event loop idle time The API addition `uv_metrics_idle_time()` is a thread safe call that allows the user to retrieve the amount of time the event loop has spent in the kernel's event provider (i.e. poll). It was done this way to allow retrieving this value without needing to interrupt the execution of the event loop. This option can be enabled by passing `UV_METRICS_IDLE_TIME` to `uv_loop_configure()`. One important aspect of this change is, when enabled, to always first call the event provider with a `timeout == 0`. This allows libuv to know whether any events were waiting in the event queue when the event provider was called. The importance of this is because libuv is tracking the amount of "idle time", not "poll time". Thus the provider entry time is not recorded when `timeout == 0` (the event provider never idles in this case). While this does add a small amount of overhead, when enabled, but the overhead decreases when the event loop has a heavier load. This is because poll events will be waiting when the event provider is called. Thus never actually recording the provider entry time. Checking if `uv_loop_t` is configured with `UV_METRICS_IDLE_TIME` always happens in `uv__metrics_set_provider_entry_time()` and `uv__metrics_update_idle_time()`. Making the conditional logic wrapping each call simpler and allows for instrumentation to always hook into those two function calls. Add struct `uv__metrics_loop_t` as a location for future metrics API additions. Add struct `uv__loop_internal_fields_t` as a location for future additions to `uv_loop_t` while also maintaining v1.x compatibility. A pointer to this struct has been placed in the `uv_loop_t::unused[1]` space to allow for fast retrieval and maintaining ABI compatibility with v1.x. Currently `uv__loop_internal_fields_t` only contains both the `uv__metrics_loop_t` and `flags` field. The reason for adding the `flags` field is because the same field was never added to `UV_LOOP_PRIVATE_FIELDS` in Windows as it was in Unix, and adding that field would break ABI. The idea for creating a struct and attaching it to `uv_loop_t` for future API enhancements was taken from a comment by bnoordhuis in: https://github.com/libuv/libuv/issues/2506#issuecomment-540050665 Tests and additional documentation has been included.

view details

Trevor Norris

commit sha 068d287b98a6443fb824f79dedbf3d9adef7ec0b

include: add internal_fields to uv_loop_t Add `internal_fields` to `uv_loop_t` to store the pointer to `uv__loop_internal_fields_t`. This naming makes more sense than just continuing to use `active_reqs.unused[1]`. To maintain ABI compatibility, shrink the `unused` array.

view details

push time in 2 months

pull request commentlibuv/libuv

Add API for measuring event loop idle time

Rebased on latest v1.x and added a commit that changes using active_reqs.unused[1] to simply internal_fields.

trevnorris

comment created time in 2 months

push eventtrevnorris/libuv

Jesse Gorzinski

commit sha 104255f5af57153a9c590a002f53cfb6c15d4dd0

build: ibmi support for cmake PR-URL: https://github.com/libuv/libuv/pull/2729 Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl> Reviewed-By: Michael Dawson <michael_dawson@ca.ibm.com> Reviewed-By: Kevin Adler <kadler@us.ibm.com> Reviewed-By: Richard Lau <riclau@uk.ibm.com>

view details

Trevor Norris

commit sha d66778561c77eb9f429ef911db57d95d9cd42398

zos: fix potential event loop stall This is a port of 70002c80 to z/OS to fix the same potential issue that could effectively enter an infinite loop (but not a busy loop) under certain conditions when polling for events.

view details

Trevor Norris

commit sha e29f27a6bad59aeb3ef34e48dcc680bbc94c81a0

unix,win: add API to measure event loop idle time The API addition `uv_metrics_idle_time()` is a thread safe call that allows the user to retrieve the amount of time the event loop has spent in the kernel's event provider (i.e. poll). It was done this way to allow retrieving this value without needing to interrupt the execution of the event loop. This option can be enabled by passing `UV_METRICS_IDLE_TIME` to `uv_loop_configure()`. One important aspect of this change is, when enabled, to always first call the event provider with a `timeout == 0`. This allows libuv to know whether any events were waiting in the event queue when the event provider was called. The importance of this is because libuv is tracking the amount of "idle time", not "poll time". Thus the provider entry time is not recorded when `timeout == 0` (the event provider never idles in this case). While this does add a small amount of overhead, when enabled, but the overhead decreases when the event loop has a heavier load. This is because poll events will be waiting when the event provider is called. Thus never actually recording the provider entry time. Checking if `uv_loop_t` is configured with `UV_METRICS_IDLE_TIME` always happens in `uv__metrics_set_provider_entry_time()` and `uv__metrics_update_idle_time()`. Making the conditional logic wrapping each call simpler and allows for instrumentation to always hook into those two function calls. Add struct `uv__metrics_loop_t` as a location for future metrics API additions. Add struct `uv__loop_internal_fields_t` as a location for future additions to `uv_loop_t` while also maintaining v1.x compatibility. A pointer to this struct has been placed in the `uv_loop_t::unused[1]` space to allow for fast retrieval and maintaining ABI compatibility with v1.x. Currently `uv__loop_internal_fields_t` only contains both the `uv__metrics_loop_t` and `flags` field. The reason for adding the `flags` field is because the same field was never added to `UV_LOOP_PRIVATE_FIELDS` in Windows as it was in Unix, and adding that field would break ABI. The idea for creating a struct and attaching it to `uv_loop_t` for future API enhancements was taken from a comment by bnoordhuis in: https://github.com/libuv/libuv/issues/2506#issuecomment-540050665 Tests and additional documentation has been included.

view details

Trevor Norris

commit sha df564f68c47c4e692a183974aee306c99dd0d8d9

include: add internal_fields to uv_loop_t Add `internal_fields` to `uv_loop_t` to store the pointer to `uv__loop_internal_fields_t`. This naming makes more sense than just continuing to use `active_reqs.unused[1]`. To maintain ABI compatibility, shrink the `unused` array.

view details

push time in 2 months

pull request commentlibuv/libuv

Add API for measuring event loop idle time

Not sure why it's giving a 404 for libuv-test-commit-windows and libuv-test-commit-windows-cmake, but that patch does appear to have fixed the z/OS issue (thanks again @richardlau).

The last bit I'd like confirmation on is using uv_loop_t.unused[1]. I was thinking we could simply rename unused to internals, or something like that. I don't believe using that field will be a problem. Best I can tell sizeof(void*) >= sizeof(unsigned int) on all supported platforms. Any thoughts on this?

Also, is there anyone who doesn't like this change? Adding uv__loop_internal_fields_t helps solidify a long term method of staying on v1.x while being able to add new features.

trevnorris

comment created time in 2 months

pull request commentlibuv/libuv

Add API for measuring event loop idle time

I have ported the commit mentioned by @richardlau and added it as the first commit in this PR. I've also rebased on latest v1.x and removed the debug commit.

@santigimeno Mind kicking off CI one more time?

trevnorris

comment created time in 2 months

push eventtrevnorris/libuv

Irek Fakhrutdinov

commit sha f06734057b81eaffb35ebe25d5740e3942daf129

zos: explicitly mark message queue events The os390 epoll_wait implementation uses poll() to detect events in both file descriptors and the message queue used for file system events. The only message queue ID is always placed at the end of the array passed to the poll() call. When the poll() call returns all FDs and the message queue are checked for events by iterating through that array. In order to distinguish the message queue from the FDs its ID value is compared with the only message queue ID we have and if it matches the message queue handler function is called. When the message queue ID is relatively small, it may have the same value as the value of one of the file descriptors. If this happens, the message queue handler is called for the matching file descriptor, and this call fails. The file descriptor ends up being unhandled and this makes the next poll() call to return immediately. Eventually this will happen again and again, leading to an infinite busy loop and high CPU usage. To prevent the incorrect interpretation of file descriptors as the message queue, a new field has been added to the epoll event struct. This field is checked instead of the ID value and the message queue handler function is never called for file descriptors. PR-URL: https://github.com/libuv/libuv/pull/2013 Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl> Reviewed-By: Richard Lau <riclau@uk.ibm.com>

view details

Irek Fakhrutdinov

commit sha 7d988e0763cfe20dc37ac6f33d5fa92d7a397877

zos: move mq check out of loop to save cpu cycles PR-URL: https://github.com/libuv/libuv/pull/2013 Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl> Reviewed-By: Richard Lau <riclau@uk.ibm.com>

view details

Irek Fakhrutdinov

commit sha 506e4bee7bb1da3a35d8f4ce877549a84b8c5f7b

zos: add checks to ensure behavior of epoll_wait PR-URL: https://github.com/libuv/libuv/pull/2013 Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl> Reviewed-By: Richard Lau <riclau@uk.ibm.com> Signed-off-by: Irek Fakhrutdinov <ifakhrutdinov@rocketsoftware.com>

view details

Ben Noordhuis

commit sha 1bcfbfd00380c63b5d163b4b70c6469f53df80c0

src: add uv__reallocf() Modeled after FreeBSD's `reallocf(3)`: a version of `realloc(3)` that frees the memory when reallocation fails, simplifying error handling in many cases. PR-URL: https://github.com/libuv/libuv/pull/2735 Reviewed-By: Santiago Gimeno <santiago.gimeno@gmail.com> Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Saúl Ibarra Corretgé <saghul@gmail.com> Reviewed-By: Richard Lau <riclau@uk.ibm.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com>

view details

Trevor Norris

commit sha feac623678ded478284812bcb2312165d84b820d

zos: fix potential event loop stall This is a port of 70002c80 to z/OS to fix the same potential issue that could effectively enter an infinite loop (but not a busy loop) under certain conditions when polling for events.

view details

Trevor Norris

commit sha 2973a2a3efc702c96aa47e00d07e6383fe27e328

Add API for measuring event loop idle time The API addition `uv_metrics_idle_time()` is a thread safe call that allows the user to retrieve the amount of time the event loop has spent in the kernel's event provider (i.e. poll). It was done this way to allow retrieving this value without needing to interrupt the execution of the event loop. This option can be enabled by passing `UV_METRICS_IDLE_TIME` to `uv_loop_configure()`. One important aspect of this change is, when enabled, to always first call the event provider with a `timeout == 0`. This allows libuv to know whether any events were waiting in the event queue when the event provider was called. The importance of this is because libuv is tracking the amount of "idle time", not "poll time". Thus the provider entry time is not recorded when `timeout == 0` (the event provider never idles in this case). While this does add a small amount of overhead, when enabled, but the overhead decreases when the event loop has a heavier load. This is because poll events will be waiting when the event provider is called. Thus never actually recording the provider entry time. Checking if `uv_loop_t` is configured with `UV_METRICS_IDLE_TIME` always happens in `uv__metrics_set_provider_entry_time()` and `uv__metrics_update_idle_time()`. Making the conditional logic wrapping each call simpler and allows for instrumentation to always hook into those two function calls. Add struct `uv__metrics_loop_t` as a location for future metrics API additions. Add struct `uv__loop_internal_fields_t` as a location for future additions to `uv_loop_t` while also maintaining v1.x compatibility. A pointer to this struct has been placed in the `uv_loop_t::unused[1]` space to allow for fast retrieval and maintaining ABI compatibility with v1.x. Currently `uv__loop_internal_fields_t` only contains both the `uv__metrics_loop_t` and `flags` field. The reason for adding the `flags` field is because the same field was never added to `UV_LOOP_PRIVATE_FIELDS` in Windows as it was in Unix, and adding that field would break ABI. The idea for creating a struct and attaching it to `uv_loop_t` for future API enhancements was taken from a comment by bnoordhuis in: https://github.com/libuv/libuv/issues/2506#issuecomment-540050665 Tests and additional documentation has been included.

view details

push time in 2 months

pull request commentlibuv/libuv

Add API for measuring event loop idle time

@miladfarca Thank you much for investigating this. Definitely not something I could have found on my own.

I'll port that change here and reorder the commits to make it the first in this PR.

@dmabupt Sorry about that. I added a debug commit that always printed those values so I could get some info. Will be removing that commit before landing. Thanks for testing.

trevnorris

comment created time in 2 months

push eventtrevnorris/libuv

Trevor Norris

commit sha 713e82762c4ab9e9f526910f3d3b17fc3e70a301

Add API for measuring event loop idle time The API addition `uv_metrics_idle_time()` is a thread safe call that allows the user to retrieve the amount of time the event loop has spent in the kernel's event provider (i.e. poll). It was done this way to allow retrieving this value without needing to interrupt the execution of the event loop. This option can be enabled by passing `UV_METRICS_IDLE_TIME` to `uv_loop_configure()`. One important aspect of this change is, when enabled, to always first call the event provider with a `timeout == 0`. This allows libuv to know whether any events were waiting in the event queue when the event provider was called. The importance of this is because libuv is tracking the amount of "idle time", not "poll time". Thus the provider entry time is not recorded when `timeout == 0` (the event provider never idles in this case). While this does add a small amount of overhead, when enabled, but the overhead decreases when the event loop has a heavier load. This is because poll events will be waiting when the event provider is called. Thus never actually recording the provider entry time. Checking if `uv_loop_t` is configured with `UV_METRICS_IDLE_TIME` always happens in `uv__metrics_set_provider_entry_time()` and `uv__metrics_update_idle_time()`. Making the conditional logic wrapping each call simpler and allows for instrumentation to always hook into those two function calls. Add struct `uv__metrics_loop_t` as a location for future metrics API additions. Add struct `uv__loop_internal_fields_t` as a location for future additions to `uv_loop_t` while also maintaining v1.x compatibility. A pointer to this struct has been placed in the `uv_loop_t::unused[1]` space to allow for fast retrieval and maintaining ABI compatibility with v1.x. Currently `uv__loop_internal_fields_t` only contains both the `uv__metrics_loop_t` and `flags` field. The reason for adding the `flags` field is because the same field was never added to `UV_LOOP_PRIVATE_FIELDS` in Windows as it was in Unix, and adding that field would break ABI. The idea for creating a struct and attaching it to `uv_loop_t` for future API enhancements was taken from a comment by bnoordhuis in: https://github.com/libuv/libuv/issues/2506#issuecomment-540050665 Tests and additional documentation has been included.

view details

Trevor Norris

commit sha ee58fbf3a507ae8e88abec2975cc023da2f816f2

REMOVE BEFORE LANDING Add printf() statements for test debugging

view details

push time in 2 months

pull request commentlibuv/libuv

Add API for measuring event loop idle time

Made a few changes to the PR:

  • Rebased on latest v1.x
  • Squashed all but the "Add printf()` statements [...]" commit (will remove that commit before it lands)
  • Change the pointer of uv__metrics_loop_t in uv__loop_internal_fields_t to a field. No reason to allocate memory twice.

@kadler Do you know someone that could test IBM i and let me know if it works? Also @jBarz was the one that did the current z/OS implementation (in 011e02e3e5b). He still work at IBM?

At the minimum I'd like to get the approval from @bnoordhuis and/or @saghul. Also @piscisaureus or @vtjnash.

trevnorris

comment created time in 2 months

push eventtrevnorris/libuv

Trevor Norris

commit sha db5e9e99eb04e624e28f687b115f3a1c6bda47e4

Add API for measuring event loop idle time The API addition `uv_metrics_idle_time()` is a thread safe call that allows the user to retrieve the amount of time the event loop has spent in the kernel's event provider (i.e. poll). It was done this way to allow retrieving this value without needing to interrupt the execution of the event loop. This option can be enabled by passing `UV_METRICS_IDLE_TIME` to `uv_loop_configure()`. One important aspect of this change is, when enabled, to always first call the event provider with a `timeout == 0`. This allows libuv to know whether any events were waiting in the event queue when the event provider was called. The importance of this is because libuv is tracking the amount of "idle time", not "poll time". Thus the provider entry time is not recorded when `timeout == 0` (the event provider never idles in this case). While this does add a small amount of overhead, when enabled, but the overhead decreases when the event loop has a heavier load. This is because poll events will be waiting when the event provider is called. Thus never actually recording the provider entry time. Checking whether the `uv_loop_t` is configured with UV_METRICS_IDLE_TIME always happens in `uv__metrics_set_provider_entry_time()` and `uv__metrics_update_idle_time()`. Making the conditional logic wrapping each call simpler and allows for instrumentation to always hook into those two function calls. Add struct `uv__metrics_loop_t` as a location for future metrics API additions. Add struct `uv__loop_internal_fields_t` as a location for future additions to `uv_loop_t` while also maintaining v1.x compatibility. A pointer to this struct has been placed in the `uv_loop_t::unused[1]` space to allow for fast retrieval and maintaining ABI compatibility with v1.x. Currently `uv__loop_internal_fields_t` only contains both the `uv__metrics_loop_t` and `flags` field. The reason for adding the `flags` field is because the same field was never added to `UV_LOOP_PRIVATE_FIELDS` in Windows as it was in Unix, and adding that field would break ABI. The idea for creating a struct and attaching it to `uv_loop_t` for future API enhancements was taken from a comment by bnoordhuis in: https://github.com/libuv/libuv/issues/2506#issuecomment-540050665 Tests and additional documentation has been included.

view details

Trevor Norris

commit sha 32f0d0156d63fdce955fe88795dee29cfbbfe78b

Add printf() statements for test debugging

view details

push time in 2 months

push eventtrevnorris/libuv

Jesse Gorzinski

commit sha 742e0ba7dfba344484bba2493a8c5ddf164a8c30

build: add aix-common.c for AIX cmake build PR-URL: https://github.com/libuv/libuv/pull/2731 Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl> Reviewed-By: Richard Lau <riclau@uk.ibm.com> Reviewed-By: Michael Dawson <michael_dawson@ca.ibm.com>

view details

Trevor Norris

commit sha 5f74382b88407fe44e39a3390801703e1f22e360

Add API for measuring event loop idle time The API addition `uv_metrics_idle_time()` is a thread safe call that allows the user to retrieve the amount of time the event loop has spent in the kernel's event provider (i.e. poll). It was done this way to allow retrieving this value without needing to interrupt the execution of the event loop. This option can be enabled by passing `UV_METRICS_IDLE_TIME` to `uv_loop_configure()`. One important aspect of this change is, when enabled, to always first call the event provider with a `timeout == 0`. This allows libuv to know whether any events were waiting in the event queue when the event provider was called. The importance of this is because libuv is tracking the amount of "idle time", not "poll time". Thus the provider entry time is not recorded when `timeout == 0` (the event provider never idles in this case). While this does add a small amount of overhead, when enabled, but the overhead decreases when the event loop has a heavier load. This is because poll events will be waiting when the event provider is called. Thus never actually recording the provider entry time. Checking whether the `uv_loop_t` is configured with UV_METRICS_IDLE_TIME always happens in `uv__metrics_set_provider_entry_time()` and `uv__metrics_update_idle_time()`. Making the conditional logic wrapping each call simpler and allows for instrumentation to always hook into those two function calls. Add struct `uv__metrics_loop_t` as a location for future metrics API additions. Add struct `uv__loop_internal_fields_t` as a location for future additions to `uv_loop_t` while also maintaining v1.x compatibility. A pointer to this struct has been placed in the `uv_loop_t::unused[1]` space to allow for fast retrieval and maintaining ABI compatibility with v1.x. Currently `uv__loop_internal_fields_t` only contains both the `uv__metrics_loop_t` and `flags` field. The reason for adding the `flags` field is because the same field was never added to `UV_LOOP_PRIVATE_FIELDS` in Windows as it was in Unix, and adding that field would break ABI. The idea for creating a struct and attaching it to `uv_loop_t` for future API enhancements was taken from a comment by bnoordhuis in: https://github.com/libuv/libuv/issues/2506#issuecomment-540050665 Tests and additional documentation has been included. FIXUP

view details

Trevor Norris

commit sha ee15fe29cabb4cdf43ad0e5f0b48ff2ee6883ec3

Add printf() statements for test debugging

view details

push time in 2 months

pull request commentlibuv/libuv

Add API for measuring event loop idle time

Only tests that are failing are on zos:

not ok 158 - metrics_idle_time
# exit code 3
# Output from process `metrics_idle_time`:
# pt: 0
# Assertion failed in ../test/test-metrics.c on line 52: pt >= 80000000
# CEE5207E The signal SIGABRT was received.
not ok 159 - metrics_idle_time_thread
# exit code 3
# Output from process `metrics_idle_time_thread`:
# pt: 0   ptt: 0
# Assertion failed in ../test/test-metrics.c on line 74: pt >= (100000000 * mthread->thread_timer_count) - 20000000

Conceptually this should be impossible, and not something I can debug on my own. There a zos group that could help me out with this?

trevnorris

comment created time in 2 months

push eventtrevnorris/libuv

Trevor Norris

commit sha 45510c5266e7f6736404cc3df85e8d176b66322c

Add printf() statements for test debugging

view details

push time in 2 months

push eventtrevnorris/libuv

Trevor Norris

commit sha 0b47553d8884087918d3f0438d357bf6fc58463f

Add printf() statements for test debugging

view details

push time in 2 months

push eventtrevnorris/libuv

Trevor Norris

commit sha ab169dcee3502cca09d5f254b4c8b8d6f4a88615

Add printf() statements for test debugging

view details

push time in 2 months

push eventtrevnorris/libuv

Trevor Norris

commit sha 6a45c42d2c2708b8e1b402cc0e48061d8653d835

FIXUP Latest code review fixes

view details

push time in 2 months

Pull request review commentlibuv/libuv

Add API for measuring event loop idle time

+#include "uv.h"+#include "task.h"+#include <string.h> /* memset */+++typedef struct {+  int metrics_run_count;+  int timer_run_count;+} mtc_t;++typedef struct {+  uv_async_t async;+  uv_loop_t mloop;+  uv_timer_t timer;+  uv_thread_t thread;+  uv_timer_t ttimer;+  uint64_t thread_timer_count;+} mthread_t;+++static void timer_noop_cb(uv_timer_t* handle) {+  (*(int*) handle->data)++;+}+++TEST_IMPL(metrics_idle_time) {+  uv_timer_t timer;+  uint64_t pt;+  int cntr;+  int r;++  cntr = 0;+  timer.data = &cntr;+  uv_loop_configure(uv_default_loop(), UV_METRICS_IDLE_TIME);+  uv_timer_init(uv_default_loop(), &timer);++  uv_timer_start(&timer, timer_noop_cb, 100, 0);+

Fixed them all.

trevnorris

comment created time in 2 months

push eventtrevnorris/libuv

Trevor Norris

commit sha 9c47c89a6768f9bf763b4059bb2aa8bd0885df1f

FIXUP Latest code review fixes

view details

push time in 2 months

Pull request review commentlibuv/libuv

Add API for measuring event loop idle time

 void uv__io_poll(uv_loop_t* loop, int timeout) {   count = 48; /* Benchmarks suggest this gives the best throughput. */   real_timeout = timeout; +  if (uv__get_internal_fields(loop)->flags & UV_METRICS_IDLE_TIME) {+    reset_timeout = 1;+    user_timeout = timeout;+    timeout = 0;+  } else {+    reset_timeout = 0;+  }+   for (;;) {+    /* Only need to set the provider_entry_time if timeout != 0. The function+     * will return early if the loop isn't configured with UV_METRICS_IDLE_TIME.+     */+    if (timeout != 0)+      uv__metrics_set_provider_entry_time(loop);

That's the way it would always happen. It just wasn't as apparent before.

Reason why idle time is not recorded when timeout == 0 is because at no time is poll actually idling. It's requesting events from the kernel's event queue then immediately returning.

trevnorris

comment created time in 2 months

push eventtrevnorris/libuv

Trevor Norris

commit sha 2b7bd9bba9451d9e82da8a51d33f16e76754af65

FIXUP Remove unused header

view details

push time in 2 months

push eventtrevnorris/libuv

Trevor Norris

commit sha da11b6947fd9fca4fd94b0b0e0243678eca1792e

Change UV_LOOP_IDLE_TIME to UV_METRICS_IDLE_TIME Feels strange to have the call be uv_metrics_idle_time but have the flag UV_LOOP_IDLE_TIME. So make them match.

view details

push time in 2 months

push eventtrevnorris/libuv

Trevor Norris

commit sha bc6940b2e04acd6491eaecb7d372dde0b4bcfa9c

Change UV_LOOP_IDLE_TIME to UV_METRICS_IDLE_TIME Feels strange to have the call be uv_metrics_idle_time but have the flag UV_LOOP_IDLE_TIME. So make them match.

view details

push time in 2 months

pull request commentlibuv/libuv

Add API for measuring event loop idle time

Alright. Fixed the valgrind test failures and cleaned up a couple things. Should be ready to go.

trevnorris

comment created time in 2 months

push eventtrevnorris/libuv

Trevor Norris

commit sha 0ea12e9471d9739a1d804a726bf6559e80f136dd

Move some mem management around, close test loops * Forgot to properly close the uv_loop_t's used in tests. * Fix tests so they don't fail tests when using valgrind. * Use internal APIs.

view details

push time in 2 months

push eventtrevnorris/libuv

Trevor Norris

commit sha d869a15e58dbc1b3ccbd068e741eaca8c46db3fb

Move some mem management around, close test loops Forgot to properly close the uv_loop_t's used in tests. Use internal APIs.

view details

push time in 2 months

pull request commentlibuv/libuv

Add API for measuring event loop idle time

Recent changes:

  • Added uv__loop_internal_fields_t to store future expansions to uv_loop_t without breaking ABI. This idea was presented by @bnoordhuis in https://github.com/libuv/libuv/issues/2506#issuecomment-540050665. This started from the fact that uv_loop_t in Windows does not have the flags field.
  • Renamed a few variables and function names to make more sense.

Related tests pass on Windows and Linux. Last problem I'm facing is running the test with valgrind causes the metrics tests to fail. So far haven't been able to figure out why but will look further into it.

trevnorris

comment created time in 2 months

push eventtrevnorris/libuv

Trevor Norris

commit sha b87001b4ebbabdada02ca55ef440f4493c10d134

WIP Add internal fields to uv_loop_t and add stuff Create a struct and attach that to the uv_loop_t. Right now this only contains a pointer to uv__metrics_loop_t* and a new flags field. Reason this was added is to allow addition of other fields in the future. This idea was taken from a comment by bnoordhuis in: https://github.com/libuv/libuv/issues/2506#issuecomment-540050665 The change works around the fact that in Windows the uv_loop_t doesn't have a flags field.

view details

push time in 2 months

push eventtrevnorris/libuv

Trevor Norris

commit sha 04cf39e36fbf381ab9863b5dd87de0a546ee58b0

WIP Add internal fields to uv_loop_t and add stuff Create a struct and attach that to the uv_loop_t. Right now this only contains a pointer to uv__metrics_loop_t* and a new flags field. Reason this was added is to allow addition of other fields in the future. This idea was taken from a comment by bnoordhuis in: https://github.com/libuv/libuv/issues/2506#issuecomment-540050665 The change works around the fact that in Windows the uv_loop_t doesn't have a flags field.

view details

push time in 2 months

pull request commentlibuv/libuv

Add API for measuring event loop idle time

So, while I was testing on Windows I came to learn that Windows doesn't have a flags field on its uv_loop_t. I assume adding it to UV_LOOP_PRIVATE_FIELDS in include/uv/win.h will break ABI?

One possible solution is that I put whether it's enabled on the uv__metrics_loop_t. But it's not explicitly documented that you can't run uv_loop_configure on a uv_loop_t prior to running uv_loop_init. In that case I guess I could return some type of UV_E* error.

@saghul Any thoughts on this?

trevnorris

comment created time in 3 months

pull request commentlibuv/libuv

Add API for measuring event loop idle time

Here are the changes I've made:

  • Squashed all commits and rebased on latest v1.x
  • Removed using atomics and unified implementation using uv_mutex_t.
  • Added Windows support (thanks @bzoz for the tip)
  • Made it so the idle time APIs are always called. They'll return early if not configured to collect idle time metrics, but did this so the functions could be instrumented regardless (@mmarchini that sound good?).

Think it's almost ready. Would appreciate another round of reviews.

trevnorris

comment created time in 3 months

Pull request review commentlibuv/libuv

Add API for measuring event loop idle time

 void uv__io_poll(uv_loop_t* loop, int timeout) {   base = loop->time;   count = 48; /* Benchmarks suggest this gives the best throughput. */ +  if (loop->flags & UV_LOOP_IDLE_TIME) {+    user_timeout = timeout;+    timeout = 0;

Resolving this since the logic has since changed.

trevnorris

comment created time in 3 months

push eventtrevnorris/libuv

Trevor Norris

commit sha 7eb5555ea3697413e2c61f49993af37c69017ae4

Add API for measuring event loop idle time The API addition uv_metrics_idle_time() is a thread safe call that allows the user to retrieve the amount of time the event loop has spent in the kernel's event provider (i.e. poll). It was done this way to allow retrieving this value without needing to interrupt the execution of the event loop. This option can be enabled by passing UV_LOOP_IDLE_TIME to uv_loop_configure(). One important aspect of this change is, when enabled, to always first enter the event provider with a timeout of zero. Because if any events were waiting to be processed then the time spent in the kernel wasn't truly idle. While this does add a small amount of overhead if the event loop is mostly idle, this overhead disappears once the event loop is placed under stress because events would have been waiting regardless. The other additional overhead is two extra calls to uv_hrtime(). Once before and once after poll has been called. Checking whether the uv_loop_t is configured with UV_LOOP_IDLE_TIME always happens in uv__metrics_set_provider_entry_time() and uv__metrics_update_idle_time(). This makes the conditional logic wrapping each call simpler and allows for instrumentation to always hook into those two function calls. The value returned from uv_metrics_idle_time() is the total accumulation of idle time since the beginning of the event loop. This allows a single interface for users to do calculations over any desired period of time. Internally the struct uv__metrics_loop_t has been added as a location for future metrics enhancements. A pointer to this struct has been placed in the uv_loop_t::unused space to allow for fast retrieval and maintaining ABI compatibility with v1.x. Tests and additional documentation has been included.

view details

push time in 3 months

Pull request review commentlibuv/libuv

Add API for measuring event loop idle time

 void uv__io_poll(uv_loop_t* loop, int timeout) {   base = loop->time;   count = 48; /* Benchmarks suggest this gives the best throughput. */ +  if (loop->flags & UV_LOOP_IDLE_TIME) {+    user_timeout = timeout;+    timeout = 0;

@indutny I've redone the logic here. Using user_timeout == -2 to control the loop was a stupid way of preventing the need to use one additional variable. I've changed the implementation and hopefully made it more apparent. PTAL.

trevnorris

comment created time in 3 months

push eventtrevnorris/libuv

TK-one

commit sha e8644693ea2449360e2f06c8b5792e923ab4aeef

doc: fix header file location PR-URL: https://github.com/libuv/libuv/pull/2720 Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Saúl Ibarra Corretgé <s@saghul.net> Reviewed-By: Richard Lau <riclau@uk.ibm.com>

view details

Ben Noordhuis

commit sha 70469dcaa65bc4608e896b4df1ad13fcf58ad984

unix: fix signal handle closing deferral The way libuv handled closing of `uv_signal_t` handles with pending events introduced an infidelity where `uv_loop_alive()` returned false while the signal handle was still in the closing-but-not-closed state. Fixes: https://github.com/libuv/libuv/issues/2721 PR-URL: https://github.com/libuv/libuv/pull/2722 Reviewed-By: Santiago Gimeno <santiago.gimeno@gmail.com>

view details

Xu Meng

commit sha af7143b6f28ad60243cfa04d240deeec5d242e0f

ibmi: set the amount of memory in use to zero On IBMi PASE, the amount of memory in use includes storage used for memory and disks. So we hard-code the amount of memory in use to zero on IBMi, based on discussion in nodejs/node#32043. PR-URL: https://github.com/libuv/libuv/pull/2732 Refs: https://github.com/nodejs/node/pull/32043 Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl> Reviewed-By: Colin Ihrig <cjihrig@gmail.com>

view details

Milad Farazmand

commit sha b5155dd2accad94c86dc89ad43d79d30dd9c971a

zos: return on realloc failure in scandir() Fixes: https://github.com/libuv/libuv/issues/2692 PR-URL: https://github.com/libuv/libuv/pull/2693 Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl> Reviewed-By: Richard Lau <riclau@uk.ibm.com>

view details

Ben Noordhuis

commit sha a9974da019b5942a2f38bb6a170a42645afd966a

zos: fix scandir() error path NULL pointer deref Commit b5155dd2 ("zos: return on realloc failure in scandir()") introduced a bug where `nl` is dereferenced when it's NULL after reallocation fails. PR-URL: https://github.com/libuv/libuv/pull/2734 Refs: https://github.com/libuv/libuv/pull/2693 Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Richard Lau <riclau@uk.ibm.com>

view details

cjihrig

commit sha e45f1ec38db882f8dc17b51f51a6684027034609

2020.03.12, Version 1.35.0 (Stable) Changes since version 1.34.2: * src: android build fix (David Carlier) * build: make code compilable for iOS on Xcode (ssrlive) * ibmi: skip unsupported fs test cases (Xu Meng) * ibmi: ensure that pipe backlog is not zero (Xu Meng) * test,udp6: fix udp_ipv6 test flakiness (Jameson Nash) * test: fix fs_event_watch_dir_recursive flakiness (Santiago Gimeno) * pipe: disallow listening on an IPC pipe (Witold Kręcicki) * build,cmake: improve buil experience (Isabella Muerte) * unix: remove support for FreeBSD < 10 (Saúl Ibarra Corretgé) * linux: simplify uv__accept() (Ben Noordhuis) * linux: assume presence of SOCK_CLOEXEC flag (Ben Noordhuis) * linux: simplify uv__dup2_cloexec() (Ben Noordhuis) * freebsd,linux: simplify uv__make_socketpair() (Ben Noordhuis) * unix: fix error handling in uv__make_socketpair() (Ben Noordhuis) * freebsd,linux: simplify uv__make_pipe() (Ben Noordhuis) * unix: fix error handling in uv__make_pipe() (Ben Noordhuis) * linux: simplify uv__async_eventfd() (Ben Noordhuis) * linux: assume the presence of inotify system calls (Ben Noordhuis) * doc: strip ICC profile from 2 jpg files (Dominique Dumont) * unix: make uv_tcp_keepalive predictable (Manuel BACHMANN) * docs: uv_setup_args() may take ownership of argv (Ben Noordhuis) * unix: fix error path in uv_setup_args() (Ben Noordhuis) * unix: fix size check in uv_get_process_title() (Ben Noordhuis) * doc: add erw7 to maintainers (erw7) * test: fixed udp4_echo_server implementation (Marek Vavrusa) * test: added udp ping benchmark (1,10,100 pingers) (Marek Vavrusa) * freebsd,linux: add recvmmsg() + sendmmsg() udp implementation (Marek Vavrusa) * win,pipe: DRY/simplify some code paths (Jameson Nash) * win: address some style nits (Jameson Nash) * win,pipe: ensure `req->event_handle` is defined (Elliot Saba) * win,pipe: consolidate overlapped initialization (Elliot Saba) * win,pipe: erase event_handle after deleting pointer (Jameson Nash) * build: fix android cmake build, build missing file (Ben Noordhuis) * test: skip some UDP tests on IBMi (Xu Meng) * test: skip some spawn test cases on IBMi (Xu Meng) * src: fix wrong method name in comment (TK-one) * test: add UV_TIMEOUT_MULTIPLIER environment var (Ben Noordhuis) * unix: fix uv_cpu_info always returning UV_ENOTDIR on OpenBSD (Ben Davies) * test: skip the pwd_shell test on IBMi (Xu Meng) * win,tty: Change to restore cursor shape with uv_tty_reset() (erw7) * win,tty: Added set cursor style to CSI sequences (erw7) * test: handle EINTR, fix EOF check in poll test (Ben Noordhuis) * unix: use socklen_t instead of size_t (Ben Noordhuis) * doc: fix header file location (TK-one) * unix: fix signal handle closing deferral (Ben Noordhuis) * ibmi: set the amount of memory in use to zero (Xu Meng) * zos: return on realloc failure in scandir() (Milad Farazmand) * zos: fix scandir() error path NULL pointer deref (Ben Noordhuis)

view details

cjihrig

commit sha 87ee20d5128402e0f826941370954d302119d085

Add SHA to ChangeLog

view details

cjihrig

commit sha 584ba94885619d75968a7c083914fd8b75442813

Now working on version 1.35.1 Fixes: https://github.com/libuv/libuv/issues/2709

view details

Trevor Norris

commit sha bcf3900c192afd644399add710687b3bbd09aeb1

Add API for measuring event loop idle time The API addition uv_metrics_idle_time() is a thread safe call that allows the user to retrieve the amount of time the event loop has spent in the kernel's event provider (i.e. poll). It was done this way to allow retrieving this value without needing to interrupt the execution of the event loop. This option can be enabled by passing UV_LOOP_IDLE_TIME to uv_loop_configure(). One important aspect of this change is, when enabled, to always first enter the event provider with a timeout of zero. Because if any events were waiting to be processed then the time spent in the kernel wasn't truly idle. While this does add a small amount of overhead if the event loop is mostly idle, this overhead disappears once the event loop is placed under stress because events would have been waiting regardless. The other additional overhead is two extra calls to uv_hrtime(). Once before and once after poll has been called. The value returned from uv_metrics_idle_time() is the total accumulation of idle time since the beginning of the event loop. This allows a single interface for users to do calculations over any desired period of time. Internally the struct uv__metrics_loop_t has been added as a location for future metrics enhancements. A pointer to this struct has been placed in the uv_loop_t::unused space to allow for fast retrieval and maintaining ABI compatibility with v1.x. Tests and additional documentation has been included.

view details

push time in 3 months

push eventtrevnorris/libuv

push time in 3 months

pull request commentlibuv/libuv

WIP: Add API for measuring event loop idle time

Did initial Windows support (much thanks @bzoz for pointing that out). Still need to do some testing.

Also, the implementation of the internal metrics functions is almost identical. Except for the atomic operations. I'm going to benchmark the performance difference between using the atomics and the mutex. Depending on the difference, I might just drop the atomics part completely and unify the implementations.

trevnorris

comment created time in 3 months

push eventtrevnorris/libuv

Trevor Norris

commit sha 851880c5dde76873cc94c79ac1f8b95612a3f267

Move get_mloop to central implementation Retrieving the uv__metrics_loop_t* from uv_loop_t::unused is going to be the same across platforms.

view details

Trevor Norris

commit sha 4066cb31a701b71d0debc88184d41cd5be6c9b02

CHECKPOINT Implement Windows support Should be mostly implemented, but need to do further testing.

view details

push time in 3 months

pull request commentlibuv/libuv

WIP: Add API for measuring event loop idle time

Added a commit that will switch to using a uv_mutex_t if the GNU C atomic extension isn't available. Don't love it, but there's a better solution then I'll stick with this.

trevnorris

comment created time in 3 months

push eventtrevnorris/libuv

Trevor Norris

commit sha 1360bda5d3331e59bb2bb7b2d76f916410dea5bb

Move uv__metrics_loop_s impl to platform specifics There may be platform specific fields that need to be stored in uv__metrics_loop_s, so move the current one to unix.

view details

Trevor Norris

commit sha 8c6a37bdf924dc892d113e8fafeebe6d1234ff21

Don't forget to free resources on error I missed that uv_loop_init will return early if uv__platform_loop_init fails, without free'ing the memory allocated for the uv__metrics_loop_t. Create a new goto for that.

view details

Trevor Norris

commit sha 6051b39ae8be57040ea7e5457398c762b83a0e10

Use mutex when atomics aren't available Platform-agnostic atomics are probably a pipe dream. So only use GNU C atomics when available. Otherwise fallback to using a mutex.

view details

push time in 3 months

push eventtrevnorris/libuv

Trevor Norris

commit sha 676b84cb8d7b6619783de00957cc2f081ac6c74b

Add test to check timeout == 0 Make sure idle_time doesn't increase when the user's timeout is 0.

view details

Trevor Norris

commit sha 37e747deacba3f36b34743eb5e44f4087c4be30d

Normalize names of metrics provider_entry_time is specific. idle_time isn't, so rename it to provider_idle_time.

view details

push time in 3 months

push eventtrevnorris/libuv

Trevor Norris

commit sha 8b6dac5ddddf720dc12d3e0f351721ec3e19b0fa

Review fixes * Make sure timer callbacks have run in test * Code style fixes * Change struct to _s, then typedef to _t * Switch int to uint64_t to prevent compiler warning

view details

push time in 3 months

pull request commentlibuv/libuv

WIP: Add API for measuring event loop idle time

My worry about GetQueuedCompletionStatusEx is this from the documentation:

Retrieves multiple completion port entries simultaneously. It waits for pending I/O operations that are associated with the specified completion port to complete.

Which leads me to believe the event's callback is called before returning. This is a serious hindrance because it's then impossible to tell exactly when the event was received. I'll investigate the I/O Completion Port API further, but a pointer in the right direction would save me a lot of time.

trevnorris

comment created time in 3 months

pull request commentlibuv/libuv

WIP: Add API for measuring event loop idle time

Review suggestions have been addressed.

Is there anyone willing to give tips on the Windows implementation? The first question I need to answer is if GetQueuedCompletionStatusEx (used in uv_poll) is similar to all unix implementations. In that GetQueuedCompletionStatusEx returns when there are events to be processed, but it is still libuv's job to process those events. Or does GetQueuedCompletionStatusEx process the events for the user then return? I can't tell when the user's callback is supposed to be called.

The other main thing is to see if there are platform-agnostic atomics. I can switch to using a mutex lock if necessary, but I'd like to prevent that overhead if possible. If anything guess I could #ifdef and only use atomics on supported platforms and use a mutex on the other. @bnoordhuis have any suggestions here?

trevnorris

comment created time in 3 months

Pull request review commentlibuv/libuv

WIP: Add API for measuring event loop idle time

 void uv__io_poll(uv_loop_t* loop, int timeout) {   base = loop->time;   count = 48; /* Benchmarks suggest this gives the best throughput. */ +  if (loop->flags & UV_LOOP_IDLE_TIME) {+    user_timeout = timeout;+    timeout = 0;

@indutny Is this explanation clear? Since it's duplicated in all poll implementations I didn't add many inline comments. If you think that's necessary then I'll be happy to.

trevnorris

comment created time in 3 months

push eventtrevnorris/libuv

Trevor Norris

commit sha 7b3f82d0d30332814f7875b50a15b5792bfab6b7

Review fixes * Make sure timer callbacks have run in test * Code style fixes * Change struct to _s, then typedef to _t

view details

push time in 3 months

Pull request review commentlibuv/libuv

WIP: Add API for measuring event loop idle time

 int uv_loop_init(uv_loop_t* loop) {   memset(loop, 0, sizeof(*loop));   loop->data = saved_data; +  mloop = (uv__metrics_loop_t*) uv__calloc(1, sizeof(uv__metrics_loop_t));+  if (mloop == NULL)+    return UV_ENOMEM;+

From @bnoordhuis comment in https://github.com/libuv/libuv/issues/2506#issuecomment-540050665:

I think that can be addressed by malloc-ing memory.

Since that's basically what I'm doing with uv__metrics_loop_t, maybe it's not a bad time to create a struct that will hold all future would-be ABI changes. Something like uv__loop_internal_t?

@bnoordhuis @saghul Thoughts?

trevnorris

comment created time in 3 months

Pull request review commentlibuv/libuv

WIP: Add API for measuring event loop idle time

 void uv_sleep(unsigned int msec) {    assert(rc == 0); }+++void uv__metrics_update_idle_time(uv_loop_t* loop) {+  uv__metrics_loop_t* mloop;+  uint64_t entry_time;+  uint64_t idle_time;+  uint64_t exit_time;++  if (!(loop->flags & UV_LOOP_IDLE_TIME))+    return;++  mloop = uv__metrics_get_mloop(loop);++  /* The thread running uv__metrics_update_idle_time() is always the same+   * thread that sets provider_entry_time. So it's unnecessary to perform the+   * atomic operation retrieving this value.+   * Reason for this check is in case this function has been called a second+   * time. Since it's always called just after uv__io_poll() to make sure the+   * idle time has been correctly calculated.+   */+  if (mloop->provider_entry_time == 0)+    return;++  /* Grab the exit_time before starting the atomic operations. */+  exit_time = uv_hrtime();++  /* It's important to run the operations in this order. It will prevent+   * accidentally reporting incorrect values and removes the need for a lock.+   */+  UV_ATOMIC_EXCHANGE(&mloop->provider_entry_time, &entry_time, 0);+  UV_ATOMIC_ADD_FETCH(&mloop->idle_time, &idle_time, exit_time - entry_time);+}+++void uv__metrics_set_provider_entry_time(uv_loop_t* loop) {+  uv__metrics_loop_t* mloop;+  uint64_t now;++  if (!(loop->flags & UV_LOOP_IDLE_TIME))+    return;

But it is always called. To simplify the logic at each call site I placed the check in the function. Otherwise all 14 call locations would need to be wrapped in this same conditional.

trevnorris

comment created time in 3 months

Pull request review commentlibuv/libuv

WIP: Add API for measuring event loop idle time

 void uv__io_poll(uv_loop_t* loop, int timeout) {   base = loop->time;   count = 48; /* Benchmarks suggest this gives the best throughput. */ +  if (loop->flags & UV_LOOP_IDLE_TIME) {+    user_timeout = timeout;+    timeout = 0;

Agreed this code is confusing, but after many attempts also turned out to be the implementation that was the least invasive. I'll try to explain:

1 - Backup the user's timeout value into user_timeout. 2 - Set timeout = 0. 3 - Run epoll_wait with timeout == 0. 4 - When it returns, set timeout = user_timeout and set user_timeout = -2 to indicate that the first pass has already run. 5 - if nfds > 0 then: 5.1 - If an event is processed (meaning w->cb() is called) then end accumulating idle time 5.2 - If a signal is received, and the user has a signal watcher set, then end accumulating idle time 6 - If after processing all fd and nevents == 0 and the original timeout == -1 || timeout > 0 then reenter epoll_wait 7 - If after processing all fd and nevents > 0 then return immediately

Everything after step 4 happens anyway. So it just forces the first call to poll to return immediately to see if any events were waiting when poll was called (since there's no other way to determine if there were).

trevnorris

comment created time in 3 months

Pull request review commentlibuv/libuv

WIP: Add API for measuring event loop idle time

 void* uv__malloc(size_t size); void uv__free(void* ptr); void* uv__realloc(void* ptr, size_t size); +typedef struct {

I knew this was the case in include/uv.h but wasn't sure about the code in src/. So I took my example from uv__allocator_t in src/uv-common.cc. But after doing a quick grep I see that pattern is still usually the case here. I'll change it to follow that pattern.

trevnorris

comment created time in 3 months

more