@gadmm Thank you for asking. I should clarify what I meant when I wrote "reasonable interval." I have no experience; we decided to move away from signals instead of using systhreads. My understanding from reading the code and related issues was that the interval is supposed to be reasonable.

Summary: Before this change, writing to shared memory looked like this:

1. Atomically write the 8-byte hash into the hash field
2. Atomically write HASHTBL_WRITE_IN_PROGRESS into the addr field
3. Serialize the OCaml data, compress, and write into heap
4. Non-atomically write the heap addr from (3) into the addr field

If a reader observes the HASHTBL_WRITE_IN_PROGRESS, it goes into a busy-wait loop, assuming that step (3) will be done soon.

However, if the writer crashes between steps 2 and 4, the hash table will contain the HASHTBL_WRITE_IN_PROGRESS value in the addr field permanently, meaning the busy-waiting readers will busy-wait forever. To mitigate this, there is a 60 second timeout.

In pratice, we do observe writers crashing and readers timing out after 60 seconds. A likely cause of such crashes is OOMs, but since there is quite a lot of code on the write path it could be a variety of things.

This diff avoids this issue entirely by changing the write path to instead:

1. Serialize the OCaml data, compress, and write to the heap
2. Atomically grab the entire 16-byte slot with the hash and known addr

Note that there is no HASHTBL_WRITE_IN_PROGRESS value. If the slot is taken, it's address is know. Thus, readers can avoid busy-waiting entirely.

The downside of this approach is that writers may allocate space on the heap, but then lose a race and fail to write to the hash table. In this case, the data written immediately becomes garbage.

While the sharedmem GC will clean this up eventually, it's also important to realize that Flow's workload does not actually race to write values in practice, and so will not encounter this case at all.

Differential Revision: D22353111

Fixed in 48355d9935060ea725689108ec96d6bd44bee8ee. Thanks for the report!