profile
viewpoint

ocsigen/lwt 544

OCaml promises and concurrent I/O

raphael-proust/cps-ssa 18

Experiments about intermediate representations in compilers

raphael-proust/d3 10

js_of_ocaml bindings for a JavaScript visualization library for HTML and SVG.

avsm/ocaml-ld 7

Dynamic reconfigurable linker for OCaml

raphael-proust/1-3-person 2

A few games with an unusually personal perspective.

raphael-proust/deriving 2

Extension to OCaml for deriving functions from type declarations.

raphael-proust/LC-Compiler 1

A small interpreter for a toy language

raphael-proust/abs 0

personal ABS (Arch Build System) repository

push eventocsigen/lwt

Raphaël Proust

commit sha c32728eda3f0198eedfc399efeb06489094800d3

Regenerate 5.5.0 Lwt_unix doc There seem to have been a mix up of some sort and the latest changes were not included.

view details

push time in 2 days

push eventraphael-proust/lwt

Raphaël Proust

commit sha 94c321376d8120a1c9de30c782730a5a3f987fee

Mention `Lwt_seq` in the doc's menu

view details

push time in 2 days

push eventocsigen/lwt

Raphaël Proust

commit sha 5586f058907324a91e07e942fc2686af7f845759

Fix docs directory structure

view details

push time in 2 days

push eventocsigen/lwt

Raphaël Proust

commit sha 7692c3a81d75e280114df1ead99153355fe2e515

Fix 5.4.x and 5.5.0 documentation

view details

push time in 2 days

push eventraphael-proust/lwt

Raphaël Proust

commit sha c134c8cb4e5bfd6ddf5cafd61fc3205187362d08

List Lwt_domain files when generating the API doc

view details

push time in 2 days

push eventocsigen/lwt

Raphaël Proust

commit sha 71dd0a74427332450e2800140030f9e03bdeff47

API doc for 5.4.x and 5.5.0

view details

push time in 2 days

PR opened ocsigen/lwt

small documentation additions
+42 -3

0 comment

3 changed files

pr created time in 2 days

create barnchraphael-proust/lwt

branch : raphael-p@small-documentation-additions

created branch time in 2 days

release ocsigen/lwt

5.5.0

released time in 5 days

push eventocsigen/lwt

Raphaël Proust

commit sha 8fa5d3d2581d6a9c8740c39c90cb04863ef4db83

Fixups for CHANGES

view details

Raphaël Proust

commit sha bab52d9744cb2d5cd3cfe86cda65ba73752998ee

Update version numbers in opam files

view details

Raphaël Proust

commit sha 8cf56b8a60d0d18e0e0cf446dca09532fab299a1

Merge pull request #904 from ocsigen/prepare-for-release-5-5-0 Prepare for release 5 5 0

view details

push time in 5 days

PR merged ocsigen/lwt

Prepare for release 5 5 0

This branch is intended to prepare the 5.5.0. It may receive some commits to fix packaging and such.

Once 5.5.0 is released, this PR can be merged.

+31 -27

1 comment

6 changed files

raphael-proust

pr closed time in 5 days

pull request commentocaml/opam-repository

5 packages from ocsigen/lwt

A force-push might be a good idea to trigger a rebuild of everything...

I've pushed an empty commit to do that.

raphael-proust

comment created time in 5 days

push eventraphael-proust/opam-repository

Raphaël Proust

commit sha 9aa93249b802353e9d56fdd0925bbcfc4e25e823

Trigger the CI again with an empty commit

view details

push time in 5 days

pull request commentocaml/opam-repository

5 packages from ocsigen/lwt

There are a lot of CI failures that seem to be unrelated to the Lwt code:

2021-11-25 13:08.56: Got resource from pool OCluster
Building on hipp.ocamllabs.io
fatal: Unable to create '/var/cache/obuilder/ocluster/git/opam-repository.git-b322f7b18d276db68487afc3dfc5a5befdd45818/.git/index.lock': File exists.

Another git process seems to be running in this repository, e.g.
an editor opened by 'git commit'. Please make sure all processes
are terminated then try again. If it still fails, a git process
may have crashed in this repository earlier:
remove the file manually to continue.
git-reset failed with exit-code 128
2021-11-25 13:08.57: Job failed: Failed: Build failed

I'm not sure if its due to packaging? To the fact that this PR publishes several packages? Or to an unrelated OCluster bug?

raphael-proust

comment created time in 6 days

PR opened ocaml/opam-repository

5 packages from ocsigen/lwt

This pull-request concerns: -lwt.5.5.0: Promises and event-driven I/O -lwt_domain.0.1.0: Helpers for using Domainslib with Lwt -lwt_ppx.2.0.3: PPX syntax for Lwt, providing something similar to async/await from JavaScript -lwt_ppx_let.5.5.0: Dummy package context for ppx_let tests -lwt_react.1.1.5: Helpers for using React with Lwt


  • Homepage: https://github.com/ocsigen/lwt
  • Source repo: git+https://github.com/ocsigen/lwt.git
  • Bug tracker: https://github.com/ocsigen/lwt/issues

:camel: Pull-request generated by opam-publish v2.1.0

+196 -0

0 comment

5 changed files

pr created time in 6 days

created tagocsigen/lwt

tag5.5.0

OCaml promises and concurrent I/O

created time in 6 days

pull request commentocsigen/lwt

Prepare for release 5 5 0

Ths CI errors are transient network errors.

raphael-proust

comment created time in 6 days

PR opened ocsigen/lwt

Prepare for release 5 5 0

This branch is intended to prepare the 5.5.0. It may receive some commits to fix packaging and such.

Once 5.5.0 is released, this PR can be merged.

+31 -27

0 comment

6 changed files

pr created time in 6 days

create barnchocsigen/lwt

branch : prepare-for-release-5-5-0

created branch time in 6 days

push eventocsigen/lwt

Sudha Parimala

commit sha ecd48c4142ebde577f3113adaec8196722030617

Add Lwt_domain

view details

Sudha Parimala

commit sha 2f1def5fff31c1cb2e068766410082ee3591fefa

port to domainslib task pool

view details

Sudha Parimala

commit sha 6aad4ad894e4ed89c211bb5a1a2aca6e72651ab0

bump domainslib version; remove unused params

view details

Sudha Parimala

commit sha b493a9792888639300401480fd3bf838200e5beb

add set_num_domains to control the number of domains

view details

Sudha Parimala

commit sha 158ccf27bed4a868ae24da9a64bfedaf68e835e2

Lwt_domain doc update

view details

Raphaël Proust

commit sha fee78f15946bf8df7a5507e543a86a45083447d8

Remove mentions of "threads" from the domain doc, misc tweaks

view details

Raphaël Proust

commit sha fdb9b210a01696a87e52e7c0d752a8e2c364e223

Small fix in domain documentation

view details

Sudha Parimala

commit sha 987453789216d1a08182fd550a668bc71753bf4a

raise Invalid_argument when number of domains less than 1; add tests for Invalid_argument & rejection when detach raises an exception

view details

Sudha Parimala

commit sha 280df9bcdd1258e6de656016dc4b22ea1ae36863

special case for one domain

view details

Sudha Parimala

commit sha 4222f6f2338e6a4a3532cf037a59d9df23d852ff

setup and teardown pool explicitly

view details

Raphaël Proust

commit sha 2d64c2120f02e21f7c2f415471a75a86b8c65504

Some more documentation for domains

view details

Raphaël Proust

commit sha 150c55dc00f2971c5751e5c4a0e94ca0f8064a08

simplify 1-domain detach

view details

Sudha Parimala

commit sha 6d42991de18321a988badcdbbfda3d7ed9ca366a

Take pool as a parameter and rebase to domainslib 0.3.1

view details

Sudha Parimala

commit sha 3e204c2d2aa4b710e30a2999420431ee4b276d3e

Apply suggestions from code review Co-authored-by: ygrek <ygrek@autistici.org>

view details

Sudha Parimala

commit sha 551b95771f1993c6653b9848891cf9777c6b7a25

Apply suggestions from code review Co-authored-by: yawaramin <yawar.amin@gmail.com>

view details

Raphaël Proust

commit sha e24393cfdc42adf2dad023e70cef6f73c9910490

Move lwt-domain in its own package

view details

Raphaël Proust

commit sha faf882f55fea9320cf0be7bf6d80e1bd05853e5a

Unix: remove leftover dependency to domainslib

view details

Raphaël Proust

commit sha f009593d54607f0f054c54f23585fd9af1051464

constrain domainslib depenedncy version for lwt_domain

view details

Raphaël Proust

commit sha 43db01e57df6dd171b27a722c43506ff2570ebc2

fix package field of dune file in domain's test

view details

Raphaël Proust

commit sha 1d2abe2b93d51ded9feb955984eb4f127a76c0b8

Update workflow to special-case lwt_domain

view details

push time in 6 days

PR merged ocsigen/lwt

Reviewers
Lwt_domain: an interface to multicore parallelism

Synopsis: The PR aims to add a module Lwt_domain that enables offloading computations to many CPU cores via Multicore's Domains.

Lwt_domain interface

This patch aims to add a new module Lwt_domain. The API is similar to Lwt_preemptive, the major difference being Lwt_preemptive runs compuatations in a new systhread and Lwt_domain computations in a new domain.

In effect, the Lwt_domain runs OCaml computations in parallel on multiple CPU cores, which is not the case with Lwt_preemptive. This is because system threads are run on a single CPU core despite being run concurrently. On the other hand, multicore domains run on the available cores.

Implementation

A pool of domains is created by the simple_init function, with the default number of domains being 4. The created domains will be alive and waiting for tasks till the time main loop is alive. The number of live domains can be adjusted via the set_bounds function.

Unlike Lwt_preemptive which creates new systhreads frequently, all live domains are reused here because repeatedly creating and shutting down new domains is an expensive process, which could be avoided.

When detach function is called it obtains a new worker from the pool of domains and the function is executed in that worker domain. After the execution is done, the worker domain will again be ready to take tasks.

A separate queue of jobs is maintained that are supposed to be executed on the main domain, being called from run_in_main.

Benchmark

A simple server that returns Fibonacci number of a given input was ported to run in parallel. The server and client code can be found here. The benchmark was run with 10 clients and 10 requests per client, it's time is shown below. Individual timings can be found in the blog post mentioned below.

Time graph of parallel Fibonacci server

image

Side note

Unix.fork is not compatible with multiple domains. Hence, the test-suite is modified to run sequentially and all the tests that run Unix.fork are run before the Lwt_domain test.


This module is intended to work with OCaml 5.0 with parallelism support. It can be used now with Multicore OCaml obtained from multicore-opam. Any suggestions/feedback welcome, benchmarks from potential applications of this module will be appreciated too!

Further reading

+358 -12

18 comments

9 changed files

Sudha247

pr closed time in 6 days

pull request commentocsigen/lwt

Lwt_domain: an interface to multicore parallelism

rebased on master to resolve merge conflict

Sudha247

comment created time in 6 days

push eventSudha247/lwt

Antonin Décimo

commit sha dad167fc5b73f19d7830fce988730494aef398e3

Support Unix.socketpair using PF_UNIX on Windows and OCaml 4.14 Support for Unix domain sockets on Windows and a socketpair emulation using PF_UNIX was added in OCaml 4.14 in PR 10192.

view details

Antonin Décimo

commit sha 1379208a448d17b0bd24f52ecbf3bc466b69c964

Add ?to_dir param from Unix.symlink to Lwt_unix.symlink wrapper

view details

Raphaël Proust

commit sha d1ae19c9b923405ce0228ba84c77bc820472abe8

Core: remove some `mutable` to fix warning

view details

Raphaël Proust

commit sha 242465351175191030a586a50d38ccc66ca054e1

Core: annotate seamingly-unused (but useful) field to suppress warning

view details

Jérôme Vouillon

commit sha 4b5ef74baeab758f8ec2102b9c1ea505790984ee

Lwt_react: no longer compare structurally the arguments of S.l[2-6]_s This avoids a failure when some inputs cannot be compared structurally (if they contain functional values, for instance). Internally, Lwt_react.S.l2 is implemented using React.S.l2 and Lwt_react.S.map_s. We were not providing an equality function to React.S.l2, so the default structural equality was used. Now, we use the function (fun _ _ -> false). So, the function parameter is applied each time any of the input change, which is exactly what we would expect if we implemented Lwt_react.S.map_s directly.

view details

Raphaël Proust

commit sha caa590f47e1ace39e3ae89b03ae2a2ee60d0025b

Document newest react safety fix

view details

Raphaël Proust

commit sha b88ed831f6528bd5eed69a9965ba854c8be354fb

Merge pull request #893 from ocsigen/react-fix Lwt_react: no longer compare structurally the inputs of S.l[2-6]_s

view details

Sora Morimoto

commit sha af835c582bb1b37719246080be41d067301fcf40

Merge pull request #899 from raphael-proust/fix-some-warnings Core: remove some `mutable` to fix warning

view details

Sora Morimoto

commit sha 3dfe4d8682efaabd2c4d2f1e3425574b0e7ef0cc

Merge pull request #884 from MisterDA/symlink-to-dir Add ?to_dir param from Unix.symlink to Lwt_unix.symlink wrapper

view details

Raphaël Proust

commit sha 969d73549ac0c3ecaadfffe8cca5ef26fe9b7408

Lwt_stream.of_lwt_seq: 'a Lwt_seq.t -> 'a Lwt_stream.t

view details

Antonin Décimo

commit sha e902075c612e87db16d0e89299419df189d04696

Support deleting symlinks in Lwt_io.with_temp_dir on Windows If a Windows symlink points to a directory, then rmdir is the correct function to call. If it points to a file, then unlink is the correct function. When pointing to a file, the symlink can also have the read-only attribute, which makes it subject of the little chmod dance.

view details

Sora Morimoto

commit sha 47dd9e51b7c9fa21af5e0cf0ed151e417846498b

Merge pull request #870 from MisterDA/socketpair-windows Support `Unix.socketpair` using `PF_UNIX` on Windows and OCaml 4.14

view details

Sora Morimoto

commit sha 99c5635ec282bda8029adedbcfb8b91a4fc71dd2

Merge pull request #886 from MisterDA/with-temp-dir-symlinks Support deleting symlinks in Lwt_io.with_temp_dir on Windows

view details

Antonin Décimo

commit sha 1e1771631d00e5cf8749388fd9ee5619d653458e

Treat ERROR_BROKEN_PIPE on read as zero-read instead of error This matches the behaviour of win32unix since 3.11.1, see OCaml PR 4790.

view details

Sora Morimoto

commit sha 44074376c67614f62a6add279a8feb8309af0602

Merge pull request #898 from MisterDA/windows-ERROR_BROKEN_PIPE-zero-read Treat ERROR_BROKEN_PIPE on read as zero-read instead of error

view details

Antonin Décimo

commit sha 564363cdfc6aae6cf3453c93f462c7d538049fb5

Use is_blocking in dup and dup2 to fix ENOTSOCK on Windows On Windows, calling Unix.clear_non_block and Unix.set_non_block is only supported on socket file descriptors. The function is_blocking already deals with the corner cases, but the current implementations of dup and dup2 don't make use of it. Thus, when then dup or dup2 wrappers are called with file socket descriptors, ENOTSOCK errors are raised. The fix is to reuse is_blocking in the wrappers. Signed-off-by: Antonin Décimo <antonin@tarides.com>

view details

Sora Morimoto

commit sha ad0f98247cbe9674c17b0a85028a4c324d7427ae

Merge pull request #869 from MisterDA/windows-dup-dup2-is-blocking-fix Use `is_blocking` in `dup` and `dup2` to fix `ENOTSOCK` on Windows

view details

Antonin Décimo

commit sha 4f2d04555cfb1bff25341f5d5d8b44194768c059

Use Windows Unicode API in C stubs, require OCaml >= 4.06

view details

Antonin Décimo

commit sha 65a47a579eb69153ddd4504a9e75223704bbd3e2

Amend Windows socketpair changes entry

view details

Sora Morimoto

commit sha 23ead070635232ba077a8cf50e1e067b805813bb

Merge pull request #900 from MisterDA/changes-entry-870 Amend Windows socketpair changes entry

view details

push time in 6 days

PullRequestReviewEvent

pull request commentocsigen/lwt

Add `?cloexec:bool` argument to wrapped `Unix` functions in `Lwt_unix`

This looks ok to me as well.

MisterDA

comment created time in 12 days

push eventSudha247/lwt

Raphaël Proust

commit sha 0b4af095f979daf33ace8fb76e4721669632f3b6

mention Lwt_domain in CHANGES

view details

push time in 13 days

Pull request review commentocsigen/lwt

Lwt_domain: an interface to multicore parallelism

 open Tester  let () =-  Test.concurrent "unix" [-    Test_lwt_unix.suite;+  Test.run "unix" [

We need to revert this. The tests can now work concurrently.

Sudha247

comment created time in 13 days

PullRequestReviewEvent

Pull request review commentocsigen/lwt

Lwt_domain: an interface to multicore parallelism

 jobs:       - run: opam depext conf-libev --install         if: ${{ matrix.libev == true }} -      - run: opam install . --deps-only --with-test-        if: ${{ matrix.ppx == true }}-       - run: opam install lwt_luv lwt_react lwt --deps-only --with-test-        if: ${{ matrix.ppx == false }} -      - run: opam exec -- make build+      - run: opam install lwt_ppx --deps-only --with-test         if: ${{ matrix.ppx == true }} +      - run: opam install lwt_domain --deps-only --with-test+        if: ${{ matrix.domain == true }}+

This pattern (default action for all workflows, ppx action if ppx=true, domain action if domain=true) is repeated below for building and then for testing. It is somewhat verbose and not very scalable (at some point we might want to add a unix flag).

It might be better to use the opam-local-packages variable. Somewhat annoyingly, opam and dune expect the package list in different formats so it's not completely trivial. Still we could possibly simplify this file.

Opinions? Ideas?

ping @smorimoto who wrote this file.

Sudha247

comment created time in 13 days

more