I think, in the worst case, you should be able to define the function using higher-level helpers like on_cancel. I could be wrong. But if that's correct, I suggest to go that route rather than introducing a new internal constructor.

Thanks @dra27.

I agree. We should aim to change Lwt to use the -W functions on Windows, as you suggest, in the long term (if this isn't superseded by libuv conversion or some other such work).

For right now, I think the best thing to do is cast, and add a comment that this is only valid because we are calling CreateProcessA, perhaps with links to this PR, the blog post, and/or the MSDN entry. Or we can change this specific code to call CreateProcessW, as in the code @dra27 linked.

The documentation Lwt_unix.fork claims that

in the child process all pending jobs are canceled

However it seems it is not entirely the case, as demonstrated by the example bellow. In this example I fork processes in a parallel iteration over a list. I expect that only the main process forks new jobs, however some forked processes do fork again while they obviously shouldn't.

With sequential iteration on the list, the problematic behavior does not happen.

Warning: Don't increase the size of the list, it rapidly leads to a fork bomb that exhausts available pids which crashes the system.

Tested with Lwt 4.2.1(ocaml 4.07.1) and 5.3.0 (ocaml 4.10.0).

open Lwt

let ( -- ) i j = List.init (j - i + 1) (fun x -> x + i)

let main_pid = Unix.getpid ()

let fork_one i =
  Lwt_io.flush_all () >>= fun () ->
  if Unix.getpid () <> main_pid then
    Format.printf
      "Error: Forking from non-main process %d (%d in the list) child of %s@."
      (Unix.getpid ())
      i
      (let ppid = Unix.getppid () in
       if ppid = main_pid then "main" else string_of_int ppid) ;
  match Lwt_unix.fork () with
  | 0 -> Lwt_unix.sleep 0.1 >>= fun _ -> exit 0
  | pid ->
      Format.printf "%d forked from %d@." pid (Unix.getpid ()) ;
      Lwt_unix.waitpid [] pid >>= fun _ -> Lwt.return_unit

let fork_many i = Lwt_list.map_p fork_one (0 -- i)

let _ =
  Format.printf "Main process pid is %d@." (Unix.getpid ()) ;
  Lwt_main.run (fork_many 3)

I'm trying to leverage dune toplevel integration (docs: https://dune.readthedocs.io/en/stable/toplevel-integration.html), my project is using lwt, and somewhere in the middle of generated output from dune top there is the following line:

#load "/home/vagrant/git/ocaml/my-project/_opam/lib/lwt/unix/lwt_unix.cma";

Both utop and rtop hang on this. I've launched rtop in gdb, and hitting ctrl+c gives the following backtrace after rtop "hangs":

Reason # #use_output "cat foo.ml";
[Detaching after fork from child process 5513]
^C
Thread 1 "ocamlrun" received signal SIGINT, Interrupt.
0x00007ffff7d507ef in epoll_wait (epfd=3, events=0x5555555d3de0, maxevents=64, timeout=59743) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30                                  
30      ../sysdeps/unix/sysv/linux/epoll_wait.c: No such file or directory.
(gdb) bt
#0  0x00007ffff7d507ef in epoll_wait (epfd=3, events=0x5555555d3de0, maxevents=64, timeout=59743) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30                              
#1  0x00007ffff70cd873 in ?? () from /lib/x86_64-linux-gnu/libev.so.4
#2  0x00007ffff70cfcde in ev_run () from /lib/x86_64-linux-gnu/libev.so.4
#3  0x00007ffff70eced8 in ev_loop (flags=2, loop=0x5555555d3870) at /usr/include/ev.h:836                                                                                    
#4  lwt_libev_loop (val_loop=<optimized out>, val_block=<optimized out>) at lwt_libev_stubs.c:102                                                                            
#5  0x0000555555583222 in caml_interprete (prog=0x7ffff710f010, prog_size=<optimized out>) at interp.c:908                                                                   
#6  0x0000555555585ac2 in caml_main (argv=0x7fffffffde78) at startup_byt.c:448
#7  0x00005555555686cc in main (argc=<optimized out>, argv=<optimized out>) at main.c:44                                                                                     

So it seems that it started the event loop. That does not seem to be the intended behavior :)

MSDN has:

The Unicode version of this function, CreateProcessW, can modify the contents of this string. Therefore, this parameter cannot be a pointer to read-only memory (such as a const variable or a literal string). If this parameter is a constant string, the function may cause an access violation.

Given the typical representation of strings in OCaml, do we ever end up in an environment where CreateProcess is CreateProcessW? I guess this is a question for anyone watching the repo. @dra27?

MSDN is silent on whether either CreateProcess function can write into lpEnvironment.

This should append cancelable_tests, not protected_tests.

What causes p to become canceled here? IIRC protected should prevent propagation of cancellation from p' to p.

Thanks!

IMO it's not worth it.

[On adding a new function] I agree.

I added a note in the CHANGES file, not sure if that's how you want to keep track of it. If not let me know and I can remove that part.

I usually write/rewrite the changelog right before release, so this is fine :)

How strongly should I encourage people to use pause rather than yield? Do we want to deprecate yield altogether or just point at pause?

It would probably be too noisy to attach a deprecation annotation to it, so I think the way you have the docs recommend pause in this PR is best.

Two relatively simple modifications. (Albeit one is simple but not trivial.)

The first commit implements Lwt_main.yield as a call to Lwt.pause and removes all the support code associated with the previous implementation. This is not trivial and may have subtle effects on the scheduling performed in Lwt_main.run.

The second commit simply avoids some List.rev calls when traversing lists (iter_p, iteri_p, etc.).

Consider the following document:

<!doctype html>
<html>
  <head>
    <img>
  </head>
  <body class='cls'>
  </body>
</html>

The browser parses this as containing a body with class cls. (JSFiddle)

When using Markup.ml the attributes on the body tag are lost because a replacement body tag seems to be generated because of the img in the head.

This is unfortunate, because in my use case the body element has structured data attributes that I want to extract.

I've added an incomplete start to a test case on my branch body-attrs-after-bad-head, but I don't really know how to resolve this without losing the desirable semantics of generating a body tag if it is missing altogether.

Hope you have some ideas. Thanks for your time.

The Markup.ml part of this was addressed by @apeschar in #55. Thanks again!

Thanks!

I didn't add the attributes to the error message, which is confusing when different errors are compared in the tests (the messages are the same).

That's probably fine for now. If it becomes confusing, I would suggest

misnested tag: 'body' in 'body'
misnested tag: 'body' with 1 attribute in 'body'
misnested tag: 'body' with 2 attributes in 'body'

to keep the error message length reasonable while still calling the user's attention to what is the likely source of difference in comparison.

The helper saves a lot of typing and sometimes an extra variable (name).

I didn't add the attributes to the error message, which is confusing when different errors are compared in the tests (the messages are the same).

This could work?

misnested tag: 'body' in 'body' with attributes 'class' = 'whatever', 'id' = 'something'

we could extend markup.ml to include attribute data in the error, and Lambda Soup to propagate attributes on misnested body elements to the generated body element using ~report.

That seems right.

We can define `Misnested_tag of string * string * (string * string list), or `Misnested_tag of Token_tag.t * string.

At least at first glance, I prefer the former, for two reasons:

• To avoid exposing an extra type or type abbreviation in the interface.
• Token_tag.t, in particular, has an additional self_closing field, and may have other fields that are not meant for the user to depend on. In general, it's an internal type and I'd like to retain the ability to change it at will, in principle.

As for the syntax in the source code of Markup.ml itself, it seems that we can get the shorter syntax by just defining a helper that will take the right parts out of t anyway, though I probably would first just use patterns like in your second example, until I became convinced that it is too annoying to do it that way :)

This is caused by a limitation of Markup.ml's design, namely that Markup.ml is a streaming parser (it deliberately does not parse the whole document before emitting signals). In contrast, HTML5 specifies a non-streaming parser that has access to the entire DOM tree throughout parsing, and can make non-local modifications to elements based on input that occurs much later than the input which created the elements. The conversion to streaming is mostly faithful, but there are a few cases (I recall three) where a correct conversion is impossible, all because they would inherently require reading the whole document in order to emit a given element correctly:

1. Nested (explicitly or due to error recovery) <body> elements, as in this issue. The spec requires

   for each attribute on the token, check to see if the attribute is already present on the body element (the second element) on the stack of open elements, and if it is not, add the attribute and its corresponding value to that element.

2. Nested <html> elements, for the same kind of reason.

3. Certain bad <table> inputs.

This is described with much less detail in Conformance status.

My idea for addressing this was to change the parser to emit richer nesting errors, so that a user who is building a DOM can implement the non-local correction themselves. It is already possible to observe the errors emitted by the parser by using the ?report argument of parse_html:

Markup.(parse_html
  ?report:(fun location e -> prerr_endline (Error.to_string ~location e))
  input

I think Error.t can be extended to carry attribute information, which can then be included when reporting the nested body tag here:

https://github.com/aantron/markup.ml/blob/6d082308caa9ef01894e5d2d0485b1d16b58f2be/src/html_parser.ml#L1647-L1648

I'd probably also add a helper that can implement the spec's check

for each attribute on the token, check to see if the attribute is already present on the body element (the second element) on the stack of open elements, and if it is not, add the attribute and its corresponding value to that element.

by walking along two attribute lists, but maybe this isn't so useful, as it depends on the user's DOM representation. At least, I would document what should be done in this case according to the spec (which the user can choose to ignore if they want to).

Thanks! I am happy to leave them in and not deprecate them.

Ok, convinced :) I will merge a PR doing this.

Thanks.

Thinking about the various effects of this, it occurs to me that we might break some users who expect Result or Seq to be transitively pulled in, if they start compiling on high-enough versions of OCaml.

The conditional Dune file is fine, since it will already be conditional in Lwt until the minimum supported OCaml version is 4.08, and the branch only adds conditions on 4.03 and 4.07.

LGTM. I suppose we should wait for https://github.com/savonet/ocaml-ssl/pull/63 and a release of ssl before merging this, so we can add the right constraint here. If you agree, please ping this PR when ssl is released.

What's the motivation for using (implicit_transitive_deps false) in Mirage?

I see that this makes Bisect_ppx transitively depend on Base. From discussions with @jeremiedimino and @rgrinberg, I got the impression that Base would eventually not be a dependency of ppxlib or other future PPX helper library. Is that still the case? I strongly prefer not to introduce Base as a dependency of Bisect_ppx.

BuckleScript support should work with this approach, as for BuckleScript support, we build the Bisect_ppx binaries using the opam ecoystem (though using esy as the package manager). However, it is broken in this PR at the moment: https://travis-ci.org/github/aantron/bisect_ppx/jobs/710719501.

Travis seems to not be reporting status to GitHub (a problem they have been having on and off for months now). The builds are running, however. They are at https://travis-ci.org/github/aantron/bisect_ppx.

I won't have time to properly review this until probably the next libuv release, but I do want to include it. Would you mind letting the 0.5.4 release proceed? I wasn't explicitly trying to support FreeBSD as of 0.5.4, but I will support it in future releases.

Thanks!

Missed the S_IFREG constant in https://github.com/aantron/luv/pull/69 - this adds it, so that Luv.File.stat can be used to check if the stat result represents a file.

Also, the Windows build is failing. It's not due to any of the warnings in the log (those are caused by Ctypes and are not fatal). My first guess is it is due to gmake perhaps not being available in typical Cygwin environments, such as the one used by opam-repository-mingw and the one distributed by esy. I'm not sure if that's properly a Dune issue (of which command it chooses to run), but I couldn't release Luv with a newly-broken Windows build in any case.

I am using Travis, for fast feedback. Travis is probably just having issues reporting status at the moment, which it seems to have had several times over the last few months, or it's something else I have to look at again. Could you restore the previous Travis rows?

• Upgrade libuv to 1.38.1 (aantron/luv#71).
• Bind S_IFMT, S_IFDIR, S_IFBLK, S_IFCHR, S_IFLNK, S_IFIFO (aantron/luv#69, Bryan Phelps).

See https://github.com/libuv/libuv/releases/tag/v1.38.1.

Done in 2b8b37d43e472247e9c1963e63759689eae91299.

Yes, this is the correct behavior according to the spec :)

The spec asks a compliant parser to sometimes produce outputs that may be counterintuitive. That's the result of trade-offs that were made in order to intuitively parse certain other malformed inputs that the authors of HTML5 found to be more common or otherwise important. This issue is, unfortunately, one of the cases that got "penalized" by these trade-offs.

See https://github.com/libuv/libuv/releases/tag/v1.38.1.

I would be happy to review a fix. The only thing to be careful of is that it complies with the HTML5 spec. Here are some relevant pieces:

• https://www.w3.org/TR/html52/syntax.html#attribute-value-double-quoted-state
• https://www.w3.org/TR/html52/syntax.html#the-in-body-insertion-mode

Are you sure that it is the newline inside the href specifically that is causing the problem? There is also a newline at the end of your input. Is that newline present in both of your tests?

My first guess as to what is happening here is that the unclosed <a> spanning the end of <p> is triggering the "reconstruct the active formatting elements" algorithm (called on by the in body insertion mode rules I linked to above). That causes formatting elements like <a> to be duplicated in multiple block elements like <p> or <body> if the formatting elements are unclosed, and there is any more input, like that final newline.

If that's correct, then this result is according to the spec (i.e. Chrome and other browsers would produce the same interpretation). Have you tried your inputs in a browser, inspecting the resulting DOMs?

You can also try parse5 online on this input: https://astexplorer.net/#/gist/e4a59515f46e8d7a1ae7ea5286730c30/341f1f298e3ce58e377ff13f497424c36fce818d. If you expand the AST, you will see that parse5 produces the same output, and the "duplication" depends on the final newline. The choice of parse5 is important, because most other JS HTML parsers (all that I have checked) don't actually comply with HTML5, and only parse an HTML-like input language without the full error recovery specified in HTML5.

@hannesm It is currently waiting for the release of https://github.com/ocaml/dune/pull/3535 :)

Thanks!

This is most likely caused by utop. As I understand it, it has some special integration with Lwt_main. In addition, it might use Lwt_main internally for its own purposes. I'm not immediately familiar with the details, so I suggest to report the issue at utop (and please cc :)).

This example might help with adapting Cobertura to OCaml.

For future reference, here is example Cobertura output and the DTD. It seems considerably more complex that Coveralls format, and seems to make more assumptions about the source language.

let ___bisect_matched_value___ = x in
match x with ...

This would have to be

let ___bisect_matched_value___ = x in
match ___bisect_matched_value___ with ...

to avoid repeating side effects caused by evaluation of x. However, it still doesn't quite work, because x might raise exceptions that are intended to be handled in exception cases of the match.

There may be a solution like

match x with
| exception ___bisect_matched_value___ ->
  (match ___bisect_matched_value with
  ...)
| ___bisect_matched_value___ ->
  (match ___bisect_matched_value with
  ...)

I'll consider it more this week.

I updated the release.

The above commit further reduced the compiled size, to 1099 bytes. This version of the library is now in npm as [email protected].

As an interesting factoid, when it comes to the source size, 60% of the published npm package is the library README :)

See https://github.com/aantron/promise/issues/54#issuecomment-646112858. It is currently a bit under 4K over 1K (thanks @cknitt), up from an earlier ~950 bytes, but, per the linked comment, there is likely a lot of the new stuff that can be easily reduced.

Thanks. That should be fixed in the new archive.

After #58 by @cknitt replaced Array fully by Belt.Array, the compiled size is 1184 bytes

Travis CI status reporting seems to be broken for PRs at the moment. There are several bug reports like this one at Travis with no satisfactory solution yet. The builds are running, though, e.g. https://travis-ci.org/github/aantron/promise/builds/700295405 ran for this PR.

Thanks! This decreased the incremental (that is, compared to someone already using Belt) compiled size from 3913 bytes to 1184 bytes.

As discussed in #57, make sure that on the JS side we import only belt_Array.js and not caml_array.js, too.

Thanks!

Yes.

1. When I last tried refmt (admittedly, ~2 years ago), I found it made the code much less legible to me. Among other things, it butchers the consistency of the .rei file for the many repetitive functions, by formatting them only locally according to the lengths of the tokens in each function signature.
2. Outside contributors are so rare that I found using a global "blind" format not worth the tradeoff in light of (1).

See also #48.

Not with memory safety. In that pseudocode, the function that is directly called on each promise is then. So if promises is [p1, p2], the code will attach a callback to p1 and the same callback to p2. If p1 resolves with Error(_), the "final" promise of allOk gets resolved with Error(_), but there is no way to eagerly remove the callback on p2 — it can only be removed when p2 also resolves. So that would be a memory leak. The pseudocode might work if promises had some kind of intelligent transitive callback removal scheme, so that fast-fail in race triggered callback removal for everything passed to all, which then triggered callback removal for everything passed to each then. But to my knowledge, no promise implementations do that. I'm not sure if such a scheme could be correct, predictable, performant, etc.

I believe the function that is directly called on p1 and p2 has to be race, because it is the only function that has fail-fast behavior, which, if implemented correctly without memory leaks, should remove callbacks it added to promises that haven't resolved. The only other case in which this happens in JS promises is all in case of rejection, but reason-promise doesn't use rejections, except at the binding level in module Promise.Js, and, anyway, it wouldn't help, because resolving with Error(_) is not a rejection at the JS level :) So that all leaves race as the only choice for a function to call directly on the user's promises.

Thanks, good catch. I chose Array for this initial version so I could write the same code for native and JS in limited time :) I'll solve this issue in a week or two and do a follow-on release.

Changelog:

• Add allOk, allOk2...allOk6, allOkArray — fast-fail versions of all (aantron/promise#54, requested Jay Mithani).
• Compatibility with BuckleScript 8.0.0 (aantron/promise#56, reported Christoph Knittel).

See https://github.com/aantron/promise/issues/54#issuecomment-646112858. It is currently a bit under 4K, up from an earlier ~950 bytes, but, per the linked comment, there is likely a lot of the new stuff that can be easily reduced.

Also, the implementation is slightly tricky for the sake of memory safety, see this comment if interested :)

https://github.com/aantron/promise/blob/ae44bad610e222e6e5d1f6825efd8f4035f68769/src/js/promise.re#L279-L289

The linked commit added allOk, allOk2...allOk6, and allOkArray, and these functions are now in npm as part of [email protected]. Thanks again for requesting them.

I've linked them from the docs at the bottom of the error handling section.

One side effect is that all the allOk* functions have bloated the compressed size of reason-promise from about 1K to nearly 4K. However, I think I can eliminate most of that in a future release by taking advantage of the fact that tuples are arrays, and simply unsafely defining all the allOk* functions as allOkArray, which is already the case for the plain all* functions and allArray.

The fixed version is now in npm as [email protected]. Thanks @cknitt for the report!

Hi,

Thank you for this great library! I was wondering it would be possible to add Promise.allOk (i.e. a result version of Promise.all) and its variations? At the moment, I believe there is nothing provided that replicates the fail-fast behavior of JavaScript's Promise.all with the result type. Thanks for your consideration!

The internal representation of lists changed in BuckleScript 8.0.0. Lists are now encoded as {hd, tl}, not as nested arrays anymore.

This breaks the functions listToArray and arrayToList and thereby all functions relying on them (Promise.all, Promise.race).

Yes, I will switch the library to the Belt functions and do a release. I don't immediately recall why I added these implementations.

Considering your suggestion further, it may be a good way to avoid this conclusively :)