So that Eclipser can be used easily on current Linux plus not needing to install all the dependencies required - could you please supply a Dockerfile for Eclipser 2.0? Based on the fuzzbench builder.Dockerfile + runner.Dockerfile this should not be too much work? (I would do it myself, but too much on my plate for afl++ at the moment ...)

I think this can be closed now. thanks!

Kitty network fuzzer added to /data/fuzzers.json

I know there is “xadd", but is there "xadd eax, eax" in the commodity software? Thanks.

It seems that this bug has not been repaired by PyVEX. I supposed that maybe this instruction won't appear in actual software.

Happy New Year!

I have problems with MoFuzz:

1. I cannot find the PDF. Just an abstract.
2. It's main reference is Zest, a fuzzer from ISSTA 2019. It's referenced often, but we didn't add it so we cannot link. Should we add Zest in the DB? Should we add ISSTA to our selection criteria (it was mentioned several time on Twitter after we did the last update).

Done with the rest :)

And yes, I can maintain aflplusplus_eclipser for future updates in Eclipser.

Although the result is promising, I would prefer not to include aflplusplus_eclipser in the core fuzzer. Since 'AFL++ without cmplog' is not included in the core fuzzer, it may be hard to check the coverage improvement made by Eclipser over the random fuzzing baseline.

I was assuming that each docker container used for fuzzing was confined to run with a single CPU core. If that was not the case, then I agree the comparison is unfair.

Actually, things get complicated here, because (Eclipser & AFL) will not fully utilize two cores even if if there are enough cores. Since Eclipser was originally intended for a single-core environment, it monitors how effectively it increases coverage, and adaptively yields resource (i.e. sleeps) to give more resource to random fuzzing module (which is AFL in Eclipser v2.0).

In my experience, the Eclipser thread will converge to utilize around 0.25 CPU cores eventually, when fuzzing time is long enough. This makes it hard to design a properly aligned comparison setting. Limiting the docker container to run with a single CPU core seems to be the best option so far.

The result of integrating Eclipser with AFL++ is interesting. If I understood the code correctly, AFL++ was run without 'cmplog' feature (https://github.com/google/fuzzbench/blob/master/fuzzers/aflplusplus_eclipser/fuzzer.py#L51).

yes

This indicates that AFL++'s 'cmplog' feature (inspired from REDQUEEN) and Eclipser's grey-box concolic testing have almost equivalent effectiveness in increasing coverage. This makes sense because Eclisper and REDQUEEN tried to address the same research problem.

I would not say so, because it is not really a fair comparison. afl/afl++ and eclipser together use 2 CPU cores/threads, whereas aflplusplus (with cmplog) is using only one. So a fair comparison would be running them aginst 2xafl or 2xafl++ (one with cmplog one without).

I did the 2xafl++ (one with cmplog, one without) vs afl++ + eclipser here: https://www.fuzzbench.com/reports/experimental/2020-12-26/ due to a bug in fuzzbench it only completed half of the benchmark time, but it shows that the mono+binary analysis is - unsurprisingly - at a larger disadvantage.

but IMHO this is fine. there is no "the best" fuzzer. because what eclipser does has difference to cmplog it will usualyl find different paths, and that helps in a fuzzing campaign. plus it is easier to deploy than symcc, plus it also works on binary-only fuzzing.

maybe the best comparison would be to run aflplusplus_qemu with cmplog + aflplusplus_qemu without cmplog vs aflplusplus_qemu without cmplog + eclipser.

And I think integrating Eclipser with AFL is still meaningful, because the difference between (Eclipser & AFL) and (AFL) directly shows how effectively Eclipser solved branches that AFL could not cover. So I think it would be the best to keep both (Eclipser & AFL) and (Eclipser & AFL++) in fuzzbench.

I don't mind, I was just curious how that would affect the ranking :) will you take over the aflplusplus_eclipser maintainance for potential upcoming eclipser changes? I could handle any afl++ changes. and do you want it as a core fuzzer?

I believe it's a good idea to keep the Dockerfile for the latest Eclipser in the pository. Please try this Dockerfile in 'dev' branch.

it build fine, thank you! I would recommend to add ENV DOTNET_CLI_TELEMETRY_OPTOUT 1

The result of integrating Eclipser with AFL++ is interesting. If I understood the code correctly, AFL++ was run without 'cmplog' feature (https://github.com/google/fuzzbench/blob/master/fuzzers/aflplusplus_eclipser/fuzzer.py#L51).

This indicates that AFL++'s 'cmplog' feature (inspired from REDQUEEN) and Eclipser's grey-box concolic testing have almost equivalent effectiveness in increasing coverage. This makes sense because Eclisper and REDQUEEN tried to address the same research problem.

And I think integrating Eclipser with AFL is still meaningful, because the difference between (Eclipser & AFL) and (AFL) directly shows how effectively Eclipser solved branches that AFL could not cover. So I think it would be the best to keep both (Eclipser & AFL) and (Eclipser & AFL++) in fuzzbench.

I believe it's a good idea to keep the Dockerfile for the latest Eclipser in the pository. Please try this Dockerfile in 'dev' branch.

btw - combining afl++ with eclipser gives it a huge performance improvement, bypassing entropic and libfuzzer: https://www.fuzzbench.com/reports/experimental/2020-12-25/index.html

So that Eclipser can be used easily on current Linux plus not needing to install all the dependencies required - could you please supply a Dockerfile for Eclipser 2.0? Based on the fuzzbench builder.Dockerfile + runner.Dockerfile this should not be too much work? (I would do it myself, but too much on my plate for afl++ at the moment ...)

It turned our that both qemu-2.10.0 (used in Eclipser 2.0) and qemu-2.3.0 (used in Eclipser 1.1) are affected by this issue.

For Eclipser 2.0, commit c0ae7fd in 'dev' branch resolves this issue.

For Eclipser 1.1, we created a new branch 'v1.x' for the maintenance, and commit 75b1bc8 resolves this issue (there were some force pushes, so please ignore the other commits).

So you can try 'dev' and 'v1.x' branches for Ubuntu 20.04.

It turned our that both qemu-2.10.0 (used in Eclipser 2.0) and qemu-2.3.0 (used in Eclipser 1.1) are affected by this issue.

For Eclipser 2.0, commit c0ae7fd in 'dev' branch resolves this issue.

For Eclipser 1.1, we created a new branch 'v1.x' for the maintenance, and commit 67804d9 resolves this issue.

So please try 'v1.x' branch for Ubuntu 20.04.

Don't stress about it; turns out most of the other tools DeepState uses ALSO fail on 20.04, except AFL and libFuzzer, so we're holding off moving until 20.04 is closer to the fuzzing community's horizon.

I will investigate this issue and try to make Eclipser v1.1 work on Ubuntu 20.04.

Short version: ubuntu 20.04 uses a later glibc, which leads to a failure to build qemu according to the recipe in the v1.1 ("old Eclipser") release. It looks like QEMU has fixed this, but presumably you are tagged to version 2.10.0 for good reasons. Any chance you can incorporate the fix here https://patchwork.openembedded.org/patch/165581/ to get Eclipser 1.1 runnable on 20.04?

We're trying to make a DeepState (https://github.com/trailofbits/deepstate) release that bases off 20.04 instead of 18.04, and we'd like to keep the "old" Eclipser in the fuzzer stable (and add Eclipser 2.0 when we get a chance; the switch to relying on AFL parallel mode makes this non-trivial). But when we try to build from docker, we run into:

/home/user/Eclipser/Instrumentor/qemu/qemu-2.3.0-pathcov-x64/linux-user/syscall.c:235:16: error: static declaration of 'gettid' follows non-static declaration
  235 | _syscall0(int, gettid)
      |                ^~~~~~
/home/user/Eclipser/Instrumentor/qemu/qemu-2.3.0-pathcov-x64/linux-user/syscall.c:164:13: note: in definition of macro '_syscall0'
  164 | static type name (void)   \
      |             ^~~~
In file included from /usr/include/unistd.h:1170,
                 from /home/user/Eclipser/Instrumentor/qemu/qemu-2.3.0-pathcov-x64/linux-user/syscall.c:27:
/usr/include/x86_64-linux-gnu/bits/unistd_ext.h:34:16: note: previous declaration of 'gettid' was here
   34 | extern __pid_t gettid (void) __THROW;
      |                ^~~~~~
/home/user/Eclipser/Instrumentor/qemu/qemu-2.3.0-pathcov-x86/linux-user/syscall.c:235:16: error: static declaration of 'gettid' follows non-static declaration
  235 | _syscall0(int, gettid)
      |                ^~~~~~
/home/user/Eclipser/Instrumentor/qemu/qemu-2.3.0-pathcov-x86/linux-user/syscall.c:164:13: note: in definition of macro '_syscall0'
  164 | static type name (void)   \
      |             ^~~~
In file included from /usr/include/unistd.h:1170,
                 from /home/user/Eclipser/Instrumentor/qemu/qemu-2.3.0-pathcov-x86/linux-user/syscall.c:27:
/usr/include/x86_64-linux-gnu/bits/unistd_ext.h:34:16: note: previous declaration of 'gettid' was here
   34 | extern __pid_t gettid (void) __THROW;
      |                ^~~~~~
make[2]: *** [/home/user/Eclipser/Instrumentor/qemu/qemu-2.3.0-pathcov-x64/rules.mak:57: linux-user/syscall.o] Error 1
make[1]: *** [Makefile:173: subdir-x86_64-linux-user] Error 2
make[1]: Leaving directory '/home/user/Eclipser/Instrumentor/qemu/qemu-2.3.0-pathcov-x64'
make: *** [Makefile:41: /home/user/Eclipser/Instrumentor/qemu/.compiled_x64] Error 1
make: *** Waiting for unfinished jobs....
/home/user/Eclipser/Instrumentor/qemu/qemu-2.3.0-pathcov-x86/linux-user/syscall.c: In function 'do_syscall':
/home/user/Eclipser/Instrumentor/qemu/qemu-2.3.0-pathcov-x86/linux-user/syscall.c:5962:29: warning: implicit declaration of function 'stime'; did you mean 'utime'? [-Wimplicit-function-declaration]
 5962 |             ret = get_errno(stime(&host_time));
      |                             ^~~~~
      |                             utime
/home/user/Eclipser/Instrumentor/qemu/qemu-2.3.0-pathcov-x86/linux-user/syscall.c:5962:29: warning: nested extern declaration of 'stime' [-Wnested-externs]
make[2]: *** [/home/user/Eclipser/Instrumentor/qemu/qemu-2.3.0-pathcov-x86/rules.mak:57: linux-user/syscall.o] Error 1
make[1]: *** [Makefile:173: subdir-i386-linux-user] Error 2
make[1]: Leaving directory '/home/user/Eclipser/Instrumentor/qemu/qemu-2.3.0-pathcov-x86'
make: *** [Makefile:37: /home/user/Eclipser/Instrumentor/qemu/.compiled_x86] Error 1

There may be other issues preventing Eclipser 1.1 from running on 20.04 but this one seems fixable, if you have time.