profile
viewpoint

cameron314/concurrentqueue 6229

A fast multi-producer, multi-consumer lock-free concurrent queue for C++11

cameron314/readerwriterqueue 2421

A fast single-producer, single-consumer lock-free queue for C++

cameron314/PNGEncoder2 93

A better PNG encoder for Flash

cameron314/microbench 62

A lightweight (3 file, single function) library for running micro-benchmarks on C++ code

cameron314/cpp11-on-multicore 17

Various synchronization primitives for multithreaded applications in C++11.

cameron314/atomic_bench 8

Microbenchmarks for some C++11 atomic primitives

cameron314/atomic_queue 6

C++ lock-free queue.

cameron314/jquery-timeago 2

Timeago is a jQuery plugin that makes it easy to support automatically updating fuzzy timestamps (e.g. "4 minutes ago").

cameron314/shoco 2

shoco is a compressor for small text strings

cameron314/vsclassifyexample 2

An example VS extension that highlights classes named "Stream" within C# files

issue closedcameron314/concurrentqueue

Very big cpu usage after processing tens of millions of enqueue/dequeue events

Hello,

I'm using evpp which has a event loop where the queue can be configured to use one of:

  • boost::lockfree::queue
  • moodycamel::ConcurrentQueue
  • std::vector

Your library performs much better but in time the performance goes very worst. The cpu usage is linearly increasing over time and we had to restart our server to fix the problem.

After we investigated the problem using perf-top we saw that most of the cpu is coming from the following line:

https://github.com/Qihoo360/evpp/blob/8984ca6b4fa410985ecc4a659fa0c9168dd95489/evpp/event_loop.cc#L316

We compiled evpp using boost and std::vector and we don't have this problem any longer. The evpp by default is using an old version of your library but i also tried with master as well and it's the same.

Silviu

closed time in 10 hours

silviucpp

issue commentcameron314/concurrentqueue

Very big cpu usage after processing tens of millions of enqueue/dequeue events

I'll fix the README which still assumed this was uncommented. Sorry about that!

If it works on your system, it should be safe, but since it's off by default it's not particularly well tested; just keep an eye on it.

silviucpp

comment created time in 17 hours

issue commentcameron314/concurrentqueue

Very big cpu usage after processing tens of millions of enqueue/dequeue events

Creating/destroying a producer token is significantly more expensive than enqueuing a single item. The idea is to reuse the token for multiple operations. Consider declaring the token higher up in the callstack, making the token a static thread-local object, or using a pool of producer tokens safely shared between threads.

silviucpp

comment created time in 19 hours

issue commentcameron314/concurrentqueue

Very big cpu usage after processing tens of millions of enqueue/dequeue events

With short-lived threads, I suggest using producer tokens when enqueueing, which would sidestep this problem entirely (the sub-queues created for each producer token are recycled after the token is destroyed).

silviucpp

comment created time in a day

issue commentcameron314/readerwriterqueue

bulk method

No technical reason, mainly a smaller benefit versus the time spent implementing it.

A form of this question has been asked before, see https://github.com/cameron314/readerwriterqueue/issues/76#issuecomment-893109612

qbq600

comment created time in 2 days

issue closedcameron314/readerwriterqueue

any tag/release planned?

I notice some small changes since the last tag - notably some TSAN stuff.

any plans of making a tag/release?

closed time in 2 days

ulph

issue commentcameron314/readerwriterqueue

any tag/release planned?

https://github.com/cameron314/readerwriterqueue/releases/tag/v1.0.6

ulph

comment created time in 2 days

created tagcameron314/readerwriterqueue

tagv1.0.6

A fast single-producer, single-consumer lock-free queue for C++

created time in 2 days

release cameron314/readerwriterqueue

v1.0.6

released time in 2 days

issue commentcameron314/readerwriterqueue

any tag/release planned?

Sure.

ulph

comment created time in 2 days

PR closed cameron314/readerwriterqueue

Add vcpkg installation instructions

Readerwriterqueue is available as a port in vcpkg, a C++ library manager that simplifies installation for readerwriterqueue and other project dependencies. Documenting the install process here will help users get started by providing a single set of commands to build readerwriterqueue, ready to be included in their projects.

We also test whether our library ports build in various configurations (dynamic, static) on various platforms (OSX, Linux, Windows: x86, x64, UWP, ARM) to keep a wide coverage for users.

I'm a maintainer for vcpkg, and here is what the port script looks like. We try to keep the library maintained as close as possible to the original library.

+13 -0

3 comments

1 changed file

LilyWangL

pr closed time in 2 days

pull request commentcameron314/readerwriterqueue

Add vcpkg installation instructions

Closing due to lack of information (and interest on my part).

LilyWangL

comment created time in 2 days

issue commentcameron314/concurrentqueue

Very big cpu usage after processing tens of millions of enqueue/dequeue events

Are you churning through short-lived threads (without using explicit tokens) by any chance?

The queue implementation is actually a collection of sub-queues, one for each producer. If you enqueue without an explicit producer token, a thread-local sub-queue is implicitly created, which is never destroyed. try_dequeue must walk through all these empty zombie sub-queues looking for an element, which can get very expensive in the worst case.

If you have a recent compiler on a major platform (e.g. x64 Linux) you can try uncommenting this #define; if it fixes the problem, then the above hunch is probably correct: https://github.com/cameron314/concurrentqueue/blob/550a9aa97cc64ecf235f13bda0b2c09b3515e47b/concurrentqueue.h#L221

silviucpp

comment created time in 2 days

push eventcameron314/concurrentqueue

Charles-Francois Natali

commit sha 14838d1f6f514d42841b953b24a0cfb60a993a7c

Make is_lock_free constexpr. This makes it possible to use it in `static_assert`, which is better than a runtime error.

view details

Cameron

commit sha 550a9aa97cc64ecf235f13bda0b2c09b3515e47b

Merge pull request #272 from cf-natali/charles-is-lock-free-constexpr Make is_lock_free constexpr.

view details

push time in 2 days

PR merged cameron314/concurrentqueue

Make is_lock_free constexpr.

This makes it possible to use it in static_assert, which is better than a runtime error.

+4 -4

1 comment

3 changed files

cf-natali

pr closed time in 2 days

pull request commentcameron314/concurrentqueue

Make is_lock_free constexpr.

Thanks for the contribution! I'm really quite tempted to merge this even though it breaks compatibility with VS2013 and earlier. I think I'll merge to master and see if anyone complains :-)

cf-natali

comment created time in 2 days

issue closedcameron314/concurrentqueue

can it support macOS

clone project and open from build/Xcode using my Xcode, it can't work, can you support.

closed time in 18 days

fuqiyang11

issue commentcameron314/concurrentqueue

can it support macOS

Closing due to inactivity, feel free to reopen if necessary.

fuqiyang11

comment created time in 18 days

issue closedcameron314/concurrentqueue

Benchmark BlockingConcurrentQueue against dlib::pipe

Would it be possible to update the benchmarks to include dlib::pipe from http://dlib.net/other.html#pipe ?

closed time in 18 days

pfeatherstone

issue closedcameron314/concurrentqueue

Add CMake support

Hi, this is a request to add cmake support for the library.

closed time in 18 days

hsdk123

issue commentcameron314/concurrentqueue

Add CMake support

Thanks!

hsdk123

comment created time in 18 days

issue closedcameron314/concurrentqueue

queue size

How to set the queue size exactly

closed time in 18 days

GREATwo

issue closedcameron314/concurrentqueue

wait_dequeue when T is not default constructible

What is the recommended pattern for

T t;
queue.wait_dequeue(t);

When T is does not have a default constructor?

closed time in 18 days

QFSW

issue closedcameron314/concurrentqueue

The order in case where multi producer thread and one consumer thread ?

Hi ! it's really great job ! i have two questions :

  1. Suppose i have multi producer threads , e.g. A, B,C....... , and have one consumer thread , e.g. F , then if A has successfully enqueued item-A , and then B has successfully enqueued item-B too, would F probably see the order which is item-B-item-A , namely F sees the item-B before seeing the item-A ?

    i guess it would be , since there is no guarantee about the order in which item-A 's visibility happens before item-B's , though the visibility of item-A and item-B both happen before F 's dequeueing action . But i ' m not sure of it , need to your confirms , thanks !

  2. If C needs to see the correct order: itemA-itemB , a token can do it for this , even though in case where multi producer threads and multi consumer threads?

closed time in 18 days

triump2020

issue closedcameron314/concurrentqueue

segment in try_dequeue

hi, I have two threads, one is producer, and the other is consumer. segment in try_dequeue

(gdb) thread apply all bt

Thread 65 (Thread 0x7fbe8dffb700 (LWP 48270)): #0 load (__m=std::memory_order_acquire, this=0x8) at /home/mmdev/gcc7/lib/gcc/x86_64-pc-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/atomic_base.h:396 #1 get_block_index_index_for_index (localBlockIndex=<synthetic pointer>, index=0, this=0x7fbf202e9208) at demoe/xlabutil/inc/concurrentqueue.h:2938 #2 get_block_index_entry_for_index (index=0, this=0x7fbf202e9208) at demoe/xlabutil/inc/concurrentqueue.h:2927 #3 dequeue<std::pair<std::basic_string<char>, std::basic_string<char> > > (element=..., this=0x7fbf202e9208) at demoe/xlabutil/inc/concurrentqueue.h:2550 #4 dequeue<std::pair<std::basic_string<char>, std::basic_string<char> > > (element=..., this=0x7fbf202e9208) at demoe/xlabutil/inc/concurrentqueue.h:1700 #5 try_dequeue<std::pair<std::basic_string<char>, std::basic_string<char> > > (item=..., this=0x30d51c0 <video_marquee_ftp_queue>) at demoe/xlabutil/inc/concurrentqueue.h:1125 #6 FtpUploadProcess () at demoe/demoserviceimpl_pb.cpp:119 #7 0x00000000016d1b3f in std::execute_native_thread_routine (__p=0x7fbf200aa6e0) at ../../../../../gcc-7.5.0/libstdc++-v3/src/c++11/thread.cc:83 #8 0x00007fbfb47b4dc5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fbfb39bc74d in clone () from /lib64/libc.so.6

closed time in 18 days

Juno119

issue commentcameron314/concurrentqueue

segment in try_dequeue

Closing for now, feel free to reopen if you can provide an example to reproduce the crash.

Juno119

comment created time in 18 days

issue closedcameron314/concurrentqueue

How to get the data of the head or tail of the queue

How to get the data of the head or tail of the queue

closed time in 18 days

NeilZhy

issue closedcameron314/concurrentqueue

How to clear the BlockingConcurrentQueue?

There is no clear function. How do I clear the queue?

closed time in 18 days

DougRogers

issue closedcameron314/concurrentqueue

Failable try_dequeue_bulk

Hi, first of all thanks for this wonderful library that makes my life so much easier!

At the same time, I wonder if there could be an efficient "failable" try_dequeue_bulk function such that if it cannot entirely fulfill my request of n elements, it does not dequeue any element at all? Currently, this function seems to best-effort dequeue a few elements often <n, which doesn't suit my use case well.

closed time in 18 days

uncttao

issue commentcameron314/concurrentqueue

Failable try_dequeue_bulk

Unfortunately such an all-or-nothing (atomic) bulk dequeue operation is not possible with the current design of the queue, since it's based around a collection of sub-queues that are each individually pumped for elements one after another.

uncttao

comment created time in 18 days

more