Now we have 3 different API compatible MPMC queues that we can swap with
swapping the header.
mt_mpmc.h, a blocking queue that uses a mutex/critical section.
lf_mpmc.h, a lock free queue that uses atomics.
sm_mpmc.h, a hybrid queue that uses atomics and a semaphore to block
when the queue is empty.
In this program, for max performance it is recommanded to use sm_mpmc.h
or mt_mpmc.h, they are designed to avoid busy waiting which frees more
CPU time to do useful work.
The windows implementation is disabled, currently registering files in
IO Ring when there is inflight IO operations causes corruptions.
Implementing a config file.
Some code cleanup
The IO Ring now supports bashing multiple submissions and can handle
multiple files at the same time.
Hashing small files using XXH3_128bits() instead of the streaming
pipeline(XXH3_128bits_reset(), XXH3_128bits_update(),
XXH3_128bits_digest()), this reduses the overhead of creating a state
and digest, coupled with the IO Ring it improves the hashing of small
files whose size is inferior to the size of IO Ring buffers
This dispatcher can not be added in a unity build and we must remove
AVX2 or AVX512 compilation flags, link xxh_x86dispatch.c in the
compilation command. The compilaiton throws two warnings about function
with internal linkage but not defined, they are defined in
xxh_x86dispatch.c so it's harmless warnings
Making the MPMC queue support when producers are consumers at the same
time by adding a variable work, mpmc_push_work() that increments work
and mpmc_task_done() that decrements work, and if work = 0 calls
mpmc_producers_finished() that pushes poinsons to wake up sleeping
threads and make them return NULL
Replacing DirQueue, a queue growable with realloc with the MPMC queue
Instead of writing directly to file_hashes.txt, hash_workers now are
using a local arena, writing everything once at the end
using #pragma once to ensure that a given header file is included only
once in a single compilation unit
Fix bug slots used before initialization,compare and swap is protecting
updating committed, but it is not protecting the memory initialization.
Adding atomic_flag commit_lock to protect against that
Fix bug multiple threads committing at the same time, fixed by using
atomic_flag commit_lock and re-checking committed after acquiring the
lock
Reorder helper functions
