This dispatcher can not be added in a unity build and we must remove
AVX2 or AVX512 compilation flags, link xxh_x86dispatch.c in the
compilation command. The compilaiton throws two warnings about function
with internal linkage but not defined, they are defined in
xxh_x86dispatch.c so it's harmless warnings
Making the MPMC queue support when producers are consumers at the same
time by adding a variable work, mpmc_push_work() that increments work
and mpmc_task_done() that decrements work, and if work = 0 calls
mpmc_producers_finished() that pushes poinsons to wake up sleeping
threads and make them return NULL
Replacing DirQueue, a queue growable with realloc with the MPMC queue
Instead of writing directly to file_hashes.txt, hash_workers now are
using a local arena, writing everything once at the end
using #pragma once to ensure that a given header file is included only
once in a single compilation unit
Fix bug slots used before initialization,compare and swap is protecting
updating committed, but it is not protecting the memory initialization.
Adding atomic_flag commit_lock to protect against that
Fix bug multiple threads committing at the same time, fixed by using
atomic_flag commit_lock and re-checking committed after acquiring the
lock
Reorder helper functions
