Replacing snprintf(), localtime() and strftime() with custom formating
functions
Reworking progress_thread(), instead of composing the progress printing
with multiple printf() and putchar, we compose it in a buffer and write
it at onse using WriteConsole in Windows and write in Linux
Fix bug in mt_mpmc.c, in Linux mutexes are not recursive.
Add arena_trim_string() to the arena API
Removing arena->path, now paths are pushed to arena->metadata
Replacing fe->owner[128] with char *owner; the owner is not pushed as a
string to arena->metadata and trimed with arena_trim_string()
Improving cache locality in arena->metadata, the memory layout is not
fe; fe->path; fe->owner.
Cache aligning all arenas except HasherContext->arena to sizeof(void *).
Pushing elements one by one instead of snprintf() in finalize_file() and
hash_worker().
Getting the full path of current directory instead of "."
Fixing bug in path formating, this allow us to remove normalize_path()
from the hot loop.
Now we have 3 different API compatible MPMC queues that we can swap with
swapping the header.
mt_mpmc.h, a blocking queue that uses a mutex/critical section.
lf_mpmc.h, a lock free queue that uses atomics.
sm_mpmc.h, a hybrid queue that uses atomics and a semaphore to block
when the queue is empty.
In this program, for max performance it is recommanded to use sm_mpmc.h
or mt_mpmc.h, they are designed to avoid busy waiting which frees more
CPU time to do useful work.
The windows implementation is disabled, currently registering files in
IO Ring when there is inflight IO operations causes corruptions.
Implementing a config file.
Some code cleanup
The IO Ring now supports bashing multiple submissions and can handle
multiple files at the same time.
Hashing small files using XXH3_128bits() instead of the streaming
pipeline(XXH3_128bits_reset(), XXH3_128bits_update(),
XXH3_128bits_digest()), this reduses the overhead of creating a state
and digest, coupled with the IO Ring it improves the hashing of small
files whose size is inferior to the size of IO Ring buffers
This dispatcher can not be added in a unity build and we must remove
AVX2 or AVX512 compilation flags, link xxh_x86dispatch.c in the
compilation command. The compilaiton throws two warnings about function
with internal linkage but not defined, they are defined in
xxh_x86dispatch.c so it's harmless warnings
Making the MPMC queue support when producers are consumers at the same
time by adding a variable work, mpmc_push_work() that increments work
and mpmc_task_done() that decrements work, and if work = 0 calls
mpmc_producers_finished() that pushes poinsons to wake up sleeping
threads and make them return NULL
Replacing DirQueue, a queue growable with realloc with the MPMC queue
Instead of writing directly to file_hashes.txt, hash_workers now are
using a local arena, writing everything once at the end
using #pragma once to ensure that a given header file is included only
once in a single compilation unit
Fix bug slots used before initialization,compare and swap is protecting
updating committed, but it is not protecting the memory initialization.
Adding atomic_flag commit_lock to protect against that
Fix bug multiple threads committing at the same time, fixed by using
atomic_flag commit_lock and re-checking committed after acquiring the
lock
Reorder helper functions
