forked from amir/filehasher
amir
c1abada7ba
Making the MPMC queue support when producers are consumers at the same time by adding a variable work, mpmc_push_work() that increments work and mpmc_task_done() that decrements work, and if work = 0 calls mpmc_producers_finished() that pushes poinsons to wake up sleeping threads and make them return NULL Replacing DirQueue, a queue growable with realloc with the MPMC queue
44 lines
2.2 KiB
Plaintext
44 lines
2.2 KiB
Plaintext
V1.0: Recursive scan
|
|
|
|
v1.1: Collects more metadata
|
|
|
|
v2.0: Multi threaded scan
|
|
|
|
v2.1: Uses AVX2 instead of SSE2
|
|
|
|
v3.0: Simple mutex/critical section based MPMC queue
|
|
|
|
v3.1: Lock free MPMC queue Vyukov-style
|
|
|
|
v3.2: Making the lock free MPMC queue growable
|
|
Add padding to avoir false sharing
|
|
Add sleep() and SwitchToThread() to limit spinning
|
|
|
|
v3.3: Fix bug slots used before initialization,compare and swap is protecting updating committed, but it is not protecting the memory initialization. Adding atomic_flag commit_lock to protect against that
|
|
Fix bug multiple threads committing at the same time, fixed by using atomic_flag commit_lock and re-checking committed after acquiring the lock
|
|
Reorder helper functions
|
|
|
|
v3.4: Rewriting hash_worker() to export file_hashes.txt
|
|
|
|
3.5: Instead of writing directly to file_hashes.txt, hash_workers now are using a local arena, writing everything once at the end
|
|
Using #pragma once to ensure that a given header file is included only once in a single compilation unit
|
|
Forcing xxhash to use the stack instead of the heap
|
|
Making the hashing buffer reusable instead of malloc every file
|
|
Implementing a general purpose arena to replace small allocations
|
|
Small improvements of the LF MPMC queue
|
|
Making the LF MPMC queue generic and in a seperate header file
|
|
|
|
4.0: Implementing a semaphore in the LF MPMC queue to wake up consumers when there is items in the queue instead of spinning (busy waiting) or sleeping, this makes the queue spin only when the slots are transitionning (multiple consumers claiming the same slot)
|
|
|
|
Making the MPMC queue platform agnostic
|
|
|
|
Align the MPMC queue to pagesize
|
|
|
|
Getting file size from FindFirstFileA() instead of CreateFileA(), since we already call FindFirstFileA() and it returns the size there is no need to open/close every file to get it's size
|
|
|
|
Replacing Malloc and strdup in scan helper function with FileEntry and path arenas
|
|
|
|
Making the MPMC queue support when producers are consumers at the same time by adding a variable work, mpmc_push_work() that increments work and mpmc_task_done() that decrements work, and if work = 0 calls mpmc_producers_finished() that pushes poinsons to wake up sleeping threads and make them return NULL
|
|
|
|
Replacing DirQueue, a queue growable with realloc with the MPMC queue
|