Commit Graph

24 Commits

Author SHA1 Message Date
82e4eef3cb Updating the LF MPMC queue and replacing DirQueue with it
Making the MPMC queue support when producers are consumers at the same
time by adding a variable work, mpmc_push_work() that increments work
and mpmc_task_done() that decrements work, and if work = 0 calls
mpmc_producers_finished() that pushes poinsons to wake up sleeping
threads and make them return NULL

Replacing DirQueue, a queue growable with realloc with the MPMC queue
2026-03-11 17:56:28 +01:00
0e3ec5b09c Replacing Malloc and strdup in scan helper function with FileEntry and path arenas 2026-03-11 16:17:22 +01:00
aef070192f Using FindFirstFileA() instead of CreateFileA() to get the file size
Since we already call FindFirstFileA() and it returns the size there is
no need to open/close every file to get it's size
2026-03-11 09:02:17 +01:00
1fa306643f Align the MPMC queue to pagesize 2026-03-09 18:01:11 +01:00
f3c4cb7b76 plat_sem_destroy(&q->items_sem); 2026-03-09 17:27:45 +01:00
7d8b4addb7 Implementing a semaphore in the MPMC queue to wake up consumers 2026-03-09 16:44:43 +01:00
a299c4a1e1 LF MPMC queue improvements
Small improvements of the LF MPMC queue

Making the LF MPMC queue generic and in a seperate header file
2026-03-09 13:21:45 +01:00
b2f444af00 Making the LF MPMC queue generic and isolating it's code in a separate header file 2026-03-09 01:14:24 +01:00
75c2592bfe Making the hashing buffer reusable instead of malloc every file 2026-03-08 21:14:58 +01:00
c846952cbf forcing xxhash to use the stack instead of the heap 2026-03-08 10:47:18 +01:00
dd0797df79 hashers now use thread local arena
Instead of writing directly to file_hashes.txt, hash_workers now are
using a local arena, writing everything once at the end

using #pragma once to ensure that a given header file is included only
once in a single compilation unit
2026-03-08 10:46:05 +01:00
ee02b83094 Updating the changelog 2026-03-07 19:51:17 +01:00
8e8e6fe2b1 Merge branch 'mpmc_queue' 2026-03-07 17:12:09 +01:00
ac78f585d9 Rewriting hash_worker() to export file_hashes.txt 2026-03-07 17:10:35 +01:00
417dbad374 Bug fixes in lock free MPMC queue
Fix bug slots used before initialization,compare and swap is protecting
updating committed, but it is not protecting the memory initialization.
Adding atomic_flag commit_lock to protect against that

Fix bug multiple threads committing at the same time, fixed by using
atomic_flag commit_lock and re-checking committed after acquiring the
lock

Reorder helper functions
2026-03-07 11:36:11 +01:00
4967591ff8 Improving the lock free mpmc queue
Making the queue growable
Add padding to avoir false sharing
Add sleep() and SwitchToThread() to limit spinning
2026-03-07 01:40:31 +01:00
86ad30788a Add changlog and binaries 2026-03-06 20:52:29 +01:00
7099c1ddd6 Implementing lock free MPMC queue 2026-03-06 20:20:28 +01:00
9b327c82a6 Implementing simple MPMC queue
Rewrinting the pipeline and progress display
2026-03-06 16:44:37 +01:00
ca1bbefeaf Fix strdup warnings 2026-02-28 19:50:26 +01:00
e591dbf347 Add support for AVX2 instead of SSE2
Must compile with -mavx2 in clang/gcc or
 /arch:AVX2 in clang-cl
2026-02-28 19:44:43 +01:00
b89526d724 Removing -resume functionality 2026-02-28 19:09:28 +01:00
1744309b50 first commit 2026-02-28 10:54:16 +01:00
92aac64cf1 Initial commit 2026-02-23 23:21:13 +00:00