amir/duplicatefinder
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Reworking IO Ring pipeline to fully support multiple infilght files

Browse Source 
Reworking the filequeue, the buffer chaining logic and the error
handling.
Renaming functions.
Fix bug in arena.
This commit is contained in:
amir
2026-04-23 14:13:48 +01:00
parent b8e577b5bb
commit ab31776658
7 changed files with 365 additions and 383 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
binaries/changelog.txt
View File

@@ -50,7 +50,7 @@ Fixing user prompt parsing
Reorganising the code
Improving the scan function
5.0: Implementing the IO Ring for windows and ui_uring for linux instead of buffered hashing, huge performance gains. The IO Ring is event driven, thread local, uses DMA and direct disk I/O, bypassing the OS cash completely, registred buffers, it supports bashing multiple submissions and can handle multiple files at the same time.
5.0: Implementing the IO Ring for windows and ui_uring for linux instead of buffered hashing, huge performance gains. The IO Ring is event driven, thread local, uses DMA and direct disk I/O, bypassing the OS cache completely, registred buffers, it supports bashing multiple submissions and can handle multiple files at the same time.
Hashing small files using XXH3_128bits() instead of the streaming pipeline(XXH3_128bits_reset(), XXH3_128bits_update(), XXH3_128bits_digest()), this reduses the overhead of creating a state and digest, coupled with the IO Ring it improves the hashing of small files whose size is inferior to the size of IO Ring buffers
fixing the xxh_x86dispatch warnings
Updating the progress printing function