massinissa/filehasher
1
0
Fork 0
forked from amir/filehasher
Code Pull Requests Activity

Add support for multiple inflight files and one shot hash small files

Browse Source 
The IO Ring now supports bashing multiple submissions and can handle
multiple files at the same time.

Hashing small files using XXH3_128bits() instead of the streaming
pipeline(XXH3_128bits_reset(), XXH3_128bits_update(),
XXH3_128bits_digest()), this reduses the overhead of creating a state
and digest, coupled with the IO Ring it improves the hashing of small
files whose size is inferior to the size of IO Ring buffers
This commit is contained in:
amir
2026-04-02 12:08:47 +01:00
parent 41ac164881
commit 0294498538
3 changed files with 354 additions and 251 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
binaries/changelog.txt
View File

@@ -50,6 +50,8 @@ Fixing user prompt parsing
Reorganising the code
Improving the scan function
5.0: Implementing the IO Ring instead of buffered hashing, huge performance gains.
5.0: Implementing the IO Ring instead of buffered hashing, huge performance gains. The IO Ring is thread local, uses DMA and direct disk I/O, bypassing the OS cash completely, it supports bashing multiple submissions and can handle multiple files at the same time.
Hashing small files using XXH3_128bits() instead of the streaming pipeline(XXH3_128bits_reset(), XXH3_128bits_update(), XXH3_128bits_digest()), this reduses the overhead of creating a state and digest, coupled with the IO Ring it improves the hashing of small files whose size is inferior to the size of IO Ring buffers
fixing the xxh_x86dispatch warnings
Updating the progress printing function