amir/filehasher
1
0
Fork 1
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Reworking process_completion() function

Browse Source
This commit is contained in:
amir
2026-05-02 16:17:17 +01:00
parent fb83c3114f
commit 73aa4808f2
5 changed files with 269 additions and 164 deletions
Download Patch File Download Diff File Expand all files Collapse all files

53
README.md
View File

@@ -26,13 +26,38 @@ It is a high performance cross platform Windows and Linux compatible program, it
UCRT64 uses the modern Universal C Runtime (ucrtbase.dll), which supports the newest APIs, UCRT64 uses the modern Universal C Runtime (ucrtbase.dll), which supports the newest APIs,
the standard MSYS2 uses the legacy msvcrt.dll and does not support IO Ring. the standard MSYS2 uses the legacy msvcrt.dll and does not support IO Ring.
To install: To install:
pacman -S mingw-w64-ucrt-x86_64-gcc
pacman -S mingw-w64-ucrt-x86_64-clang pacman -S mingw-w64-ucrt-x86_64-clang
pacman -S mingw-w64-ucrt-x86_64-cmake or:
pacman -S mingw-w64-ucrt-x86_64-gcc
pacman -Syu pacman -Syu
And add to path: And add to path:
C:\msys64\ucrt64\bin C:\msys64\ucrt64\bin
Additionally, to use clang-cl install the latest version of Windows SDK and MSVC, or at least select these in Visual Studio Installer:
* MSVC Build tools fo x64/86.
* C++ Build tools core features.
* MSBuild support for LLVM (clang-cl) toolset.
* Windows Universal C runtime.
* Windows Universal CRT SDK.
* Windows 11 SDK.
And use the MSVC command prompt or run a script to add MSVC environment variables to current session.
Ex: for PowerShell Terminal save as .ps1 (not persistent):
```ps1
# Add MS visual studio environment variables
cmd /c '"C:\Program Files (x86)\Microsoft Visual Studio\18\BuildTools\VC\Auxiliary\Build\vcvarsall.bat" x64 && set' |
ForEach-Object {
if ($_ -match "^(.*?)=(.*)$") {
Set-Item -Path "Env:$($matches[1])" -Value $matches[2]
}
}
```
Optional: to use the build system
pacman -S mingw-w64-ucrt-x86_64-cmake
The build system uses Ninja and fallsback to make, in Windows it prefers clang-cl > gcc > clang, and in Linux gcc > clang.
### Using a build system ### Using a build system
| Command | Description| | Command | Description|
| :--- | :--- | | :--- | :--- |
@@ -52,7 +77,7 @@ clang -g -O0 file_hasher.c xxhash.c xxh_x86dispatch.c -o filehasher
clang-cl /Zi /Od file_hasher.c xxhash.c xxh_x86dispatch.c clang-cl /Zi /Od file_hasher.c xxhash.c xxh_x86dispatch.c
## Linux ## Linux
**Requirements**: GCC, CMake and Ninja **Requirements**: GCC or clang, optional CMake, Ninja or make.
### Using a build system ### Using a build system
| Command | Description| | Command | Description|
@@ -109,13 +134,16 @@ While both systems share the same core concept, their APIs and management styles
| **Partial Updates** | Supports `IORING_REGISTER_FILES_UPDATE` to swap specific indices. | No partial updates; a new registration replaces the entire table. | | **Partial Updates** | Supports `IORING_REGISTER_FILES_UPDATE` to swap specific indices. | No partial updates; a new registration replaces the entire table. |
| **Scope of Operations** | Extremely broad (files, sockets, timers, signals, etc.). | Primarily focused on file storage (read, write, flush). | | **Scope of Operations** | Extremely broad (files, sockets, timers, signals, etc.). | Primarily focused on file storage (read, write, flush). |
### Completion Wait count ### Completion Wait count and peek
To avoid busy waiting when receiving CQEs, we can use io_uring_submit_and_wait() in Linux by entering a wait count, To avoid busy waiting when receiving CQEs, we can use io_uring_submit_and_wait() in Linux by entering a wait count,
the threads sleeps until the count of CQEs are received, in windows the wait_count is present in SubmitIoRing() the threads sleep until the count of CQEs are received, in windows the wait_count is present in SubmitIoRing()
but is not implemented yet, so we wait with a completion event for a single completion. Another limitation on the completion but is not implemented yet, so we wait with a completion event for a single completion. Another limitation on the completion
event is that the kernel will waik up the thread only when receiving the first CQE, after that we need to drain the completion event is that the kernel will waik up the thread only when receiving the first CQE, after that we need to drain the completion
queue completely before sleeping again, or we enter an eternal slumber. And my config, each time the thread wakes up queue completely before sleeping again, or we enter an eternal slumber.
it receives rarely more than 3 to 5 CQEs and most of the time only one CQE. In the other hand, in Linux we can batch pop completions with io_uring_peek_batch_cqe() + io_uring_cq_advance(),
in Windows we can only pop one completion at a time with PopIoRingCompletion() (equivalent to io_uring_peek_cqe() + io_uring_cqe_seen()).
To simulate the same behavior as the Linux functions we use a double loop, an outer loop to control how much we wait
and in inner loop to drain all the available completions.
### Filtering CQEs ### Filtering CQEs
@@ -149,12 +177,15 @@ IO Ring implementation.
"Increase the limit to solve this warning.\n"); "Increase the limit to solve this warning.\n");
``` ```
The Memlock limit in Linux restricts the amount of memory a process can The Memlock limit in Linux restricts the amount of memory that can be
"lock" into physical RAM using the mlock() family of system calls. This "locked" into physical RAM using the mlock() family of system calls. This
prevents the operating system from swapping that memory out to disk. prevents the operating system from swapping that memory out to disk.
And registering buffers will lock the buffers memory so the hardware And registering buffers will lock the buffers memory so the hardware
can access it directly without kernel intervention and prevents the kernel from can access it directly without kernel intervention and prevents the kernel from
swapping it to the SSD or HDD. Increase the limit to be able to register the buffers. swapping it to the SSD or HDD.
This limit does not apply to a single process, but it applies to what all the runnig processes can lock, so in order
to be able to register the buffers, we need to set it to unlimited or increase it to at least:
num_hash_threads * NUM_BUFFERS_PER_THREAD * IORING_BUFFER_SIZE + extra memory reserved for other processes.
#### *Modifying the Limit* #### *Modifying the Limit*
The method for changing the memlock limit depends on whether you are The method for changing the memlock limit depends on whether you are
@@ -166,7 +197,7 @@ the /etc/security/limits.conf file. Add the following lines:
```conf ```conf
# Example for a specific user (replace 'username'), unlimited or a custom value in KB # Example for a specific user (replace 'username'), unlimited or a custom value in KB
username soft memlock unlimited username soft memlock unlimited
username hard memlock unlimited username hard memlock unlimitedhttps://wiki.postgresql.org/wiki/AIO
``` ```
```conf ```conf
# Example for all users # Example for all users

84
build.bat
View File

@@ -2,14 +2,23 @@
setlocal enabledelayedexpansion setlocal enabledelayedexpansion
:: ============================================================================ :: ============================================================================
:: build.bat - Build script with compiler preference: clang-cl > gcc > clang :: build.bat
:: Usage: build [Release|Debug] [clean]
:: ============================================================================ :: ============================================================================
:: Get script directory (project root)
set SCRIPT_DIR=%~dp0
set SCRIPT_DIR=%SCRIPT_DIR:~0,-1%
:: ---------------------------------------------------------------------------
:: Default values
:: ---------------------------------------------------------------------------
set BUILD_TYPE=Release set BUILD_TYPE=Release
set CLEAN_BUILD=0 set CLEAN_BUILD=0
:: --------------------------------------------------------------------------
:: Parse arguments :: Parse arguments
:: --------------------------------------------------------------------------
:parse_args :parse_args
if "%~1"=="" goto :main if "%~1"=="" goto :main
@@ -29,27 +38,34 @@ if /i "%~1"=="clean" (
goto :parse_args goto :parse_args
) )
:: Unknown argument fallback (the *)
echo Unknown argument: %~1 echo Unknown argument: %~1
echo Usage: .\%~nx0 [Release^|Debug] [clean] echo Usage: build [Release^|Debug] [clean]
exit /b 1 exit /b 1
:main :main
set BUILD_DIR=build\windows\%BUILD_TYPE% set BUILD_DIR=%SCRIPT_DIR%\build\windows\%BUILD_TYPE%
echo === Building filehasher (%BUILD_TYPE%) === echo === Building filehasher (%BUILD_TYPE%) ===
:: --------------------------------------------------------------------------
:: Clean if requested :: Clean if requested
:: --------------------------------------------------------------------------
if %CLEAN_BUILD%==1 ( if %CLEAN_BUILD%==1 (
echo Cleaning... echo Cleaning...
if exist "%BUILD_DIR%" rmdir /s /q "%BUILD_DIR%" 2>nul if exist "%BUILD_DIR%" rmdir /s /q "%BUILD_DIR%" 2>nul
) )
:: Create build dir :: --------------------------------------------------------------------------
:: Create build directory
:: --------------------------------------------------------------------------
if not exist "%BUILD_DIR%" mkdir "%BUILD_DIR%" if not exist "%BUILD_DIR%" mkdir "%BUILD_DIR%"
pushd "%BUILD_DIR%" pushd "%BUILD_DIR%"
:: Find compiler in preferred order :: --------------------------------------------------------------------------
:: Compiler selection
:: --------------------------------------------------------------------------
set CC= set CC=
where clang-cl >nul 2>&1 where clang-cl >nul 2>&1
if !ERRORLEVEL! equ 0 ( if !ERRORLEVEL! equ 0 (
echo Compiler: clang-cl ^(preferred^) echo Compiler: clang-cl ^(preferred^)
@@ -71,12 +87,14 @@ if !ERRORLEVEL! equ 0 (
goto :find_generator goto :find_generator
) )
echo ERROR: No suitable compiler found! (clang-cl, gcc, or clang required) echo ERROR: No suitable compiler found!
popd popd
exit /b 1 exit /b 1
:: --------------------------------------------------------------------------
:: Generator selection (prefer ninja)
:: --------------------------------------------------------------------------
:find_generator :find_generator
:: Find Ninja for build system
set GEN= set GEN=
where ninja >nul 2>&1 where ninja >nul 2>&1
if !ERRORLEVEL! equ 0 ( if !ERRORLEVEL! equ 0 (
@@ -86,35 +104,67 @@ if !ERRORLEVEL! equ 0 (
echo Generator: Default echo Generator: Default
) )
:: --------------------------------------------------------------------------
:: Configure :: Configure
:: --------------------------------------------------------------------------
echo. echo.
echo Configuring CMake... echo Configuring CMake...
set CMD=cmake ../../.. %GEN% %CC% -DCMAKE_BUILD_TYPE=%BUILD_TYPE% -DCMAKE_EXPORT_COMPILE_COMMANDS=ON
:: --------------------------------------------------------------------------
:: compile_commands.json logic
:: --------------------------------------------------------------------------
set EXPORT_COMPILE_COMMANDS=OFF
if /i "%BUILD_TYPE%"=="Release" (
if exist "%SCRIPT_DIR%\compile_commands.json" (
echo compile_commands.json already exists - skipping generation
) else (
echo compile_commands.json will be generated
set EXPORT_COMPILE_COMMANDS=ON
)
)
set CMD=cmake "%SCRIPT_DIR%" %GEN% %CC% -DCMAKE_BUILD_TYPE=%BUILD_TYPE% -DCMAKE_EXPORT_COMPILE_COMMANDS=%EXPORT_COMPILE_COMMANDS%
echo !CMD! echo !CMD!
!CMD! !CMD!
if !ERRORLEVEL! neq 0 (echo ERROR: Configuration failed & popd & exit /b 1) if !ERRORLEVEL! neq 0 (
echo ERROR: Configuration failed
popd
exit /b 1
)
:: --------------------------------------------------------------------------
:: Build :: Build
:: --------------------------------------------------------------------------
echo. echo.
echo Building... echo Building...
cmake --build . --config %BUILD_TYPE% cmake --build . --config %BUILD_TYPE%
if !ERRORLEVEL! neq 0 (echo ERROR: Build failed & popd & exit /b 1) if !ERRORLEVEL! neq 0 (
echo ERROR: Build failed
popd
exit /b 1
)
:: Check if compile_commands.json exists in the current build directory :: --------------------------------------------------------------------------
if exist "compile_commands.json" ( :: Copy compile_commands.json (only if generated)
:: --------------------------------------------------------------------------
if /i "%EXPORT_COMPILE_COMMANDS%"=="ON" (
if exist "compile_commands.json" (
echo. echo.
echo clangd: compile_commands.json generated echo clangd: compile_commands.json generated
:: Copy from current build dir up two levels to the project root copy /Y "compile_commands.json" "%SCRIPT_DIR%\compile_commands.json" >nul 2>&1
copy /Y "compile_commands.json" "..\..\..\compile_commands.json" >nul 2>&1
if !ERRORLEVEL! equ 0 ( if !ERRORLEVEL! equ 0 (
echo clangd: Copied to project root echo clangd: Copied to project root
) else ( ) else (
echo clangd: Could not copy to project root echo clangd: Copy failed
)
) )
) )
popd popd
echo. echo.
echo === Build Complete === echo === Build Complete ===
echo Executable: %BUILD_DIR%\filehasher.exe echo Executable: %BUILD_DIR%\filehasher.exe

27
build.sh
View File

@@ -74,7 +74,7 @@ mkdir -p "${BUILD_DIR}"
cd "${BUILD_DIR}" cd "${BUILD_DIR}"
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# Find compiler (prefer gcc, fallback to clang) # Compiler selection (prefer gcc, fallback to clang)
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
echo -e "${YELLOW}Detecting compiler...${NC}" echo -e "${YELLOW}Detecting compiler...${NC}"
@@ -135,7 +135,7 @@ fi
echo echo
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# Find build system (prefer ninja) # Generator selection (prefer ninja)
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
echo -e "${YELLOW}Selecting build system...${NC}" echo -e "${YELLOW}Selecting build system...${NC}"
@@ -165,6 +165,22 @@ echo
# Configure # Configure
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
echo -e "${YELLOW}Configuring CMake...${NC}" echo -e "${YELLOW}Configuring CMake...${NC}"
# --------------------------------------------------------------------------
# compile_commands.json logic
# --------------------------------------------------------------------------
EXPORT_COMPILE_COMMANDS=OFF
if [[ "$BUILD_TYPE" == "Release" ]]; then
if [[ -f "${SCRIPT_DIR}/compile_commands.json" ]]; then
echo -e " compile_commands.json already exists - skipping generation"
else
echo -e " compile_commands.json will be generated"
EXPORT_COMPILE_COMMANDS=ON
fi
fi
echo -e " Build type: ${BUILD_TYPE}" echo -e " Build type: ${BUILD_TYPE}"
echo -e " Compiler: ${CC_NAME}" echo -e " Compiler: ${CC_NAME}"
echo -e " Generator: ${GENERATOR_NAME}" echo -e " Generator: ${GENERATOR_NAME}"
@@ -173,7 +189,7 @@ cmake "${SCRIPT_DIR}" \
-G "${GENERATOR}" \ -G "${GENERATOR}" \
-DCMAKE_BUILD_TYPE="${BUILD_TYPE}" \ -DCMAKE_BUILD_TYPE="${BUILD_TYPE}" \
-DCMAKE_C_COMPILER="${CC_BINARY}" \ -DCMAKE_C_COMPILER="${CC_BINARY}" \
-DCMAKE_EXPORT_COMPILE_COMMANDS=ON -DCMAKE_EXPORT_COMPILE_COMMANDS=${EXPORT_COMPILE_COMMANDS}
if [[ $? -ne 0 ]]; then if [[ $? -ne 0 ]]; then
echo -e "${RED}CMake configuration failed!${NC}" echo -e "${RED}CMake configuration failed!${NC}"
@@ -243,12 +259,13 @@ echo -e " Output: ${BUILD_DIR}/"
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# Copy compile_commands.json for clangd # Copy compile_commands.json for clangd
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
if [[ -f "${BUILD_DIR}/compile_commands.json" ]]; then if [[ "${EXPORT_COMPILE_COMMANDS}" == "ON" ]]; then
if [[ -f "${BUILD_DIR}/compile_commands.json" ]]; then
echo -e " clangd: compile_commands.json generated" echo -e " clangd: compile_commands.json generated"
# Always copy the latest version
cp "${BUILD_DIR}/compile_commands.json" "${SCRIPT_DIR}/compile_commands.json" cp "${BUILD_DIR}/compile_commands.json" "${SCRIPT_DIR}/compile_commands.json"
echo -e " clangd: Copied to project root" echo -e " clangd: Copied to project root"
fi
fi fi
echo echo

1
config.h
View File

@@ -15,6 +15,7 @@
#define IORING_BUFFER_SIZE KiB(256) #define IORING_BUFFER_SIZE KiB(256)
#define NUM_BUFFERS_PER_THREAD 32 #define NUM_BUFFERS_PER_THREAD 32
#define MAX_ACTIVE_FILES 16 #define MAX_ACTIVE_FILES 16
#define MAX_WAIT_COUNT (NUM_BUFFERS_PER_THREAD / 2)
#define SUBMIT_TIMEOUT_MS 10000 #define SUBMIT_TIMEOUT_MS 10000
#define IORING_DEBUG_PRINTS 0 #define IORING_DEBUG_PRINTS 0

244
platform.c
View File

@@ -5,6 +5,7 @@
#include "lf_mpmc.h" #include "lf_mpmc.h"
#include "arena.c" #include "arena.c"
#include <stdint.h>
#include <stdio.h> #include <stdio.h>
// xxhash include // xxhash include
@@ -1149,12 +1150,6 @@ typedef struct {
uint32_t MaxVersion; uint32_t MaxVersion;
} IoRingCapabilities; } IoRingCapabilities;
typedef struct {
BUILD_READ_RETURN_VALUE ResultCode;
uint32_t Information;
uintptr_t UserData;
} IoRingCQE;
// ------------------------ IO Ring Abstraction ------------------------- // ------------------------ IO Ring Abstraction -------------------------
#if defined(_WIN32) || defined(_WIN64) #if defined(_WIN32) || defined(_WIN64)
@@ -1192,13 +1187,20 @@ static int close_ioring(ThreadIoContext *thread_ctx) {
#define MAKE_BUF_INFO(a, l) \ #define MAKE_BUF_INFO(a, l) \
(IORING_BUFFER_INFO) { .Address = (a), .Length = (uint32_t)(l) } (IORING_BUFFER_INFO) { .Address = (a), .Length = (uint32_t)(l) }
static int ioring_submit(ThreadIoContext *thread_ctx, uint32_t wait_count, static int ioring_submit(ThreadIoContext *thread_ctx, uint32_t *submitted) {
uint32_t *submitted) {
HRESULT hr = SubmitIoRing(thread_ctx->ring, 0, SUBMIT_TIMEOUT_MS, submitted);
// HRESULT hr = SubmitIoRing(ring, wait_count, SUBMIT_TIMEOUT_MS, submitted);
// The wait_count in windows is not implemented yet, so we wait with a // uint32_t wait_count = MIN(thread_ctx->num_submissions, MAX_WAIT_COUNT);
// completion event for a single completion
// The wait_count in windows is not implemented yet, so we wait in
// ioring_pop_completion()
HRESULT hr;
// if (wait_count > 0) {
// hr = SubmitIoRing(ring, wait_count, SUBMIT_TIMEOUT_MS, submitted);
// } else {
hr = SubmitIoRing(thread_ctx->ring, 0, SUBMIT_TIMEOUT_MS, submitted);
// }
if (thread_ctx->num_submissions > 0) { if (thread_ctx->num_submissions > 0) {
WaitForSingleObject(thread_ctx->completion_event, SUBMIT_TIMEOUT_MS); WaitForSingleObject(thread_ctx->completion_event, SUBMIT_TIMEOUT_MS);
} }
@@ -1222,7 +1224,7 @@ static void ioring_register_buffers(ThreadIoContext *thread_ctx,
error_msg, (unsigned int)hr); error_msg, (unsigned int)hr);
} }
// Submit registration // Submit registration
ioring_submit(thread_ctx, 0, NULL); ioring_submit(thread_ctx, NULL);
} }
#if USE_REGISTERED_FILES #if USE_REGISTERED_FILES
@@ -1293,52 +1295,69 @@ static BUILD_READ_RETURN_VALUE ioring_build_read(ThreadIoContext *thread_ctx,
return hr; return hr;
} }
static int ioring_pop_completion(IoRingHandle ring, IoRingCQE *cqe) { static void ioring_process_completions(ThreadIoContext *thread_ctx) {
uint32_t waited = 0;
uint32_t target = MIN(thread_ctx->num_submissions, MAX_WAIT_COUNT);
while (waited < target) {
// ---- Drain all available CQEs (non-blocking) ----
while (1) {
IORING_CQE win_cqe; IORING_CQE win_cqe;
while (1) { HRESULT hr = PopIoRingCompletion(thread_ctx->ring, &win_cqe);
HRESULT hr = PopIoRingCompletion(ring, &win_cqe);
if (hr == S_FALSE) if (hr == S_FALSE) {
// No CQE available // No more CQEs available right now
return 0; break;
if (FAILED(hr))
return -1;
// Unlike linux, The Windows implementation treats buffer and file
// registration as an asynchronous operation that we submit to the ring,
// similar to a read or write. Those operations produce CQEs (completion
// queue entries) that we filter here using
// cqe.UserData == USERDATA_REGISTER
if (win_cqe.UserData == USERDATA_REGISTER)
continue;
cqe->ResultCode = win_cqe.ResultCode;
cqe->Information = win_cqe.Information;
cqe->UserData = win_cqe.UserData;
// Check for error and print warning
if (FAILED(win_cqe.ResultCode)) {
char error_msg[256];
FormatMessageA(FORMAT_MESSAGE_FROM_SYSTEM | FORMAT_MESSAGE_IGNORE_INSERTS,
NULL, win_cqe.ResultCode, 0, error_msg, sizeof(error_msg),
NULL);
// Try to get the file path from the buffer
IoBuffer *buf = (IoBuffer *)win_cqe.UserData;
const char *file_path = "unknown";
if (buf && buf->file && buf->file->fe) {
file_path = buf->file->fe->path;
} }
if (FAILED(hr)) {
fprintf(stderr, "WARNING: PopIoRingCompletion failed (0x%lx)\n", hr);
return;
}
// Skip internal registration completions
if (win_cqe.UserData == USERDATA_REGISTER) {
continue;
}
IoBuffer *buf = (IoBuffer *)win_cqe.UserData;
FileReadContext *file = buf->file;
if (SUCCEEDED(win_cqe.ResultCode)) {
buf->result = 0;
buf->bytes_read = win_cqe.Information;
} else {
buf->result = win_cqe.ResultCode;
buf->bytes_read = 0;
char error_msg[256];
FormatMessageA(
FORMAT_MESSAGE_FROM_SYSTEM | FORMAT_MESSAGE_IGNORE_INSERTS, NULL,
win_cqe.ResultCode, 0, error_msg, sizeof(error_msg), NULL);
fprintf(stderr, fprintf(stderr,
"WARNING: I/O completion error for file '%s' - Error: %s (Code: " "WARNING: I/O completion error for file '%s' - Error: %s "
"0x%lx)\n", "(Code: 0x%lx)\n",
file_path, error_msg, win_cqe.ResultCode); buf->file->fe->path, error_msg, win_cqe.ResultCode);
} }
return 1; file->active_reads--;
file->reads_completed++;
thread_ctx->num_submissions--;
// Count only "real" completions toward wait budget
waited++;
}
// ---- If we already waited enough, exit ----
if (waited >= target) {
break;
}
// ---- Otherwise wait for more completions ----
WaitForSingleObject(thread_ctx->completion_event, SUBMIT_TIMEOUT_MS);
} }
} }
@@ -1431,9 +1450,10 @@ static void ioring_register_buffers(ThreadIoContext *thread_ctx,
struct rlimit limit; struct rlimit limit;
getrlimit(RLIMIT_MEMLOCK, &limit); getrlimit(RLIMIT_MEMLOCK, &limit);
fprintf(stderr, fprintf(
"WARNING: Buffer registration failed due to memlock limits " stderr,
"(ENOMEM).\n" "WARNING: Buffer registration failed due to Memlock limit, Error: "
"Cannot allocate memory (code: -12, ENOMEM).\n"
"See README for more informations.\n"); "See README for more informations.\n");
} else { } else {
@@ -1530,10 +1550,11 @@ static int ioring_build_read(ThreadIoContext *thread_ctx,
return 0; return 0;
} }
static int ioring_submit(ThreadIoContext *thread_ctx, uint32_t wait_count, static int ioring_submit(ThreadIoContext *thread_ctx, uint32_t *submitted) {
uint32_t *submitted) {
int ret; int ret;
uint32_t wait_count = MIN(thread_ctx->num_submissions, MAX_WAIT_COUNT);
if (wait_count > 0) { if (wait_count > 0) {
ret = io_uring_submit_and_wait(&((IoUring *)thread_ctx->ring)->ring, ret = io_uring_submit_and_wait(&((IoUring *)thread_ctx->ring)->ring,
wait_count); wait_count);
@@ -1553,58 +1574,46 @@ static int ioring_submit(ThreadIoContext *thread_ctx, uint32_t wait_count,
return 0; return 0;
} }
static int ioring_pop_completion(IoRingHandle ring, IoRingCQE *cqe) { static void ioring_process_completions(ThreadIoContext *thread_ctx) {
IoUring *impl = (IoUring *)thread_ctx->ring;
struct io_uring_cqe *cqe_ptr = NULL; struct io_uring_cqe *cqes[NUM_BUFFERS_PER_THREAD];
int ret = io_uring_peek_cqe(&((IoUring *)ring)->ring, &cqe_ptr); unsigned count =
io_uring_peek_batch_cqe(&impl->ring, cqes, NUM_BUFFERS_PER_THREAD);
if (ret == -EAGAIN) { if (count == 0) {
// No CQE available return;
return 0;
} }
if (ret < 0) { for (unsigned i = 0; i < count; i++) {
// Error struct io_uring_cqe *cqe = cqes[i];
fprintf(stderr, "WARNING: io_uring_peek_cqe error - Error: %s (Code: %d)\n",
strerror(-ret), ret);
return -1;
}
if (!cqe_ptr) { int res = cqe->res;
return 0;
}
int res = cqe_ptr->res; IoBuffer *buf = (IoBuffer *)cqe->user_data;
FileReadContext *file = buf->file;
if (res >= 0) { if (res >= 0) {
cqe->ResultCode = 0; buf->result = 0;
cqe->Information = (uint32_t)res; buf->bytes_read = (uint32_t)res;
} else { } else {
cqe->ResultCode = res; buf->result = res;
cqe->Information = 0; buf->bytes_read = 0;
fprintf(stderr,
"WARNING: I/O completion error for file '%s' - Error: %s (Code: "
"%d)\n",
buf->file->fe->path, strerror(-res), res);
} }
cqe->UserData = (uintptr_t)cqe_ptr->user_data; file->active_reads--;
file->reads_completed++;
io_uring_cqe_seen(&((IoUring *)ring)->ring, cqe_ptr); thread_ctx->num_submissions--;
// Check for error and print warning
if (res < 0) {
// Try to get the file path from the buffer
IoBuffer *buf = (IoBuffer *)cqe->UserData;
const char *file_path = "unknown";
if (buf && buf->file && buf->file->fe) {
file_path = buf->file->fe->path;
} }
fprintf( // Mark CQE as seen, equivalent to io_uring_cqe_seen() but marks multiple CQEs
stderr, io_uring_cq_advance(&impl->ring, count);
"WARNING: I/O completion error for file '%s' - Error: %s (Code: %d)\n",
file_path, strerror(-res), res);
}
return 1;
} }
FileHandle ioring_open_file(FileEntry *fe) { FileHandle ioring_open_file(FileEntry *fe) {
@@ -1790,22 +1799,22 @@ static void return_buffer(ThreadIoContext *ctx, IoBuffer *buf) {
} }
// -------------------------- Process completions --------------------------- // -------------------------- Process completions ---------------------------
static void process_completions(ThreadIoContext *thread_ctx) { // static void process_completions(ThreadIoContext *thread_ctx) {
IoRingCQE cqe; // IoRingCQE cqe;
//
while (ioring_pop_completion(thread_ctx->ring, &cqe) == 1) { // while (ioring_pop_completion(thread_ctx->ring, &cqe) == 1) {
//
IoBuffer *buf = (IoBuffer *)cqe.UserData; // IoBuffer *buf = (IoBuffer *)cqe.UserData;
FileReadContext *file = buf->file; // FileReadContext *file = buf->file;
//
buf->result = cqe.ResultCode; // buf->result = cqe.ResultCode;
buf->bytes_read = cqe.Information; // buf->bytes_read = cqe.Information;
//
file->active_reads--; // file->active_reads--;
file->reads_completed++; // file->reads_completed++;
thread_ctx->num_submissions--; // thread_ctx->num_submissions--;
} // }
} // }
// -------------------- File operations ----------------------- // -------------------- File operations -----------------------
static int init_file(ThreadIoContext *thread_ctx, FileReadContext *file, static int init_file(ThreadIoContext *thread_ctx, FileReadContext *file,
@@ -2074,8 +2083,7 @@ static THREAD_RETURN hash_worker_ioring(void *arg) {
FileQueue fq; FileQueue fq;
memset(&fq, 0, sizeof(fq)); memset(&fq, 0, sizeof(fq));
uint32_t submitted = 0; uint32_t submitted;
uint32_t wait_count;
// Main pipeline loop // Main pipeline loop
for (;;) { for (;;) {
@@ -2083,13 +2091,14 @@ static THREAD_RETURN hash_worker_ioring(void *arg) {
// Submit new reads // Submit new reads
build_pending_reads(thread_ctx, &fq, worker_ctx); build_pending_reads(thread_ctx, &fq, worker_ctx);
wait_count = MIN(thread_ctx->num_submissions, NUM_BUFFERS_PER_THREAD - 6);
submitted = 0; submitted = 0;
ioring_submit(thread_ctx, wait_count, &submitted); ioring_submit(thread_ctx, &submitted);
// Process completions // Process completions
process_completions(thread_ctx); ioring_process_completions(thread_ctx);
// Hash files
hash_ready_files(thread_ctx, &fq, worker_ctx);
#if IORING_DEBUG_STATS #if IORING_DEBUG_STATS
printf( printf(
@@ -2098,9 +2107,6 @@ static THREAD_RETURN hash_worker_ioring(void *arg) {
thread_ctx->active_files, fq.count); thread_ctx->active_files, fq.count);
#endif #endif
// Hash files
hash_ready_files(thread_ctx, &fq, worker_ctx);
// Exit condition // Exit condition
if (!thread_ctx->submitting && thread_ctx->active_files == 0 && if (!thread_ctx->submitting && thread_ctx->active_files == 0 &&
thread_ctx->num_submissions == 0) { thread_ctx->num_submissions == 0) {