CODEWITHSUNDEEP
linuxsolid-state-driveaio
Nicola
Nicola

Reputation: 405

Why io_submit(..., nr, ...) might submit less requests than nr?

I'm using io_submit(..., nr, ...) with nr up to 128 but I usually get fewer requests submitted.

According to the manual IO_SUBMIT(2), this is legit but I wonder: why? Also, is there a way to know which request was submitted right away - without checking io_getevents()?

From the manual:

On success, io_submit() returns the number of iocbs submitted (which may be less than nr, or 0 if nr is zero). For the failure return, see NOTES.

Upvotes: 2

Views: 671

Answers (2)

Taz'dingo
Taz'dingo

Reputation: 1

One possible reason to cause 1 returned by io_submit with param nr > 1 is that param iocbpp is not an array of iocb pointer. A mistake I made is passing the address of the iocb array:

struct iocb* io_request = (struct iocb *)(malloc((sizeof(struct iocb)) * len));
//... io_prep_pwrite stuff
ret = io_submit(ctx, len, &req.io_request); // return 1

If I passing +1 address will get EFAULT(bad address):

struct iocb* io_request = (struct iocb *)(malloc((sizeof(struct iocb)) * len));
//... io_prep_pwrite stuff
ret = io_submit(ctx, len, (&req.io_request)+1); // return -14

To fix:

struct iocb* request[len];
struct iocb* io_request = (struct iocb *)(malloc((sizeof(struct iocb)) * len));
//... io_prep_pwrite stuff
for (int i = 0; i < len_; i++) {
    request[i] = &(io_request[i]);
}
ret = io_submit(ctx, len, request); // return len

Thanks to Mike. As Mike mentioned, argument problems with one of the requests will cause a short response, but no error value is return by io_submit to debug with. We can pass iocbpp at return position, to get the error value, iocbpp + 5 in his example, iocbpp + 1 in my example.

Upvotes: 0

Mike Andrews
Mike Andrews

Reputation: 3205

One possible reason you're getting fewer results submitted is that you've reached a limit of 128 requests outstanding. Maybe you called io_submit with 128, and then call it again before all 128 requests have completed? See the nr_requests entry in sysfs for your device for the limit on the number of requests its willing to handle:

# cat /sys/block/sda/queue/nr_requests
128

It's up to your code to track how many requests are outstanding and to avoid trying to call io_submit with a number of requests that'd push it over this limit.

Now, that might not be why your call is submitting fewer requests than you asked. If you look at the kernel source, in fs/aio.c, you can see all the failure conditions in __io_submit_one that could trigger a short response. There could be argument problems with one of the requests, or maybe it can't allocate slab memory for the kiocb, etc. But most likely, it's aio_get_req that's failing due to that nr_requests queue depth limit.

If you look at the SYSCALL_DEFINE3(io_submit... in that same file, you can see it's straightforward to figure out which requests weren't submitted. It's simply the position in the array you handed to io_submit. For example, if io_submit returns 5, then you know that the requests that you submitted in array indices 0-4 have been submitted. Anything between 5 and nr - 1 has not been submitted.

Upvotes: 2

Related Questions