CODEWITHSUNDEEP
javaclinux
Jacob Davis-Hansson
Jacob Davis-Hansson

Reputation: 2663

Under what circumstances may c's write()-function write less data than requested?

I'm writing to a file in chunks of about 10b to 16 000b, when suddenly every chunk gets cut off at around 4050 bytes (specifically, 4050, 4051 and 4074 bytes, in that order). This means subsequent writes will overwrite data that ought to have been written, messing up my data. Any chunk under 4050b is written out fine.

Unfortunately, I can't reproduce it. All I have is the messed up file, so I'm debugging looking for something that would cause this.

Debugging it, I realize c's write() function gets called under the hood (this is Java code, FileChannel.write(), but the standard library calls c's write and only checks that bytes written > 0), and that the docs for it do not guarantee that it will write out all data asked for, only that it will tell you how much was written.

I don't check bytes written back in Java land (but I do get it, the function signature is almost the exact same), so the fix is dead simple. However, since I can't reproduce this, I don't know that I've fixed the actual problem. So therefore I'm hoping some c guru can tell me either that I'm smoking crack, or that there are legitimate circumstances where write() will be unable to write more than about 4050 bytes at a time.

This is running on a 64-bit Linux Kernel, version 3.0.0-26.

Edit: To expand as per comments below:

I'm writing to a normal file system file. I'm not sure what non-blocking means in this context, I'm not using callbacks or anything, but explicitly telling the OS to flush to disk is not done for each chunk. The file is opened using Java's RandomAccessFile, 'rw' mode.

Upvotes: 1

Views: 1636

Answers (3)

Susam Pal
Susam Pal

Reputation: 34294

From man 2 write:

   The number of bytes written may be less than  count  if,  for  example,
   there  is  insufficient space on the underlying physical medium, or the
   RLIMIT_FSIZE resource limit is encountered (see setrlimit(2)),  or  the
   call was interrupted by a signal handler after having written less than
   count bytes.  (See also pipe(7).)

From man 2 setrlimit:

   RLIMIT_FSIZE
          The maximum size of files that the process may create.  Attempts
          to extend a file beyond this  limit  result  in  delivery  of  a
          SIGXFSZ  signal.   By default, this signal terminates a process,
          but a process can catch this signal instead, in which  case  the
          relevant  system  call  (e.g., write(2), truncate(2)) fails with
          the error EFBIG.

These limits can be seen using the following commands:

ulimit -Sf
ulimit -Hf

Or by using this C program:

#include <stdio.h>
#include <errno.h>
#include <sys/resource.h>

int main()
{
    struct rlimit rl;
    if (getrlimit(RLIMIT_FSIZE, &rl) == 0) {
        printf("%d %d\n", rl.rlim_cur, rl.rlim_max);
    } else {
        fprintf(stderr, "error: %d\n", errno);
    }
    return 0;
}

Upvotes: 5

Mats L&#246;fstr&#246;m
Mats L&#246;fstr&#246;m

Reputation: 151

From the Linux Programmer's Manual, on write(1): "write() writes up to count bytes to the file..."

How much is written depends on what you are writing to and whether you are using synchronous writes or not.

E.g. if it's a pipe och a socket, you can't write more than a pipe/socket buffer full (in synchronous mode: More than what's available in the buffer).

Upvotes: 4

Peter Lawrey
Peter Lawrey

Reputation: 533870

The simplest way to reproduce the problem is to have a slow consumer. This will cause the send buffer to be almost full when you call write() meaning not all the data you give it can be written. You can reduce the send and receiver buffer sizes to make this problem more obvious.

Upvotes: 2

Related Questions