CODEWITHSUNDEEP
clinuxstderr
Mark Richards
Mark Richards

Reputation: 187

Move stderr fd pointer to start

I'm writing a wrapper that forks an execv() process. Output from the child is captured in stderr. When waitpid() releases, I can read the contents of stderr and report it back.

In my case, I want to dynamically allocate a buffer and write stderr into this buffer.

To size the buffer I can realloc(), but this is not very efficient and I have found it tends to bloat memory pool use. Rather, I'd like to know the size without altering its pointer.

Consider the following, and note that MyStderr is just a placeholder for stderr:

int size=0;
int ch=0;
// int MyStderr is the stderr fd
FILE *nCountFD = fdopen(MyStderr, "r");
while ((ch = getc(nCountFD)!=EOF)
{
    ++size;
}
printf("Size is %ld\n", size);

Here I get the size. However, now the file pointer for MyStderr is at the end of its buffer.

I tried using lseek.

lseek() fails against stderr, so I can't use it here. At least, this is what my testing and stackoverflow searching indicates.

So ...

  1. Is there a way to get the size without moving MyStderr to eof ?

or

  1. Is there an lseek method that will work with MyStderr ?

Note. Here is the only solution I can think of, using realloc..

char *buf=(char*)malloc(80);
char *NewBuf=NULL;
int n=80;
while ((ch = getc(nCountFD)!=EOF)
{
    buf[i]=ch;
    ++size;
    if (size>n)
    {
       n=n+80;
       NewBuf= realloc(buf, n);
       // some code to make sure it works here //
       buf=NewBuf;
    }
}
printf("Size is %ld\n", size);

And now the update

Rather than build the functionality to work around the fact that stderr is unbuffered, I determined to make the initial malloc of my result buffer sufficiently large so that realloc() would be unlikely in the majority of cases. And if realloc() happens, the original allocation is doubled in size for each realloc() as suggested.

In testing (100000 iterations) this works very well with no leaking or discernible bloating.

I owe a great deal to the Stack Overflow community. Thank you all.

The code below won't run stand-alone. It's placed here to illustrate what I did.

.. after all the code that parses the commandline, forks, does execv, and cleans up...

while (waitpid(nPID, &status, 0) != nPID)
    ;
i = 0;
nFD = fdopen(nErrFD, "r");
if (!nFD) {
    snprintf(cErrMsg, 80, "Cannot open fd[%i].  Failed to spaw process",
            nErrFD);
    cbuf = strcpy(cbuf, cErrMsg);
    goto NECerror;
}

close(nErrFD);


cbuf = calloc(nBufSz, sizeof(char));
memset(cbuf, 0x00, nBufSz);
i = 0;
while ((ch = getc(nFD)) != EOF) {
    cbuf[i] = (char) ch;
    ++size;
    ++i;
    if (size > nBufSz) {
        nBufSz = nBufSz + nBaseBufSz;
        NewBuf = realloc(cbuf, nBufSz);
        if (NewBuf == NULL) {
            snprintf(cErrMsg, 80,
                    "Internal error:cannot allocate [%i] bytes", nBufSz);
            cbuf = strcpy(cbuf, cErrMsg);
            fclose(nFD);
            goto NECerror;
        }
        cbuf = NewBuf;
        free(NewBuf);
    }

}
fclose(nFD);

Upvotes: 1

Views: 450

Answers (4)

Potatoswatter
Potatoswatter

Reputation: 137890

As others have said, pipes are not seekable. All this means, however, is that you need to ensure stderr is not a pipe. In the wrapper, before execv, do

close( STDERR_FILENO );
open( error_log_path, O_RDWR | O_TRUNC ); // opens into STDERR_FILENO

The open call is guaranteed to set the standard error stream because the lowest free file descriptor is always the one assigned.

Then, the fork should duplicate the file descriptor, so the parent process sits at byte zero while the child starts writing from the beginning. There is no need to seek.

This assumes you aren't also trying to pipe the error output somewhere else at the same time. In that case, the child process would have to duplicate the error output to the externally-visible pipe and the parent's buffering file, or you would have to stream through the parent instead of using waitpid, with the child stderr and parent stderr being two different pipes.

Upvotes: 2

mrb
mrb

Reputation: 3330

Can you write the output from stderr into a temporary file? Then you can have the ability to seek randomly throughout the file, check the size, use mmap() on it, etc. You can even do these while you're still receiving data.

Upvotes: 2

Adam Rosenfield
Adam Rosenfield

Reputation: 400512

Pipes are not seekable -- if you're reading data from a pipe that's being provided by the other process, then you cannot seek its data stream. You have to store the data as its read.

The reason you're probably seeing lots of memory thrashing is due to the way your expanding your buffer -- you only expand it by a constant amount every time it's exceeded. You'll have much better success if you double its size when it's exceeded: this is a well-known technique for dynamic arrays.

For example:

// Error checking omitted for expository purposes
size_t size = 4096;
char *buf = malloc(size);
char *bufPtr = buf;
ssize_t n;
while((n = read(MyStderr, bufPtr, 4096) > 0)
{
    bufPtr += n;

    // Make sure we always have at least 4096 bytes of free space in the
    // buffer for the next read
    if(size - (bufPtr - buf) < 4096)
    {
        size *= 2;  // Watch out for overflow!
        buf = realloc(buf, size);
    }
}

// The (bufPtr - buf) bytes of buf now hold the entirety of the process's stderr

Upvotes: 4

paulsm4
paulsm4

Reputation: 121799

No - stderr is unbuffered. Your only choice is to buffer the input yourself. I'd recommend implementing a circular buffer (if you can afford to lose the oldest output), or just create a fixed buffer of some "max sizse". IMHO...

Upvotes: 0

Related Questions