CODEWITHSUNDEEP
c++linuxshellunixpipe
user997112
user997112

Reputation: 30615

Fastest way to read a pipe from C/C++ program?

If I want to pipe bytes of data in to a C/C++ program on Linux like this:

cat my_file | ./my_app

but:

  1. We cannot assume the piped data is going to originate from a file
  2. We wish to interpret the data as bytes in the file (as opposed to strings)

what would be the fastest technique to read the pipe from the C/C++ application?

I have done a little research and found:

but I am not sure if there is a better way, or which of the above would be better.

EDIT: There is a performance requirement on this, hence why I am asking for the technique with the smallest overhead.

Upvotes: 1

Views: 10721

Answers (2)

Basile Starynkevitch
Basile Starynkevitch

Reputation: 1

Why do you care that much about performance?

1 gigabyte from /dev/urandom can be piped into wc in 1 minutes (and wc is running 15% of the time, waiting for data on the rest) ! Just try time (head -1000000000c /dev/urandom|wc)

But the fastest way would be to use the read(2) syscall with a quite big buffer (e.g. 64Kbytes to 256Kbytes).

Of course, read Advanced Linux Programming and carefully syscalls(2) related man pages.

Study for inspiration the source code of the Linux kernel, of GNU libc, of musl-libc. They all are open source projects, so feel free to contribute to them and to improve them.

But I bet that in practice using popen, or stdin, or reading from std::cin won't add much overhead.

You could also increase the stdio buffer with setvbuf(3).

See also this question.

(If you read from stdin the file descriptor is STDIN_FILENO which is 0)

You might be interested by time(7), vdso(7), syscalls(2)

You certainly should read documentation of GCC and this draft report.

You could use machine learning techniques to optimize performance.

Look into the MILEPOST GCC and Ctuning projects. Consider joining the RefPerSys one. Read of course Understanding machine learning: From theory to algorithms ISBN 978-1-107-05713-5

Upvotes: 4

QuestionC
QuestionC

Reputation: 10064

When you pipe data in like that, the piped input is the standard input. Just read from cin (or stdin) like a normal console program.

Just use std::cin.read(). There's no reason to deal with popen() or its ilk.

Just to clarify... there is no pipe-specific way to read the input. As far as your program is concerned, there's cin and that's it.

This question might help you out on the speed front though... Why is reading lines from stdin much slower in C++ than Python?

Upvotes: 2

Related Questions