CODEWITHSUNDEEP
linuxbashstdout
Hongyi Zhang
Hongyi Zhang

Reputation: 19

Dealing with data stream with linux bash

I am dealing with live streaming data from babeltrace command. The command will continuously output log. Now I want to separate those logs and save into different files. For example, logs between 0~10 seconds save to file1, logs between 10~20 seconds save to file2 ... Is this possible to realize this function with Linux bash when executing the command at the same time?

I have tried:

    babeltrace --input-format=lttng-live Test | tee file1 file2 ...

But tee will redirect everything in all the files.

Upvotes: 1

Views: 957

Answers (1)

William Pursell
William Pursell

Reputation: 212674

You really need to just write a program to do it if you want anything robust. You could try something like:

babeltrace --input-format=lttng-live Test | { i=0; while : $((i++)); do 
    timeout 10 sh -c "cat > file-$i"; done; }

But I wouldn't trust that in a production environment.

== Edit == The reason you shouldn't trust this in a production environment. I don't know the implementation details of timeout and cat, and even if I knew them well I wouldn't trust them not to change (with time or platform). I suspect timeout will send a signal to terminate the shell. What happens if cat has read some data but not yet written it? Will cat terminate gracefully and flush all of its buffers, or will any unwritten data be discarded? Does timeout send a TERM or an INT or (gasp) a KILL? Is that behavior standardized? Is it standardized that cat will terminate gracefully when it receives the signal? I don't know, and it's easier to just write a robust program that deals with those issues than it is to try to rely on timeout and cat.

Another thought (I still wouldn't trust this to be robust):

{ while sleep 10; do echo SENTINEL; done & babeltrace ...; } |
    awk '/^SENTINEL/{of="output-file" idx++; next} {print > of}' of=output-file0

The basic idea here is that you let a single awk process read the output of babeltrace and write an indicator into the output stream every 10 seconds that causes awk to change the output file. Note that if your output stream contains a line that matches '^SENTINEL', it will be discarded. A slightly more robust way to do this would be (cough, cough, humor me here) to have awk register a signal handler that changes the output file name and have a timer send it a signal, but a) I don't know if you can register signal handler with any but esoteric awk, and b) seriously, did I say that would be more robust?! If you expect the process to run for a long time, you'll probably want to close the output files within awk as well.

Upvotes: 1

Related Questions