Is there a shell command to delay a buffer?

Question

I am looking for a shell command X such as, when I execute:

command_a | X 5000 | command_b

the stdout of command_a is written in stdin of command_b (at least) 5 seconds later.

A kind of delaying buffer.

As far as I know, buffer/mbuffer can write at constant rate (a fixed number of bytes per second). Instead, I would like a constant delay in time (t=0 is when X read a command_a output chunk, at t=5000 it must write this chunk to command_b).

[edit] I've implemented it: https://github.com/rom1v/delay

Answer 1

As it seemed such a command dit not exist, I implemented it in C: https://github.com/rom1v/delay

delay [-b <dtbufsize>] <delay>

Answer 2

Your question intrigued me, and I decided to come back and play with it. Here is a basic implementation in Perl. It's probably not portable (ioctl), tested on Linux only.

The basic idea is:

• read available input every X microseconds
• store each input chunk in a hash, with current timestamp as key
• also push current timestamp on a queue (array)
• lookup oldest timestamps on queue and write + discard data from the hash if delayed long enough
• repeat

Max buffer size

There is a max size for stored data. If reached, additional data will not be read until space becomes available after writing.

Performance

It is probably not fast enough for your requirements (several Mb/s). My max throughput was 639 Kb/s, see below.

Testing

# Measure max throughput:
$ pv < /dev/zero | ./buffer_delay.pl > /dev/null

# Interactive manual test, use two terminal windows:
$ mkfifo data_fifo
terminal-one $ cat > data_fifo
terminal-two $ ./buffer_delay.pl < data_fifo

# now type in terminal-one and see it appear delayed in terminal-two.
# It will be line-buffered because of the terminals, not a limitation 
# of buffer_delay.pl

buffer_delay.pl

#!/usr/bin/perl
use strict;
use warnings;
use IO::Select;
use Time::HiRes qw(gettimeofday usleep);
require 'sys/ioctl.ph';

$|++;

my $delay_usec = 3 * 1000000; # (3s) delay in microseconds
my $buffer_size_max = 10 * 1024 * 1024 ; # (10 Mb) max bytes our buffer is allowed to contain.
                              # When buffer is full, incoming data will not be read
                              # until space becomes available after writing
my $read_frequency = 10;      # Approximate read frequency in Hz (will not be exact)

my %buffer;                   # the data we are delaying, saved in chunks by timestamp
my @timestamps;               # keys to %buffer, used as a queue
my $buffer_size = 0;          # num bytes currently in %buffer, compare to $buffer_size_max

my $time_slice = 1000000 / $read_frequency; # microseconds, min time for each discrete read-step

my $sel = IO::Select->new([\*STDIN]);
my $overflow_unread = 0;      # Num bytes waiting when $buffer_size_max is reached

while (1) {
    my $now = sprintf "%d%06d", gettimeofday;  # timestamp, used to label incoming chunks

    # input available?
    if ($overflow_unread || $sel->can_read($time_slice / 1000000)) {

        # how much?
        my $available_bytes;
        if ($overflow_unread) {
            $available_bytes = $overflow_unread;
        }
        else {
            $available_bytes = pack("L", 0);
            ioctl (STDIN, FIONREAD(), $available_bytes);
            $available_bytes = unpack("L", $available_bytes);
        }

        # will it fit?
        my $remaining_space = $buffer_size_max - $buffer_size;
        my $try_to_read_bytes = $available_bytes;
        if ($try_to_read_bytes > $remaining_space) {
            $try_to_read_bytes = $remaining_space;
        }

        # read input
        if ($try_to_read_bytes > 0) {
            my $input_data;
            my $num_read = read (STDIN, $input_data, $try_to_read_bytes);
            die "read error: $!" unless defined $num_read;
            exit if $num_read == 0;       # EOF
            $buffer{$now} = $input_data;  # save input
            push @timestamps, $now;       # save the timestamp
            $buffer_size += length $input_data;
            if ($overflow_unread) {
                $overflow_unread -= length $input_data;
            }
            elsif (length $input_data < $available_bytes) {
                $overflow_unread = $available_bytes - length $input_data;
            }
        }
    }

    # write + delete any data old enough
    my $then = $now - $delay_usec; # when data is old enough
    while (scalar @timestamps && $timestamps[0] < $then) {
        my $ts = shift @timestamps;
        print $buffer{$ts} if defined $buffer{$ts};
        $buffer_size -= length $buffer{$ts};
        die "Serious problem\n" unless $buffer_size >= 0;
        delete $buffer{$ts};
    }

    # usleep any remaining time up to $time_slice
    my $time_left = (sprintf "%d%06d", gettimeofday) - $now;
    usleep ($time_slice - $time_left) if $time_slice > $time_left;
}

Feel free to post comments and suggestions below!

Answer 3

This might work

time_buffered () {
   delay=$1
   while read line; do
       printf "%d %s\n" "$(date +%s)" "$line"
   done | while read ts line; do
       now=$(date +%s)
       if (( now - ts < delay)); then
           sleep $(( now - ts ))
       fi
       printf "%s\n" "$line"
   done
}

commandA | time_buffered 5 | commandB

The first loop tags each line of its input with a timestamp and immediately feeds it to the second loop. The second loop checks the timestamp of each line, and will sleep if necessary until $delay seconds after it was first read before outputting the line.

Answer 4

Something like this?

#!/bin/bash
while :
do
   read line 
   sleep 5
   echo $line
done

Save the file as "slowboy", then do

chmod +x slowboy

and run as

command_a | ./slowboy | command_b

Answer 5

I know you said you're looking for a shell command, but what about using a subshell to your advantage? Something like:

command_a | (sleep 5; command_b)

So to grep a file cat-ed through (I know, I know, bad use of cat, but just an example):

cat filename | (sleep 5; grep pattern)

A more complete example:

$ cat testfile
The
quick
brown
fox
$ cat testfile | (sleep 5; grep brown)
# A 5-second sleep occurs here
brown

Or even, as Michale Kropat recommends, a group command with sleep would also work (and is arguably more correct). Like so:

$ cat testfile | { sleep 5; grep brown; }

Note: don't forget the semicolon after your command (here, the grep brown), as it is necessary!