CODEWITHSUNDEEP
unixprocessmetrics
alexanderkuk
alexanderkuk

Reputation: 1701

Is there any way to get very precise information (number of used bytes, number of CPU instructions) about process in Unix?

Here is the problem: there is a set of programs written in different languages (mostly Perl and Python). Every program x reads lines from stdin, does some work (parses line, updates data structes, no long queries to DB or fancy network communication, even disk IO is rare), and maybe prints something to stdout. The task is to write such program f, that given x and stdin, will sample such lines, that were most computationally hard for x. The idea is to use such lines for testing and benchmarking of x in future.

Here is the thing I am stuck into: f wraps x, reads a single line l from stdin, x is ready to process l, f passes l to x and immideatly starts to collect staticts about x. The thing is that I can not find any metric that would distinct computationaly hard and easy lines. For now I've tried two approaches:

  1. Dump whole /proc/[x pid]/stat between runs of x. It almost does not change (even CPU ticks).
  2. Just monitor x state (using the same /proc/[x pid]/stat) and try to measure time it was running. It does not differ between lines.

Maybe there are some high precision metrics? Like number of CPU commands run or number of bytes in memory used?

Here is actual code in Python that I've written, it is full of details so it is the last thing to read it I think https://gist.github.com/alexanderkuk/5630079#file-f-py .

Upvotes: 4

Views: 172

Answers (1)

user2404501
user2404501

Reputation:

There are a lot of problems with your code. First, this:

def command_is_running(pid):
    with open('/proc/%d/stat' % pid) as stat:
        stats = stat.read()
        return ' R ' in stats

def wait_command_processes_line(pid):
    # stats = ...
    while command_is_running(pid):
      # stats = update_stats(stats, pid)
    return stats

is a busy loop. It will eat as much CPU as it can, reading .../stat repeatedly until the R goes away. It can't be a good idea to run an extra CPU-hogging process while you're trying to get an accurate timing of CPU usage.

I don't know of any way to put a process to sleep until another process's run state changes, so I can't offer an efficient replacement for the busy loop. But that doesn't matter because of the second problem: process state isn't as predictable as you want it to be.

You've made an assumption that the process will become runnable the instant you write some data into its pipe, and will stay runnable for the duration of the processing of that input. It would be very hard to guarantee that this is true. You've said "disk IO is rare" but you'd have to do better than that and completely eliminate it, including page faults. That's hard, and you probably haven't done it. So I think your problem is not that /proc/PID/stat contains the wrong information, but that you're reading it at the wrong times.

You might get around the disk IO problem by treating D state the same as R. But it still looks kludgy.

Instead of looking at process runnability, you should find a better indicator that the child process has finished handling the most recent input line. You said it "maybe prints something to stdout". If you can arrange for it to always print something to stdout for every input line, then the parent process could wait for that output and sample the child's CPU usage when it appears.

If you can't get the child process to provide an external indication of completion for each input line, an alternative might be to consider it done with an input line when it tries to read the next input line. Basically you'd be using ptrace to implementing a specialized strace-like utility, recording the times of the reads on the input pipe, writing a line into the pipe only after your tracing tells you that it's trying to read.

Maybe you can even do that with strace and some clever shell scripting.

Another variation on that idea would be to use gdb to set a breakpoint in the child process at the start of its input processing loop, and set up a script to be run every time the breakpoint is hit. The script would collect timing information, write the next line to the pipe, and then do a gdb cont.

Upvotes: 1

Related Questions