How to prevent a process from spawning more childs

Question

I am working on an online judge for conducting ACM-ICPC like competitions on my college LAN. For that I require that the judge may be safe enough to prevent malicious programs from executing themselves on my server. (An example of such a program would be )

int main(){
    while(1) fork();
}

Lets call the executable of this program testcode.

This program would cause my server running the judge to freeze. Obviously i dont want that to happen.So to prevent that i tried using ptrace.I came up with the following code:(Lets call the executable of this code monitor)

#include <sys/ptrace.h>
#include <sys/types.h>
#include <sys/wait.h>
#include <unistd.h>
#include <string.h>
#include <errno.h>
#include <sys/user.h>
#include <sys/syscall.h>
#include <sys/reg.h>
#include<stdio.h>
#include <sys/ptrace.h>
#include <sys/types.h>
#include <sys/wait.h>
#include <unistd.h>
#include <string.h>
#include <errno.h>
#include <sys/user.h>
#include <sys/syscall.h>
#include <sys/reg.h>
#include<stdio.h>
#include<signal.h>
#include<sys/prctl.h>
#include<stdlib.h>
#define NOBANNEDSYS 40
int bannedsys[50]={2,14,12,15,26,37,38,39,39,40,41,42,46,47,48,49,50,60,61,63,72,83,88,120,102,182,183,190};

int main(int argc,char **argv) {
                int insyscall=0;
                if(argc!=2) {
                    fprintf(stderr,"Usage: %s <prog name> ",argv[0]);
                    exit(-1);
                    }

  int status = 0;
  int syscall_n = 0;
  int entering = 1;
  int amp;
  struct user_regs_struct regs;
  int pid = fork();

  if ( !pid ) {
  prctl(PR_SET_PDEATHSIG, SIGKILL);
    ptrace( PTRACE_TRACEME, 0, 0, 0 );
    execlp( argv[1],argv[1], 0 );
    }

  else {

    //ptrace( PTRACE_SINGLESTEP ,pid, 0, 0 );
    // ptrace( PTRACE_SYSCALL, pid, 0, 0 ); 
    while (1) {

        wait( &amp); 
            if ( WIFEXITED( amp ) ) break;

                //ptrace( PTRACE_SINGLESTEP ,pid, 0, 0 ); 

    if(insyscall==0){
        ptrace( PTRACE_GETREGS, pid, 0,&regs ); 
      int i=0;
      for(i=0;i<NOBANNEDSYS;i++) if(regs.orig_eax==bannedsys[i]) {  kill(pid,SIGKILL);
                                printf("%d killed due to illegal system call\n",pid);

                                    abort();
                                    }

        insyscall=1;
               }
       else insyscall=0;
      // ptrace(PTRACE_CONT,pid,0,0);
     // wait(&amp);

    ptrace( PTRACE_SYSCALL, pid, 0, 0 );
     // puts("Here");
//ptrace(PTRACE_CONT, pid, 0, 0);

    }

  }

  return 0;
}

This code works quite well in blocking system calls that may cause problem.But when the code to be monitored contains fork calls in a loop like the testcode the machine freezes because of thrashing.The reason i could make out is while the original process is killed by the monitor code its child survives and continues to carry the fork bomb. How to fix the monitor code so that it can be deployed successfully?

PS:Portability is not a concern.I am looking for a linux specific answer.

EDIT: I used setrlimit for setting the maximum number of child processes to 0 before calling exec.This till now appears to be a a good solution.Though it would be nice to hear from the community if there are still loopholes in the monitor code.

Answer 1

You can stop the child and trace new processes:

PTRACE_O_TRACEFORK (since Linux 2.5.46) Stop the child at the next fork(2) call with (SIGTRAP | PTRACE_EVENT_FORK << 8) and automatically start tracing the newly forked process, which will start with a SIGSTOP. The PID for the new process can be retrieved with PTRACE_GETEVENTMSG.

You can have a working example with these changes:

/* ... */
  ptrace(PTRACE_SETOPTIONS,pid,NULL, PTRACE_SYSCALL | PTRACE_O_TRACEFORK) ;
  while (1) {
    printf("Waiting\n");
    pid = wait(&amp);
    printf("Waited %d\n", amp);
    if (WIFEXITED(amp)) {
      break;
    }
    if (WSTOPSIG(amp) == SIGTRAP)
    {
      int event = (amp >> 16) & 0xffff;
      if (event ==  PTRACE_EVENT_FORK) {
        printf("fork caught\n");
        pid_t newpid;
        ptrace(PTRACE_GETEVENTMSG, pid, NULL, (long) &newpid);
        kill(newpid, SIGKILL);
        kill(pid, SIGKILL);
        break;
      }
    }
    if (insyscall == 0) {
      ptrace(PTRACE_GETREGS, pid, 0, &regs);
      int i = 0;
      for (i = 0; i < NOBANNEDSYS; i++) if (regs.orig_eax == bannedsys[i]) {
        kill(pid, SIGKILL);
        printf("%d killed due to illegal system call\n", pid);

        abort();
      }

      insyscall = 1;
    } else {
      insyscall = 0;
    }
    ptrace(PTRACE_CONT, pid, NULL, 0);
  }

Reference

Answer 2

For what you're doing, I would strongly recommend investigating seccomp, which will allow you to disallow a process from using any system calls at all other than exit, read and write (only to file descriptors which are already open), and sigreturn.