CODEWITHSUNDEEP
javalinuxbashshelltomcat
reynoldsnlp
reynoldsnlp

Reputation: 1210

bash command fails in tomcat/java Runtime.getRuntime.exec(), but works from command line

I have a tomcat webapp that runs a process in the shell because several of the utilities are not available in java. This code works perfectly on other machines, but there is a mysterious problem on my public server.

String[] textAnalysisPipeline = {
    "/bin/sh",
    "-c",
    "/bin/cat " + inputfileLoc +
    " | tee /tmp/debug1 | " + loadJar + " " + jarOptLookupLoc + " " + optHfstLoc +
    " 2>/dev/null | " + "tail -n+5" + // get rid of the header that hfst-ol.jar produces
    " | tee /tmp/debug2 | cut -f 1-2" + // get rid of the "0.0" weights
    " | tee /tmp/debug3 | " + cgConvLoc +
    " | tee /tmp/debug4 | " + vislcg3Loc + " -g " + vislcg3DisGrammarLoc +  // disambiguate with the constraint grammar
    " | tee /tmp/debug5 > " + outputfileLoc};
log.debug("Text analysis pipeline: "+textAnalysisPipeline[2]);
Process process = Runtime.getRuntime().exec(textAnalysisPipeline);
process.waitFor();

I print the string to the log, and it looks like this: (path/to/ is not the actual paths)

/bin/cat /path/to/inputFile | tee /tmp/debug1 | java -jar /path/to/hfst-ol.jar /path/to/analyser.ohfst 2>/dev/null | tail -n+5 | tee /tmp/debug2 | cut -f 1-2 | tee /tmp/debug3 | /usr/local/bin/cg-conv | tee /tmp/debug4 | /path/to/vislcg3 -g /path/to/grammar.rlx | tee /tmp/debug5 > /path/to/outputFile

If I copy this pipeline from the log and run it from the bash command line, I get the desired output all the way to the end of the pipeline. However, when the tomcat server runs the command, it produces an empty file. The debug files debug1 and debug2 are as expected, but debug3 and thereafter is empty, which suggests that the pipeline fails at cut -f 1-2 (see UPDATE 1 below).

OS - Fedora 22
java - openjdk 1.8.0_77
tomcat - 7.0.39
sh --> bash - 4.3.42(1)-release

================================================================
UPDATE 1:

This does not seem to be a problem with cut. I wrote a short python script, cut.py to achieve the same functionality as cut -f 1-2 (remove '\t0.0' from the end of each line)

import re, sys
myRE = re.compile( r'\s+0\.0$' )
for line in sys.stdin :
    sys.stdout.write( myRE.sub( '', line ) )

Using cut.py in place of cut, I get the same problem. With the server debug3 and beyond is empty, but if I copy-paste from the log to the interactive shell, everything works fine.

================================================================
UPDATE 2:

I also wrote a simple bash script to run the pipeline so that tomcat/java only runs the bash script with one argument for the input/output filename. If I run the script from an interactive shell, it works, but the results are no different in tomcat, using cut or cut.py in the shell script.

Upvotes: 2

Views: 2090

Answers (5)

FJunqueira
FJunqueira

Reputation: 108

By default installation on most systems, Tomcat does not have the same environment as your user. It does not run, for instance, your .bashrc .profile or whatever you have in your login script, so all variables set in your user shell environment are different.

You can check that by comparing the env command with both users: yours and by having it called by your java program, something like:

String[] textAnalysisPipeline = {"/bin/sh","-c","/usr/bin/env > /tmp/env.txt"}; //or wherever the 'env' command is in your system
Process process = Runtime.getRuntime().exec(textAnalysisPipeline);
...

And compare the content of the /tmp/env.txt with the env execution with your user... they are probably very different.

Look for the following variables:

  • PATH
  • CLASSPATH
  • JAVA_HOME

I have already the same problem in the past. I suggest the following approach:

  1. Use absolute path for everything, including the calls for "java", "tee", "tail" and libs (jar files)... you have in your command;

  2. Change the environment configuration under which your Tomcat runs to reach all the applications you call in your command (usually by calling a script that configures all the necessary PATH variable (do not forget the JAVA_HOME and CLASSPATH to your jar files!). Check the Startup.sh and Catalina.sh for a suitable place to include your stuff;

  3. Change your command to redirect the error output not to /dev/null as it is in your message, but to some log file somewhere in your system that your application can write (usually /tmp/exec.log is just fine) so you can be sure of the shell execution problem. I bet it will be something like: sh: ****: command not found or Error: Unable to access jarfile or some message of your application not being able to locate an object, so you will be sure of what application you call in your script is not in the PATH environment variable or what libs you don't have in it... usually both

For additional info on this, check https://www.mulesoft.com/tcat/tomcat-classpath

Hope this helps...

Upvotes: 3

CHW
CHW

Reputation: 2664

When you copy the command from the log and run it from the command line, do you execute it as the same user than the user running Tomcat?

The Tomcat user might not have enough permissions to read one of the files in your command.

On my Ubuntu system, the default Tomcat 7 user is called tomcat7. Running a command as another user can be done using the sudo command :

sudo -u tomcat7 /bin/sh -c 'myCommand'

Source : How do you process an entire command as sudo that involves redirecting or piping?.

Upvotes: 0

user2832874
user2832874

Reputation:

There's a pattern in scripts and (sometimes especially) crontabs where the $PATH variable that you are using for your script differs from what you are using for your interactive shell. This results in commands failing mysteriously in one environment but working elsewhere.

I would strongly recommend embedding the full path of all of the tools that you are using into the invocation that you are building. It's possible that cut is in a directory that is not picked up when you run that process. Putting in the full path like /usr/bin/cut will make certain two things: that no one sneaks in an unexpected version of cut into your path, and also that it won't matter if the environment you are in doesn't have /usr/bin in $PATH.

Upvotes: 0

tripleee
tripleee

Reputation: 189317

This is mainly an aside or perhaps a workaround, but your pipeline can be simplified significantly. The cat is useless and the tail and cut can be replaced with a single Awk script.

tee /tmp/debug1 <path/to/inputFile |
java -jar path/to/hfst-ol.jar path/to/analyser.ohfst 2>/dev/null |
tee /tmp/debug2 |
awk -F '\t' 'NR>5 { print $1 FS $2 }' |
tee /tmp/debug3 |
/usr/local/bin/cg-conv |
tee /tmp/debug4 |
path/to/vislcg3 -g path/to/grammar.rlx |
tee /tmp/debug5 > path/to/outputFile

Upvotes: 3

SkyWalker
SkyWalker

Reputation: 29130

You can use ProcessBuilder. Use the single-string form of Runtime.exec is not a good style. Better option is to use ProcessBuilder, and split up the arguments yourself rather than relying on Java to split them for you which it does very naively.

ProcessBuilder pb = new ProcessBuilder("/bin/bash", "-c", /*...*/);
pb.redirectErrorStream(true);
Process p = pb.start();

Upvotes: 1

Related Questions