CODEWITHSUNDEEP
pythonshellpipekaldi
Green 绿色
Green 绿色

Reputation: 2956

Python and Kaldi: Broken Pipe Error

The setup is the following: I am working with Kaldi tools. They are called in bash scripts of a certain form. Those bash scripts are called by a wrapper program written in python that submits the tasks to a Sun Grid Engine.

The command I want to execute is the following 

feat-to-dim 'ark:copy-feats scp:train.scp ark:- |' -

Executing this in the command line, yields the correct result '40' together with the warning 

WARNING (feat-to-dim[5.0.23-f7b2f]:Close():kaldi-io.cc:501) Pipe copy-feats scp:train.scp ark:- | had nonzero return status 13

However, if I wrap this in the following way:

python -c "import os; os.system(\"feat-to-dim 'ark:copy-feats scp:train.scp ark:- |' -\")"

the program copy-feats fails with the error:

ERROR (copy-feats[5.0.23-f7b2f]:Write():kaldi-matrix.cc:1240) Failed to write matrix to stream

After various stack traces and following errors, the following warning is printed additionally: 

WARNING (feat-to-dim[5.0.23-f7b2f]:Close():kaldi-io.cc:501) Pipe copy-feats scp:train.scp ark:- | had nonzero return status 134

This is what I've found out: feat-to-dim closes the pipe in advance whereas copy-feats tries to continue writing output. Since it is not possible, copy-feats is terminated. 13 probably indicates the broken pipe error.

For Python, this is a serious issue why it turns it into an error and terminates. But in this case, it's not Python which yields this error, but copy-feats. Therefore, stuff like try/catch in python or trapping the pipe seem not to have any success in this case.

Moreover, the following lines work perfectly fine without any warnings or errors:

python -c "import os; os.system(\"copy-feats scp:train.scp ark:-\")" > cp
python -c "import os; os.system(\"feat-to-dim ark:cp -\")"

and the following line yields the simple error message cat: write error: Broken pipe and exit status 256:

python -c "import os; os.system(\"feat-to-dim ark:'cat cp |' -\")"

Do you have any further suggestions here?

Upvotes: 0

Views: 1147

Answers (1)

Nikolay Shmyrev
Nikolay Shmyrev

Reputation: 25220

13 probably indicates the broken pipe error.

13 is an answer (feature dimension), you can use it as a result, you can ignore all other errors like this

with open(os.devnull, "w") as devnull:
    subprocess.call("feat-to-dim 'ark:copy-feats scp:feats.scp ark:- |' -", shell=True, stderr=devnull)

This is what I've found out: feat-to-dim closes the pipe in advance whereas copy-feats tries to continue writing output.

This is a kaldi design, it tries to read just first feature and simply dumps the rest but since the pipe has no mean to terminate writer child, it has to exit in such a bad way. An option is to read full child output in feat-to-dim, but that will be slower.

You can probably open Kaldi bug about that.

Upvotes: 0

Related Questions