CODEWITHSUNDEEP
slurm
GoingMyWay
GoingMyWay

Reputation: 17468

SLURM, using srun to print outputs

I am using srun to run my program, however, it cannot print the output.

me@home:~$ srun -p K80q --gres=gpu:1 -N 1 python3 main.py 
2019-05-15 19:56:43.305156: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-05-15 19:56:43.543516: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1392] Found device 0 with properties: 
name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.8235
pciBusID: 0000:85:00.0
totalMemory: 11.17GiB freeMemory: 11.10GiB
2019-05-15 19:56:43.543567: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1471] Adding visible gpu devices: 0
2019-05-15 19:56:43.900189: I tensorflow/core/common_runtime/gpu/gpu_device.cc:952] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-05-15 19:56:43.900248: I tensorflow/core/common_runtime/gpu/gpu_device.cc:958]      0 
2019-05-15 19:56:43.900257: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] 0:   N 
2019-05-15 19:56:43.900619: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1084] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10761 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:85:00.0, compute capability: 3.7)

I only got the above output and it cannot print the information I expected. How can I fix it?

By the way, simply define a test code

import tensorflow 

if __name__ == '__main__':
    for i in range(10):
        print('Hello')

It can print Hello 10 times.

Update:

After 20 minutes, it outputs some information I expected. How can I make it output immediately?

Upvotes: 4

Views: 2382

Answers (1)

damienfrancois
damienfrancois

Reputation: 59110

Try the -u option of srun:

-u, --unbuffered By default the connection between slurmstepd and the user launched application is over a pipe. The stdio output written by the application is buffered by the glibc until it is flushed or the output is set as unbuffered. See setbuf(3). If this option is specified the tasks are executed with a pseudo terminal so that the application output is unbuffered.

Upvotes: 4

Related Questions