CODEWITHSUNDEEP
bashubuntuwindows-subsystem-for-linuxncocdo-climate
WDS
WDS

Reputation: 337

Faster way of Appending/combining thousands (42000) of netCDF files in NCO

I seem to be having trouble properly combining thousands of netCDF files (42000+) (3gb in size, for this particular folder/variable). The main variable that i want to combine has a structure of (6, 127, 118) i.e (time,lat,lon) Im appending each file 1 by 1 since the number of files is too long.

I have tried:

for i in input_source/**/**/*.nc; do ncrcat -A -h append_output.nc $i append_output.nc ; done

but this method seems to be really slow (order of kb/s and seems to be getting slower as more files are appended) and is also giving a warning:

ncrcat: WARNING Intra-file non-monotonicity. Record coordinate "forecast_period" does not monotonically increase between (input file file1.nc record indices: 17, 18) (output file file1.nc record indices 17, 18) record coordinate values 6.000000, 1.000000

that basically just increases the variable "forecast_period" 1-6 n-times. n = 42000files. i.e. [1,2,3,4,5,6,1,2,3,4,5,6......n] And despite this warning i can still open the file and ncrcat does what its supposed to, it is just slow, at-least for this particular method

I have also tried adding in the option:

--no_tmp_fl

but this gives an eror:

ERROR: nco__open() unable to open file "append_output.nc"

full error attached below

If it helps, im using wsl and ubuntu in windows 10. Im new to bash and any comments would be much appreciated.

enter image description here

Upvotes: 2

Views: 1291

Answers (2)

Charlie Zender
Charlie Zender

Reputation: 6322

Either of these commands should work:

ncrcat --no_tmp_fl -h *.nc

or

ls input_source/**/**/*.nc | ncrcat --no_tmp_fl -h append_output.nc

Your original command is slow because you open and close the output files N times. These commands open it once, fill-it up, then close it.

Upvotes: 1

Basileios
Basileios

Reputation: 325

I would use CDO for this task. Given the huge number of files it is recommended to first sort them on time (assuming you want to merge them along the time axis). After that, you can use 

cdo cat *.nc outfile

Upvotes: 1

Related Questions