CODEWITHSUNDEEP
ncocdo-climate
Jonathan Wille
Jonathan Wille

Reputation: 45

How to concatenate using ncrcat or cdo merge with packed netcdf files

I have ERA5 files that I am trying to concatenate into monthly files. It appears the files have been packed to reduce size making the data type within the file a short. When I try ncrcat, it will warn about encountering a packing attribute "add_offset", then concatenate all the files together. However the values of the data become messed up. I tried using ncpdq -U to unpack the files, then ncrcat to concatenate which works. But the resulting files are too large to be useful and when I try ncpdq to repack the resulting file I receive a malloc() failure which seems related to a memory/RAM issue.

I've also tried cdo merge which strangely works perfectly for most of the concatenations, but a few of the files fail and output this error "Error (cdf_put_vara_double): NetCDF: Numeric conversion not representable"

So is there anyway to concatenate these files while they are still packed, or at least a way to repack the large files once they are concatenated

Upvotes: 1

Views: 3923

Answers (2)

ClimateUnboxed
ClimateUnboxed

Reputation: 8107

When data is packed, CDO will often throw an error due to too much loss of precision,

cdo -b32 mergetime in*.nc out.nc

should do the trick and avoid the error. If you want to then compress the files you can try this:

cdo -z zip_9 copy out.nc out_compressed.nc

Upvotes: 1

Charlie Zender
Charlie Zender

Reputation: 6352

Instead of repacking the large files once they are concatenated you could try netCDF4 compression, e.g.,

ncpdq -U -7 -L 1 inN.nc in_upk_cmpN.nc # Loop over N
ncrcat in_upk_cmp*.nc out.nc

Good luck!

Upvotes: 2

Related Questions