CODEWITHSUNDEEP
netcdf
kati 95
kati 95

Reputation: 21

how to combine some NETCDF file in a single one

I have 3 netcdf files which are very huge and I want to merge them all in a single .nc file according to the time dimension. I have tried to combine them by xarray but it wasn't a good way because my PC could not handle huge data and I got this error: MemoryError: Unable to allocate 17.9 GiB for an array with shape (73, 10000, 10000) and data type float32. I want to know if there is another way to deal with the netcdf files directly and combine them without using the xarrays and not to encounter memory problems again? here is the code I have used:

path1 = '/final1_netcdf_VH.nc'
path2 = '/all/final2_netcdf_VH.nc'
path3 = '/final3_netcdf_VH.nc'

xarr1 = xarray.open_dataset(path1)
xarr2 = xarray.open_dataset(path2)
xarr3 = xarray.open_dataset(path3)

ultimate = xarray.merge([xarr1, xarr2, xarr3], compat="equals")

ultimate.to_netcdf('/ultimate_netcdf_VH.nc')

Upvotes: 1

Views: 1485

Answers (1)

Robert Wilson
Robert Wilson

Reputation: 3397

You should be able to do this using my nctoolkit package (https://nctoolkit.readthedocs.io/en/latest/). The following should work:

import nctoolkit as nc
path1 = '/final1_netcdf_VH.nc'
path2 = '/all/final2_netcdf_VH.nc'
path3 = '/final3_netcdf_VH.nc'

ultimate = nc.open_data([path1, path2, path3])
ultimate.merge_time()
ultimate.to_nc('/ultimate_netcdf_VH.nc')

This uses CDO as a backend. If you are comfortable with the command line and have CDO installed, you would just need to do this:

cdo -mergetime /final1_netcdf_VH.nc /all/final2_netcdf_VH.nc /final3_netcdf_VH.nc /ultimate_netcdf_VH.nc

Upvotes: 1

Related Questions