CODEWITHSUNDEEP
pythonduplicatesnetcdfdimensionpython-xarray
StubBurn
StubBurn

Reputation: 126

Dimension coordinates has fill values and hence, values are repeating

I received some data files (NetCDF) from a colleague and I am trying to concatenate these files into a single file, so that I am able to run it with some of my previous scripts. The issue is that the NetCDF files I received have dimension "height" with fill values of 1e20. This is causing an error when I try almost any xarray operations on the files, because there are duplicate values in the dimension coordinates.

Here is the info on one of the files (you can see the repeating values in the final values of the "height" coordinates): 

<xarray.Dataset> Dimensions:  (height: 1061) 
Coordinates:   * height   (height) float64 0.0 10.0 20.0 30.0 ... 1e+20 1e+20 1e+20 1e+20 
Data variables:
            pres     (height) float32 ...
            tdry     (height) float32 ...
            rh       (height) float32 ...
            u_wind   (height) float32 ...
            v_wind   (height) float32 ...
            mr       (height) float32 ...
            theta    (height) float32 ...
            theta_e  (height) float32 ...
            theta_v  (height) float32 ...
            lat      (height) float32 ...
            lon      (height) float32 ...
            alt      (height) float32 ...
            wdir     (height) float32 ...

I thought may be reindexing to a new index with no fill values might help, but I am also unable to do that as the error pops up again that:

ValueError: cannot reindex or align along dimension 'height' because the index has duplicate values

If someone could help me out here, it would be a great favour! I have been struggling with this for a while, but maybe the solution is quite simple and my beginner's status is not really being helpful here. :/

Upvotes: 1

Views: 697

Answers (2)

Keisuke FUJII
Keisuke FUJII

Reputation: 1406

How about removing the duplicate entries as suggested in this post?

.isel method should work even if coordinates have duplicated entries. The below script may work

_, index = np.unique(ds['time'], return_index=True)
ds.isel(time=index)

Upvotes: 1

ClimateUnboxed
ClimateUnboxed

Reputation: 8087

This is not a python solution but I was wondering if CDO or NCO might not help you resolve this easier/faster?

If the files are for different times you could try 

cdo mergetime input_t*.nc output.nc

(* wildcard for the names)

More generally you can try to cat files with:

cdo cat input_t*.nc output.nc

I'm not sure how cdo will deal with the missing height coordinates though

You can also append files in nco with 

ncks -A appended_file.nc target_file.nc

Not sure which (if any) of these solutions will work, but I hope one might help.

Upvotes: 0

Related Questions