CODEWITHSUNDEEP
pythonnetcdfcdo-climate
jairovs
jairovs

Reputation: 13

problem with regriding a netCDF file using nctoolkit

I am using daily data to calculate monthly averages using ensemble_mean. Once I have the file with the monthly average, I regrid the file from 0.1 to 0.25 degrees using another file as the target grid. The ensemble mean goes well, but when trying to regrid the file I get the following error:

ValueError: CDO error: Error (cdf_put_vara_double): NetCDF: Numeric conversion not representable. Tip: check if missing values are incorrectly set to large actual values!

This happens only in certain months. For some others, the regridding process works perfectly.

The code I am using is:

import nctoolkit as nc

ds = nc.open_data("/home/omi_data/HCHO/data/2006/12/*.nc4")
ds1=nc.open_data("/home/omi_data/NO2/data/2006/07/OMI-Aura_L3-OMNO2d_2006m0702_v003-2019m1121t032327.he5.ncml.nc4")
ds.ensemble_mean('key_science_data_column_amount')
ds.regrid(ds1)
ds.to_nc('/home/omi_data/HCHO/data/2006/monthly_average/HCHO_0612.nc4')

Data link

Upvotes: 1

Views: 601

Answers (2)

Robert Wilson
Robert Wilson

Reputation: 3397

This problem appears to be caused by issues in the raw data. The netCDF files say the data format is F32. However, one of the files actually has data values that are outside the maximum range accepted by F32. That's a mistake during file creation. This is causing problems in CDO when nctoolkit calls it. As the error said, you have data can cannot be represented with 32-bit. Essentially, what you will have to do is correct the raw data before processing it. Just set anything outside the valid range to NA. The following should work:

ds = nc.open_data("/home/omi_data/HCHO/data/2006/12/*.nc4")
ds.as_missing([3.40282347E+38, 1e50])
ds.as_missing([-1e50, -3.40282347E+38])
ds1=nc.open_data("/home/omi_data/NO2/data/2006/07/OMI-Aura_L3- 
OMNO2d_2006m0702_v003-2019m1121t032327.he5.ncml.nc4")
ds.ensemble_mean('key_science_data_column_amount')
ds.regrid(ds1)
ds.to_nc('/home/omi_data/HCHO/data/2006/monthly_average/HCHO_0612.nc4')

Alternatively, you could also change the units to something more sensible. You really don't want to work with files with such large values, as you will easily run into limits with whatever numerical precision you are working with. However, the units are slightly confusing. The file says "molecules/cm^2". However, you can get positive and negative values. I don't understand that, so I can't provide guidance on changing the units.

Upvotes: 0

ClimateUnboxed
ClimateUnboxed

Reputation: 8077

When using cdo from the command line this error is due to a precision change, and is resolved by converting shorts to floats using the option "-b f32".

This is a command-line based quick fix, but I'm sure Robert can come up with a better fix within his package itself.

for file in /home/omi_data/HCHO/data/2006/12/*.nc4 ; do cdo -b f32 -f nc4 copy $file ${file%????}_flt.nc4 ; done

This converts all the files to files with floats, and then in your python code you need to refer to

/home/omi_data/HCHO/data/2006/12/*_flt.nc4

to ensure you only pick up the converted files. As I said, a clunky quick fix. EDIT: I'm pasting in Robert's comment from above, so this credit is his, you can change precision in his package by using

ds.set_precision("F32")

You can also do this from within python itself using the cdo package, I think this is the equivalent (untested, I hope the wildcard works like this.)

from cdo import *
cdo = Cdo()
ifile="/home/omi_data/HCHO/data/2006/12/*.nc4"
gridfile="/home/omi_data/NO2/data/2006/07/OMI-Aura_L3-OMNO2d_2006m0702_v003-2019m1121t032327.he5.ncml.nc4"

cdo.ensmean(input = ifile, output = "ensmean.nc", options = '-b f32')
cdo.remapbil(gridfile,input="ensmean.nc", output="ensmean_regrid.nc")

Upvotes: 1

Related Questions