CODEWITHSUNDEEP
csvnetcdfcdo-climate
Brian
Brian

Reputation: 87

Problems converting netCDF to CSV file using CDO

This post is very long because I want to explain better the context.

My main data source is in netCDF format and I want to convert into CSV file.

For a while, I've been using Python to convert. As an example, I use a netCDF data previously modified (m > mm, hourly > daily) and then converted into a CSV file. This is how it looks like:

enter image description here

It can be noted that:

As shown in the image above is the desired data presentation.

After, I decided to use Climate Data Operator (CDO) because it was more easy to apply some functions. Inside Ubuntu on WSL, I converted netCDF files into CSV ones. These were the codes I used:

First code

cdo -outputtab,date,lat,lon,value era5land_total_precipitation_daily_feb-nov_2017_mm.nc > test-1_tp.csv

The output data is shown as

enter image description here

The first row where variable names are located is preceded by "#" when it shouldn't be there. Also, values are delimited by blank spaces.

Second code

cdo -outputtab,date,lat:6,lon:6,value:8 era5land_total_precipitation_daily_feb-nov_2017_mm.nc | grep -v '#' | sed -e 's/ */,/g' >> test-1_tp.csv

Third code

cdo -outputtab,date,lat:6,lon:6,value:8 era5land_total_precipitation_daily_feb-nov_2017_mm.nc | sed 's/[[:space:]]/,/g' > test-1_tp.csv

Last two lines of code show an equal output

enter image description here

The problem is partially solved with delimited values by commas although in some observations the last two values are still divided by a blank space. Also, there's no variable names at the top.

Fourth code

cdo -outputtab,date,lat:6,lon:6,value:8 era5land_total_precipitation_daily_feb-nov_2017_mm.nc | awk 'FNR==1{ row=$2","$3","$4","$5; print row } FNR1=1{ row=$1","$2","$3","$4; print row}' > test-1_tp.csv

The output contains

enter image description here

The result of the last code is the closest to what I want to obtain except second row in the content of the image needs to be deleted and latitude/longitude values are still rounded. Any Suggestion to obtain a dataset like image 1?

Extra help: does anyone know the meaning of these pieces of codes?

...| grep -v '#' | sed -e 's/ */,/g'

...| sed 's/[[:space:]]/,/g'

...| awk 'FNR==1{ row=$2","$3","$4","$5; print row } FNR1=1{ row=$1","$2","$3","$4; print row}'

Thanks a lot in advance!

Upvotes: 2

Views: 1926

Answers (1)

Brian
Brian

Reputation: 87

I asked in another forum the same question and I received the following solution:

cdo -outputtab,date,lat:6,lon:6,value:8 infile.nc | grep -v '#' | tr -s ' ' | sed -e 's/ /,/g;s/^.//;s/.$//' >> outfile.csv

Keep in mind

  • first row contains generic names of variables: v1, v2, v3 and v4 instead of date, latitude, longitude and tp.
  • all values are delimited by commas.

Upvotes: 2

Related Questions