Reading Time Series from netCDF with python

Question

I'm trying to create time series from a netCDF file (accessed via Thredds server) with python. The code I use seems correct, but the values of the variable amb reading are 'masked'. I'm new into python and I'm not familiar with the formats. Any idea of how can I read the data?

This is the code I use:

import netCDF4
import pandas as pd
import datetime as dt
import matplotlib.pyplot as plt
from datetime import datetime, timedelta # 

dayFile = datetime.now() - timedelta(days=1)
dayFile  = dayFile.strftime("%Y%m%d")

url='http://nomads.ncep.noaa.gov:9090/dods/nam/nam%s/nam1hr_00z' %(dayFile)

# NetCDF4-Python can open OPeNDAP dataset just like a local NetCDF file
nc = netCDF4.Dataset(url)
varsInFile = nc.variables.keys()

lat = nc.variables['lat'][:]
lon = nc.variables['lon'][:]
time_var = nc.variables['time']
dtime = netCDF4.num2date(time_var[:],time_var.units)

first = netCDF4.num2date(time_var[0],time_var.units)
last = netCDF4.num2date(time_var[-1],time_var.units)
print first.strftime('%Y-%b-%d %H:%M')
print last.strftime('%Y-%b-%d %H:%M')

# determine what longitude convention is being used
print lon.min(),lon.max()

# Specify desired station time series location
# note we add 360 because of the lon convention in this dataset
#lati = 36.605; loni = -121.85899 + 360.  # west of Pacific Grove, CA
lati = 41.4; loni = -100.8 +360.0  # Georges Bank

# Function to find index to nearest point
def near(array,value):
idx=(abs(array-value)).argmin()
return idx

# Find nearest point to desired location (no interpolation)
ix = near(lon, loni)
iy = near(lat, lati)
print ix,iy

# Extract desired times.  

# 1. Select -+some days around the current time:
start = netCDF4.num2date(time_var[0],time_var.units)
stop = netCDF4.num2date(time_var[-1],time_var.units)
time_var = nc.variables['time']
datetime = netCDF4.num2date(time_var[:],time_var.units)

istart = netCDF4.date2index(start,time_var,select='nearest')
istop = netCDF4.date2index(stop,time_var,select='nearest')
print istart,istop

# Get all time records of variable [vname] at indices [iy,ix]
vname = 'dswrfsfc'

var = nc.variables[vname]

hs = var[istart:istop,iy,ix]

tim = dtime[istart:istop]

# Create Pandas time series object
ts = pd.Series(hs,index=tim,name=vname)

The var data are not read as I expected, apparently because data is masked:

>>> hs
masked_array(data = [-- -- -- ..., -- -- --],
             mask = [ True  True  True ...,  True  True  True],
       fill_value = 9.999e+20)

The var name, and the time series are correct, as well of the rest of the script. The only thing that doesn't work is the var data retrieved. This is the time serie I get:

>>> ts
2016-10-25 00:00:00.000000   NaN
2016-10-25 01:00:00.000000   NaN
2016-10-25 02:00:00.000006   NaN
2016-10-25 03:00:00.000000   NaN
2016-10-25 04:00:00.000000   NaN
...   ...    ...   ...   ... 
2016-10-26 10:00:00.000000   NaN
2016-10-26 11:00:00.000006   NaN
Name: dswrfsfc, dtype: float32

Any help will be appreciated!

Answer 1

I checked your approach with xarray. Works great to extract Solar radiation data! I can add that the first point is not defined (NaN) because the model starts calculating there, so there is no accumulated radiation data (to calculate hourly global radiation). So that is why it is masked.

Something everyone overlooked is that the output is not correct. It does look ok (at noon= sunshine, at nmidnight=0, dark), but the daylength is not correct! I checked it for 52 latitude north and 5.6 longitude (east) (November) and daylength is at least 2 hours too much! (The NOAA Panoply viewer for Netcdf databases gives similar results)

Answer 2

Hmm, this code looks familiar. ;-)

You are getting NaNs because the NAM model you are trying to access now uses longitude in the range [-180, 180] instead of the range [0, 360]. So if you request loni = -100.8 instead of loni = -100.8 +360.0, I believe your code will return non-NaN values.

It's worth noting, however, that the task of extracting time series from multidimensional gridded data is now much easier with xarray, because you can simply select a dataset closest to a lon,lat point and then plot any variable. The data only gets loaded when you need it, not when you extract the dataset object. So basically you now only need:

import xarray as xr 

ds = xr.open_dataset(url)  # NetCDF or OPeNDAP URL
lati = 41.4; loni = -100.8  # Georges Bank

# Extract a dataset closest to specified point
dsloc = ds.sel(lon=loni, lat=lati, method='nearest')

# select a variable to plot
dsloc['dswrfsfc'].plot()

Full notebook here: http://nbviewer.jupyter.org/gist/rsignell-usgs/d55b37c6253f27c53ef0731b610b81b4