How do I ingest tide gauge data from the NOAA HTTP API into Thingsboard Professional?

Question

NOAA provides tidal and weather data through their own http API, and I would like to be able to use their API to get data into ThingsBoard (Professional) every six minutes to overlay with my device data (their data are updated every 6 minutes). Can someone walk me through the details of using the correct Integrations or Rule chains to get the time series data added to the database? It would also be nice to only use the metadata once. Below you can see how to get the most recent tide gauge level (water level) using their API.

For example, to see the latest tide gauge water level for a tide gauge (in this case, tide gauge 8638610), the API allows for getting the most recent water level information -- https://api.tidesandcurrents.noaa.gov/api/prod/datagetter?date=latest&station=8638610&product=water_level&datum=navd&units=metric&time_zone=lst_ldt&application=web_services&format=json

That call produces the following JSON: {"metadata":{"id":"8638610","name":"Sewells Point","lat":"36.9467","lon":"-76.3300"},"data":[{"t":"2022-02-08 22:42", "v":"-0.134", "s":"0.003", "f":"1,0,0,0", "q":"p"}]}

The Data Converter was fairly easy to construct (except maybe the noaa_data.data[0, 0] used in the code below):

//function Decoder(payload,metadata)
var noaa_data = decodeToJson(payload);
var deviceName = noaa_data.metadata.id;
var dataType = 'water_level';
var latitude = noaa_data.metadata.lat;
var longitude = noaa_data.metadata.lon;
var waterLevelData = noaa_data.data[0, 0];

//functions
function decodeToString(payload) {
    return String.fromCharCode.apply(String, payload);
}

var result = {
    deviceName: deviceName,
    dataType: dataType,
    time: waterLevelData.t,
    waterLevel: waterLevelData.v,
    waterLevelStDev: waterLevelData.s,
    latitude: latitude,
    longitude: longitude
}

function decodeToJson(payload) {
    var str = decodeToString(payload);
    var data = JSON.parse(str);
    return data;
}

return result;

which has an Output:

{
"deviceName": "8638610",
"dataType": "water_level",
"time": "2022-02-08 22:42",
"waterLevel": "-0.134",
"waterLevelStDev": "0.003",
"latitude": "36.9467",
"longitude": "-76.3300"
}

I am not sure what process to use to get the data into ThingsBoard to be displayed as a device alongside my other device data.

Thank you for your help.

Answer 1

If you have a specific(and small) number of stations to grab then you can do the following:

1. Create the devices in Thingsboard manually
2. Go into rule chains, create a water stations rule chain
3. For each water station place a 'Generator' node, selecting the originator as required.
4. Route these into an external "Rest API" node.
5. Route the result of the post into a blue script node and put your decoder in there
6. Route result to telemetry

Example rule chain

More complex solution but more scalable:

1. Use a single generator node
2. Route the message into a blue script. This will contain a list of station id's that you want to pull info for. By setting the output of a script to the following you can make it send out multiple messages in sequence:

return [{msg:{}, metadata:{}, msgType{}, ...etc...]

3. Route the blue script into the rest api call and get the station data
4. Do some post processing with another blue script node if you need to. Don't decode the data here though.
5. Route all this into another rest api node and POST the data back to your HTTP integration endpoint (if you don't have one you will need to create it. Fairly simple)
6. Connect your data converter to this integration.
7. Finally, modify your output so that it is accepted by the converter output

{
"deviceName": "8638610",
"deviceType": "water-station",
"telemetry": {
        "dataType": "water_level",
        "time": "2022-02-08 22:42",
        "waterLevel": "-0.134",
        "waterLevelStDev": "0.003",
        "latitude": "36.9467",
        "longitude": "-76.3300"
    }
}

Rough example

Above is how I would do it if I didn't want to use any external services. If you're AWS savvy I'd say set up a CRON job to trigger a lambda function every 6 minutes and post into your platform. Either will work.