Problems combing fetch and res

Question

I am currently trying to add 10 different values in my database through a forloop and then respond with "res 200". However it seems like res 200 runs before the loop have even started and I keep getting error "Cannot set headers after they are sent to the client" when moving it arround.

My code:

app.post("/api/v1/getWeatherData",async(req,res)=>{
  let sources = await db.query("SELECT * FROM weather where element = 'mean(air_temperature P1D)' LIMIT 10;")
  console.log(sources.rows)
  try{
    let count = 1
    sources.rows.map(async(source)=>{
      sleep(5000)
      fetch(`https://frost.met.no/observations/v0.jsonld?sources=${source.source_id}&referencetime=${(source.valid_from).split("T")[0]}%2F2022-02-20&elements=mean(air_temperature%20P1D)&fields=value%2C%20referenceTime`, {
              method: "get",
              body: JSON.stringify(),
              headers: {
                Authorization:
                  "MYSECRET",
              },
            })
              .then((res) => res.json())
              .then(async tempData => {
                /* tempData.data.map(async (currentWeatherData)=>{
                  await db.query("INSERT INTO weather_data(weather_id,element,time,value) values ($1,'mean(air_temperature P1D)',$2,$3);",[input.rows[0].weather_id,currentWeatherData.referenceTime.split("T")[0],parseInt(currentWeatherData.observations[0].value)])
                }) */
                console.log("completed"); 
                count+=1
                console.log("happend")
    
        })
    })
if (count >=10){
    console.log(count)
    res.status(200).json({
      status: "success",
      data: {
        value: "Oppdatert",
      },
    });
  }
  }
  catch(err){console.log(err)}
})```

Answer 1

You have to wait for the fetch() operations to be done before your counter will be valid. The simplest way to do that is to sequence them by using await like this:

app.post("/api/v1/getWeatherData", async (req, res) => {
    try {
        let sources = await db.query(
            "SELECT * FROM weather where element = 'mean(air_temperature P1D)' LIMIT 10;")
        let count = 1;
        for (let source of sources.rows) {
            await sleep(5000);
            let res = await fetch(
                `https://frost.met.no/observations/v0.jsonld?sources=${source.source_id}&referencetime=${(source.valid_from).split("T")[0]}%2F2022-02-20&elements=mean(air_temperature%20P1D)&fields=value%2C%20referenceTime`, {
                    method: "get",
                    headers: {
                        Authorization: "MYSECRET",
                    },
                });
            let tempData = await res.json();
            /* tempData.data.map(async (currentWeatherData)=>{
              await db.query("INSERT INTO weather_data(weather_id,element,time,value) values ($1,'mean(air_temperature P1D)',$2,$3);",[input.rows[0].weather_id,currentWeatherData.referenceTime.split("T")[0],parseInt(currentWeatherData.observations[0].value)])
            }) */
            console.log("completed");
            count += 1
            console.log("happend")

        }
        if (count >= 10) {
            console.log(count)
            res.status(200).json({
                status: "success",
                data: {
                    value: "Oppdatert",
                },
            });
        }
    } catch (err) { 
        console.log(err);
        res.sendStatus(500);
    }
});

Summary of Changes:

1. Add await to fetch() and to res.json()
2. Move try/catch up to catch all errors, including on the await db.query()
3. Remove the body argument from the fetch() call as no body should be sent with the GET request.
4. Add res.sendStatus(500) to the catch handler so a response of some kind is always sent. You can obviously change that to whatever response is appropriate when there's an error, but you must send some response.
5. Add await in front of sleep(5000) since it wasn't actually doing anything without that. Note, this means it will take 50 seconds (5 * 10) to send a response back to the client.

Note:

In the case where count is not >= 10, you probably need an else statement and send some sort of response there because all paths through this code must send some sort of http response. I'm not sure what you're trying to test for there.

---

If you want to and can run all the requests in parallel, you could also do that by accumulating the promises and using Promise.all() to know when they are all done like this:

app.post("/api/v1/getWeatherData", async (req, res) => {
    try {
        let sources = await db.query(
            "SELECT * FROM weather where element = 'mean(air_temperature P1D)' LIMIT 10;")
        let count = 1;
        await Promise.all(sources.rows.map(async source => {
            let res = await fetch(
                `https://frost.met.no/observations/v0.jsonld?sources=${source.source_id}&referencetime=${(source.valid_from).split("T")[0]}%2F2022-02-20&elements=mean(air_temperature%20P1D)&fields=value%2C%20referenceTime`, {
                    method: "get",
                    headers: {
                        Authorization: "MYSECRET",
                    },
                });
            let tempData = await res.json();
            /* tempData.data.map(async (currentWeatherData)=>{
              await db.query("INSERT INTO weather_data(weather_id,element,time,value) values ($1,'mean(air_temperature P1D)',$2,$3);",[input.rows[0].weather_id,currentWeatherData.referenceTime.split("T")[0],parseInt(currentWeatherData.observations[0].value)])
            }) */
            console.log("completed");
            count += 1
            console.log("happend")
        }));
        if (count >= 10) {
            console.log(count)
            res.status(200).json({
                status: "success",
                data: {
                    value: "Oppdatert",
                },
            });
        }
    } catch (err) {
        console.log(err);
        res.sendStatus(500);
    }
});