CODEWITHSUNDEEP
phphtmlpostgresqlcurlweb-development-server
XCeptable
XCeptable

Reputation: 1267

Fetch data from a remote server periodically using curl

I am trying to fetch JSON data from a remote server using curl and save it in the database. The data on the remote server is updated each minute.

The curl script is running fine with fetching the data. But it is fetching data around every 10 seconds. What I want is it fetches the data once in a minute.

I searched SO and found that cronjob and windows task scheduler is the solutions provided for the scheduled jobs using curl. My problem is I need not run the web page containing curl so it may start fetching data from the server but I have to read data once in a minute.

Because the data is updated each minute's 5th second like 9:10:05 -> 9:11:05 ->9:12:05 ->... One option is to use cronjob OR windows scheduler to run the web page at e.g. 3rd second and close at e.g. 7th second so each minute one value is read. But this doesn't seem a good or reliable solution for me or is it?

My question is how do I read the data from the remote server once in a minute using curl. Is there some programming solution or using cronjob/windows task scheduler is the choice?

I am doing development on Windows workstation but I need to deploy it on server running Ubuntu.

EDIT

I have 2 more questions about my code:

1) As I stated in OP above, the curl is fetching data constantly from the source like it is an infinite loop running. Is this default behavior of curl or I need to do some configuration to fetch the data e.g. once through it ?

2) I need to keep the browser window opened so the curl start fetching the data. It doesn't work if the webpage is not opened. Is it default behavior of curl or it can also run as backend job so the webpage containing it should not be kept opened all the time ?

Here is my code:

Index.PhP

HTML + PHP + JS for data input and display

Connect.PhP

Database connection

RequestData.PhP

<!DOCTYPE html>
<html lang="en">
<head>
    <title>Weather Data</title>
    <meta charset="utf-8">
    <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.3/jquery.min.js"></script>

</head>
<body> 

    <h2>Connect.php</h2>

                <div>
                        <?php

                            require("Connection.php");

                            $curl = curl_init();

                            curl_setopt_array($curl, array(
                            CURLOPT_URL => "https://cors.io/?http://api.holfuy.com/live/?s=759&pw=h1u5l4kka&m=JSON&tu=C&su=m/s",
                            CURLOPT_RETURNTRANSFER => true,
                            CURLOPT_ENCODING => "",
                            CURLOPT_MAXREDIRS => 10,
                            CURLOPT_TIMEOUT => 30,
                            CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
                            CURLOPT_CUSTOMREQUEST => "GET",
                            CURLOPT_HTTPHEADER => array(
                            "Cache-Control: no-cache"
                            ),
                            ));

                            $response = curl_exec($curl);
                            $Wdata = json_decode($response, true);

                            $tem = $Wdata["temperature"];
                            $hum = $Wdata["humidity"];

                            $err = curl_error($curl);

                            curl_close($curl);

                            if ($err) {
                                        echo "cURL Error #:" . $err;
                                        } else {
                            // Here code to save $response into database;

                            echo $response;

                            echo "<br>";
                            echo $tem;
                            echo "<br>";
                            echo $hum;

                            }

                         try{


                            $sql = "INSERT INTO weatherdata (humidity, temprature)
                                    VALUES ('".$tem."','".$hum."')";
                            echo "<meta http-equiv='refresh' content='0'>";

                            ($conn->query($sql)); 

                                //$conn = null;
                            }
                            catch(PDOException $e)
                            {

                            }

                        ?>

                    </div>


        </body>
    </html>

Upvotes: 0

Views: 1168

Answers (2)

Fahad Bhuyian
Fahad Bhuyian

Reputation: 325

If you don't want to use cron job and if you want to use this page directly by browser then use this code .

use meta tag for refresh after five minute using find content = 300 

 <!DOCTYPE html>
<html lang="en">
<head>
<meta http-equiv="refresh" content="300">
<title>Weather Data</title>
<meta charset="utf-8">
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.3/jquery.min.js"> 
</script>

</head>
<body> 

<h2>Connect.php</h2>

            <div>
                    <?php

                        require("Connection.php");

                        $curl = curl_init();

                        curl_setopt_array($curl, array(
                        CURLOPT_URL => "https://cors.io/?http://api.holfuy.com/live/?s=759&pw=h1u5l4kka&m=JSON&tu=C&su=m/s",
                        CURLOPT_RETURNTRANSFER => true,
                        CURLOPT_ENCODING => "",
                        CURLOPT_MAXREDIRS => 10,
                        CURLOPT_TIMEOUT => 30,
                        CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
                        CURLOPT_CUSTOMREQUEST => "GET",
                        CURLOPT_HTTPHEADER => array(
                        "Cache-Control: no-cache"
                        ),
                        ));

                        $response = curl_exec($curl);
                        $Wdata = json_decode($response, true);

                        $tem = $Wdata["temperature"];
                        $hum = $Wdata["humidity"];

                        $err = curl_error($curl);

                        curl_close($curl);

                        if ($err) {
                                    echo "cURL Error #:" . $err;
                                    } else {
                        // Here code to save $response into database;

                        echo $response;

                        echo "<br>";
                        echo $tem;
                        echo "<br>";
                        echo $hum;

                        }

                     try{


                        $sql = "INSERT INTO weatherdata (humidity, temprature)
                                VALUES ('".$tem."','".$hum."')";
                        echo "<meta http-equiv='refresh' content='0'>";

                        ($conn->query($sql)); 

                            //$conn = null;
                        }
                        catch(PDOException $e)
                        {

                        }

                    ?>

                </div>


    </body>
</html>

Upvotes: 0

Alex
Alex

Reputation: 154

I would say using cronjob is the best way to go about doing this! (Assuming you are using a linux distro) In cron you select the interval, but if you would like you could use a sleep function to delay the command being run several seconds:

*/1 * * * *    sleep 5; mycommand 
^^                ^^
every minute   five seconds

EDIT: Your post doesn't seem really clear on what kind of Operating System you are running, I suggested using cronjob in my post but this is only available to linux systems. Windows task scheduler could be used if you are running it from a windows system although I am not sure how this one works so I wont be able to help you set it up.

Upvotes: 2

Related Questions