user823527
Reputation: 3712
Resource limitations on php curl
I have a page where I retrieve content from many urls with curl. I found that after a certain number of urls retrieved, curl returns null. Is there a limit on the resources that curl will retrieve in any given session.
I do a curl_init() before retrieving each url and a curl_close() after getting the content. I thought that this would free resources right away would prevent a situation of reaching limits. Are there issues with calling curl_init() too many times.
Here is a sample of the php code where I retrieve the urls.
<?php
function get_curl() {
// Initialize a new curl resource
$ch = curl_init();
// Get only the content not the headers
curl_setopt($ch, CURLOPT_HEADER, 0);
// Return the value instead of printing the response to browser
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
// Use a user agent to mimic a browser
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.5) Gecko/20041107 Firefox/1.0');
return $ch;
// Remember to always close the session and free all resources
}
// Make sure curl is installed
$curl_exists = function_exists('curl_init');
if ( !$curl_exists ) {
return;
}
$todayfeast = date("Ymd");
$ctitleff = 'http://feed.evangelizo.org/reader.php?date='.$todayfeast.'&type=reading_lt&lang=FR&content=FR';
$creadff = 'http://feed.evangelizo.org/reader.php?date='.$todayfeast.'&type=reading&lang=FR&content=FR';
$ch = get_curl();
curl_setopt($ch, CURLOPT_URL, $ctitleff);
$titleff = curl_exec($ch);
curl_close($ch);
$ch = get_curl();
curl_setopt($ch, CURLOPT_URL, $creadff);
$readff = curl_exec($ch);
curl_close($ch);
$ctitlepf = 'http://feed.evangelizo.org/reader.php?date='.$todayfeast.'&type=reading_lt&lang=FR&content=PS';
$creadpf = 'http://feed.evangelizo.org/reader.php?date='.$todayfeast.'&&type=reading&lang=FR&content=PS';
$ch = get_curl();
curl_setopt($ch, CURLOPT_URL, $ctitlepf);
$titlepf = curl_exec($ch);
curl_close($ch);
$ch = get_curl();
curl_setopt($ch, CURLOPT_URL, $creadpf);
$readpf = curl_exec($ch);
curl_close($ch);
$titlesf = 'http://feed.evangelizo.org/reader.php?date='.$todayfeast.'&type=reading_lt&lang=FR&content=SR';
$readsf = 'http://feed.evangelizo.org/reader.php?date='.$todayfeast.'&type=reading&lang=FR&content=SR';
$ch = get_curl();
curl_setopt($ch, CURLOPT_URL, $titlesf);
$titlesf = curl_exec($ch);
curl_close($ch);
$ch = get_curl();
curl_setopt($ch, CURLOPT_URL, $readsf);
$readsf = curl_exec($ch);
curl_close($ch);
$titlegf = 'http://feed.evangelizo.org/reader.php?date='.$todayfeast.'&type=reading_lt&lang=FR&content=GSP';
$readgf = 'http://feed.evangelizo.org/reader.php?date='.$todayfeast.'&type=reading&lang=FR&content=GSP';
$ch = get_curl();
curl_setopt($ch, CURLOPT_URL, $titlegf);
$titlegf = curl_exec($ch);
curl_close($ch);
$ch = get_curl();
curl_setopt($ch, CURLOPT_URL, $readgf);
$readgf = curl_exec($ch);
curl_close($ch);
// Send info
$params = array(
'titleff' => $titleff,
'readff' => $readff,
'titlepf' => $titlepf,
'readpf' => $readpf,
'titlesf' => $titlesf,
'readsf' => $readsf,
'titlegf' => $titlegf,
'readgf' => $readgf,
);
echo json_encode($params);
?>
EDIT
The problem only happens when I call this php script through ajax.
<script>
$('#readingf').ready(function(){
$.ajax({
url: "loadfrench.php",
type: "GET",
data: { },
cache: false,
async: true,
dataType: 'json',
success: function (response) {
if (response != '')
{
$('#titleff').html(response.titleff);
$('#readff').html(response.readff);
$('#titlepf').html(response.titlepf);
$('#readpf').html(response.readpf);
$('#titlesf').html(response.titlesf);
$('#readsf').html(response.readsf);
$('#titlegf').html(response.titlegf);
$('#readgf').html(response.readgf);
alert ('titleff '+response.titleff+' readff '+response.readff);
}
},
error: function (request, status, error) {
alert ("status "+status+" error "+error+"responseText "+request.responseText);
},
});
});
</script>
If I have this file instead, and include in my main page with <?php require("filename.php") ?> it works.
<?php
function get_curl() {
// Initialize a new curl resource
$ch = curl_init();
// Get only the content not the headers
curl_setopt($ch, CURLOPT_HEADER, 0);
// Return the value instead of printing the response to browser
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
// Use a user agent to mimic a browser
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.5) Gecko/20041107 Firefox/1.0');
return $ch;
// Remember to always close the session and free all resources
}
// Make sure curl is installed
$curl_exists = function_exists('curl_init');
if ( !$curl_exists ) {
return;
}
$ch = get_curl();
$todayfeast = date("Ymd");
/* $titleff = $readff = $titlepf = $readpf = $titlesf = $readsf = $titlegf = $readgf = "blank"; */
$ctitleff = 'http://feed.evangelizo.org/reader.php?date='.$todayfeast.'&type=reading_lt&lang=FR&content=FR';
$creadff = 'http://feed.evangelizo.org/reader.php?date='.$todayfeast.'&type=reading&lang=FR&content=FR';
curl_setopt($ch, CURLOPT_URL, $ctitleff);
$titleff = curl_exec($ch);
curl_setopt($ch, CURLOPT_URL, $creadff);
$readff = curl_exec($ch);
$ctitlepf = 'http://feed.evangelizo.org/reader.php?date='.$todayfeast.'&type=reading_lt&lang=FR&content=PS';
$creadpf = 'http://feed.evangelizo.org/reader.php?date='.$todayfeast.'&&type=reading&lang=FR&content=PS';
curl_setopt($ch, CURLOPT_URL, $ctitlepf);
$titlepf = curl_exec($ch);
curl_setopt($ch, CURLOPT_URL, $creadpf);
$readpf = curl_exec($ch);
$titlesf = 'http://feed.evangelizo.org/reader.php?date='.$todayfeast.'&type=reading_lt&lang=FR&content=SR';
$readsf = 'http://feed.evangelizo.org/reader.php?date='.$todayfeast.'&type=reading&lang=FR&content=SR';
curl_setopt($ch, CURLOPT_URL, $titlesf);
$titlesf = curl_exec($ch);
curl_setopt($ch, CURLOPT_URL, $readsf);
$readsf = curl_exec($ch);
$titlegf = 'http://feed.evangelizo.org/reader.php?date='.$todayfeast.'&type=reading_lt&lang=FR&content=GSP';
$readgf = 'http://feed.evangelizo.org/reader.php?date='.$todayfeast.'&type=reading&lang=FR&content=GSP';
curl_setopt($ch, CURLOPT_URL, $titlegf);
$titlegf = curl_exec($ch);
curl_setopt($ch, CURLOPT_URL, $readgf);
$readgf = curl_exec($ch);
curl_close($ch);
// Send info
$params = array(
'titleff' => $titleff,
'readff' => $readff,
'titlepf' => $titlepf,
'readpf' => $readpf,
'titlesf' => $titlesf,
'readsf' => $readsf,
'titlegf' => $titlegf,
'readgf' => $readgf,
);
echo json_encode($params);
?>
<?php
$curl_exists = function_exists('curl_init');
// Make sure curl is installed
if ( $curl_exists ) {
// Initialize a new curl resource
$ch = curl_init();
// Get only the content not the headers
curl_setopt($ch, CURLOPT_HEADER, 0);
// Return the value instead of printing the response to browser
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
// Use a user agent to mimic a browser
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.5) Gecko/20041107 Firefox/1.0');
}
?>
<div id="readfre" style="overflow:hidden;">
<div id="readingf" class="tabber" style="width:100%; margin-top: 10px; margin-right:0px;">
<?php
$todayfeast = date("Ymd");
$ctitleff = 'http://feed.evangelizo.org/reader.php?date='.$todayfeast.'&type=reading_lt&lang=FR&content=FR';
$creadff = 'http://feed.evangelizo.org/reader.php?date='.$todayfeast.'&type=reading&lang=FR&content=FR';
if ( $curl_exists ) {
curl_setopt($ch, CURLOPT_URL, $ctitleff);
$titleff = curl_exec($ch);
}
if ( $curl_exists ) {
curl_setopt($ch, CURLOPT_URL, $creadff);
$readff = curl_exec($ch);
}
?>
<div id="readf" class="tabbertab">
<h2>PremiËre Lecture</h2>
<div id='titleff' class='rtitle'>
<?php echo $titleff; ?>
</div>
<div id='readff' class='rtext'>
<?php echo $readff; ?>
</div>
</div>
<?php
$ctitlepf = 'http://feed.evangelizo.org/reader.php?date='.$todayfeast.'&type=reading_lt&lang=FR&content=PS';
$creadpf = 'http://feed.evangelizo.org/reader.php?date='.$todayfeast.'&&type=reading&lang=FR&content=PS';
if ( $curl_exists ) {
curl_setopt($ch, CURLOPT_URL, $ctitlepf);
$titlepf = curl_exec($ch);
}
if ( $curl_exists ) {
curl_setopt($ch, CURLOPT_URL, $creadpf);
$readpf = curl_exec($ch);
}
?>
<div id="readp" class="tabbertab">
<h2>Psaume</h2>
<div id='titlepf' class='rtitle'>
<?php echo $titlepf; ?>
</div>
<div id='readpf' class='rtext'>
<?php echo $readpf; ?>
</div>
</div>
<?php
$titlesf = 'http://feed.evangelizo.org/reader.php?date='.$todayfeast.'&type=reading_lt&lang=FR&content=SR';
$readsf = 'http://feed.evangelizo.org/reader.php?date='.$todayfeast.'&type=reading&lang=FR&content=SR';
if ( $curl_exists ) {
curl_setopt($ch, CURLOPT_URL, $titlesf);
$titlesf = curl_exec($ch);
}
if ( $curl_exists ) {
curl_setopt($ch, CURLOPT_URL, $readsf);
$readsf = curl_exec($ch);
}
?>
<div id="reads" class="tabbertab">
<h2>DeuxiËme Lecture</h2>
<div id='titlesf' class='rtitle'>
<?php echo $titlesf; ?>
</div>
<div id='readsf' class='rtext'>
<?php echo $readsf; ?>
</div>
</div>
<?php
$titlegf = 'http://feed.evangelizo.org/reader.php?date='.$todayfeast.'&type=reading_lt&lang=FR&content=GSP';
$readgf = 'http://feed.evangelizo.org/reader.php?date='.$todayfeast.'&type=reading&lang=FR&content=GSP';
if ( $curl_exists ) {
curl_setopt($ch, CURLOPT_URL, $titlegf);
$titlegf = curl_exec($ch);
}
if ( $curl_exists ) {
curl_setopt($ch, CURLOPT_URL, $readgf);
$readgf = curl_exec($ch);
}
?>
<div id="readg" class="tabbertab">
<h2>Evangile</h2>
<div id='titlegf' class='rtitle'>
<?php echo $titlegf; ?>
</div>
<div id='readgf' class='rtext'>
<?php echo $readgf; ?>
</div>
</div>
<?php
// Remember to always close the session and free all resources
if ( $curl_exists ) {
curl_close($ch);
}
?>
</div>
</div>
Upvotes: 1
Views: 2535
Answers (1)
jkgeyti
Reputation: 2404
I cannot say exactly why it happens, but seeing you are fetching from the same server, I would not create a new curl handle every time. Simply use the same and only change CURLOPT_URL. That way, the server will reuse the TCP connection, instead of creating a new one every time:
$ch = get_curl();
curl_setopt($ch, CURLOPT_URL, $ctitleff);
$titleff = curl_exec($ch);
curl_setopt($ch, CURLOPT_URL, $creadff);
$readff = curl_exec($ch);
...
curl_close($ch);
Not sure it's an answer, but it's something to try :)
Also, you might want to take a look at curl-multi-exec()
Upvotes: 1
Related Questions
- PHP CURL library gives error 'Rate limit exceeded' of type 'invalid_access_error'
- php curl execution time limitation
- Curl request limit to 100 results
- Rate limit outgoing PHP+curl requests
- Dealing with large curl response - PHP
- PHP Multi Curl maxing out @ 1020 requests?
- curl: How to limit size of GET?
- What is the size or length limit of curl?
- memory limit in php
- Prevent memory overflow