PHP: How to download a webpage (aka web scrapping) with PHP
Posted on 03. Oct, 2009 by Dragos in Coding, PHP
There are many ways of downloading web pages, or web content. Personally I like to use cURL for my web scrapping needs, but sometimes I also use fsockopen and file_get_contents.
Here are 3 different functions that will allow you to download web content.
cURL:
function getData($url) {
if($url!='localhost' && $url!='http://localhost') {
$ch=curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($ch,CURLOPT_USERAGENT,"Mozilla/6.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.1) Gecko/2008070208 Firefox/3.0.3");
curl_setopt($ch, CURLOPT_FOLLOWLOCATION,3);
$result['data']=curl_exec($ch);
$result['error']=curl_error($ch);
curl_close($ch);
return $result;
}
else return $result['error']='err';
}
fsockopen
function getData($url) {
$arr=parse_url($url);
$fp = fsockopen($arr['host'], 80, $errno, $errstr, 30);
if(!$fp) {
return false;
}else {
// send headers
$out = "GET ".fsockopen($arr['host'], 80, $errno, $errstr, 30)." HTTP/1.1\r\n";
$out .= "Host: ".str_replace('http://'.$arr['host'],'',$url)."\r\n";
$out .= "User-Agent: FSOCKOPEN\r\n";
$out .= "Connection: Close\r\n\r\n";
fwrite($fp, $out);
while(!feof($fp)) {
$contents .= fgets($fp, 4096);
};
fclose($fp);
return $contents;
}
}
file_get_contents
function getData($url) {
return file_get_contents($url);
}
As you see the easiest way of downloading web content is by using the file_get_contents function, but if you need more options, especially if you are working with the headers, then cURL is the best way to go for you.
Translate this post
Related posts:
-
http://ferryardhana.co.cc/ ferry ardhana
-
http://www.jrconsumer.com/ Travel Trailers
-
Hakan
-
http://twitter.com/okeyoyna_ Okey Oyna