PHP: How to download a webpage (aka web scrapping) with PHP

Posted on 03. Oct, 2009 by Dragos in Coding, PHP

There are many ways of downloading web pages, or web content. Personally I like to use cURL for my web scrapping needs, but sometimes I also use fsockopen and file_get_contents.

Here are 3 different functions that will allow you to download web content.

cURL:

function getData($url) {
    if($url!='localhost' && $url!='http://localhost') {
        $ch=curl_init();
        curl_setopt($ch, CURLOPT_URL, $url);
        curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
        curl_setopt($ch,CURLOPT_USERAGENT,"Mozilla/6.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.1) Gecko/2008070208 Firefox/3.0.3");
        curl_setopt($ch, CURLOPT_FOLLOWLOCATION,3);
        $result['data']=curl_exec($ch);
        $result['error']=curl_error($ch);
        curl_close($ch);
        return $result;
    }
    else return $result['error']='err';
}

fsockopen

function getData($url) {
    $arr=parse_url($url);
    $fp = fsockopen($arr['host'], 80, $errno, $errstr, 30);
    if(!$fp) {
        return false;
    }else {
    // send headers
        $out = "GET ".fsockopen($arr['host'], 80, $errno, $errstr, 30)." HTTP/1.1\r\n";
        $out .= "Host: ".str_replace('http://'.$arr['host'],'',$url)."\r\n";
        $out .= "User-Agent: FSOCKOPEN\r\n";
        $out .= "Connection: Close\r\n\r\n";
        fwrite($fp, $out);
        while(!feof($fp)) {
            $contents .= fgets($fp, 4096);
        };
        fclose($fp);
        return $contents;
    }
}

file_get_contents

function getData($url) {
return file_get_contents($url);
}

As you see the easiest way of downloading web content is by using the file_get_contents function, but if you need more options, especially if you are working with the headers, then cURL is the best way to go for you.

                    
                    

Related posts:

1. JavaScript: Send function as a parameter to another function (callbacks)