PHP: How to download a webpage (aka web scrapping) with PHP PHP: Hvordan laste ned en nettside (aka web scrapping) med PHP

Posted on 03. Posted on 03. Oct, 2009 by Dragos in Coding , PHP Oktober 2009 av Dragos i Coding, PHP

There are many ways of downloading web pages, or web content. Det er mange måter å laste ned nettsider, eller webinnhold. Personally I like to use cURL for my web scrapping needs, but sometimes I also use fsockopen and file_get_contents . Personlig liker jeg å bruke cURL for min web opphugging behov, men noen ganger jeg også bruke fsockopen og file_get_contents.

Here are 3 different functions that will allow you to download web content. Her er 3 forskjellige funksjoner som vil tillate deg å laste ned webinnhold.

cURL : cURL:

 function getData($url) { funksjon getData ($ url) (
    if($url!='localhost' && $url!='http://localhost') { if ($ url! = 'localhost' & & $ url! = 'http://localhost') (
        $ch=curl_init(); $ ch = curl_init ();
        curl_setopt($ch, CURLOPT_URL, $url); curl_setopt ($ ch, CURLOPT_URL, $ url);
        curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE); curl_setopt ($ ch, CURLOPT_RETURNTRANSFER, TRUE);
        curl_setopt($ch,CURLOPT_USERAGENT,"Mozilla/6.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.1) Gecko/2008070208 Firefox/3.0.3"); curl_setopt ($ ch, CURLOPT_USERAGENT, "Mozilla/6.0 (Windows; U; Windows NT 5.1; no; rv: 1.9.0.1) Gecko/2008070208 Firefox/3.0.3");
        curl_setopt($ch, CURLOPT_FOLLOWLOCATION,3); curl_setopt ($ ch, CURLOPT_FOLLOWLOCATION, 3);
        $result['data']=curl_exec($ch); $ resultat [ 'data'] = curl_exec ($ ch);
        $result['error']=curl_error($ch); $ resultat [ 'error'] = curl_error ($ ch);
        curl_close($ch); curl_close ($ ch);
        return $result; return $ result;
    } )
    else return $result['error']='err'; else return $ resultat [ 'error'] = 'feile';
} ) 

fsockopen fsockopen

 function getData($url) { funksjon getData ($ url) (
    $arr=parse_url($url); $ arr = parse_url ($ url);
    $fp = fsockopen($arr['host'], 80, $errno, $errstr, 30); $ fp = fsockopen ($ arr [ 'host'], 80, $ errno, $ errstr, 30);
    if(!$fp) { if (! $ fp) (
        return false; return false;
    }else { ) else (
    // send headers / / Send overskriftene
        $out = "GET ".fsockopen($arr['host'], 80, $errno, $errstr, 30)." HTTP/1.1\r\n"; $ out = "GET". fsockopen ($ arr [ 'host'], 80, $ errno, $ errstr, 30). "HTTP/1.1 \ r \ n";
        $out .= "Host: ".str_replace('http://'.$arr['host'],'',$url)."\r\n"; $ out .= "Host:". str_replace ( 'http://'. $ arr [ 'host'],'',$ url). "\ r \ n";
        $out .= "User-Agent: FSOCKOPEN\r\n"; $ out .= "User-Agent: fsockopen \ r \ n";
        $out .= "Connection: Close\r\n\r\n"; $ out .= "Connection: Close \ r \ n \ r \ n";
        fwrite($fp, $out); fwrite ($ fp, $ out);
        while(!feof($fp)) { while (! feof ($ fp)) (
            $contents .= fgets($fp, 4096); $ innholdet .= fgets ($ fp, 4096);
        }; );
        fclose($fp); fclose ($ fp);
        return $contents; return $ innhold;
    } )
} ) 

file_get_contents file_get_contents

 function getData($url) { funksjon getData ($ url) (
return file_get_contents($url); return file_get_contents ($ url);
} ) 

As you see the easiest way of downloading web content is by using the file_get_contents function, but if you need more options, especially if you are working with the headers, then cURL is the best way to go for you. Som du ser den enkleste måten å laste ned web-innhold ved hjelp av file_get_contents funksjon, men hvis du trenger flere muligheter, særlig hvis du jobber med overskrifter, så cURL er den beste veien å gå for deg.

Related posts: Relaterte artikler:

1. JavaScript: Send function as a parameter to another function (callbacks) JavaScript: Send fungere som en parameter til en annen funksjon (tilbakeanrop)