PHP Tutorial: Making a webcrawler!

Hey!

Don't you know what a webcrawler is? A webcrawler is used by search engines like:
Google, Yahoo and bing. These search engines got bots running 24/7 searching for new websites. How do these bots work? Easy.
Keep reading to find out how to make one yourself!


Let me show you in steps:

Step 1: Bot starts and gets a URL.
Step 2: The bot opens the URL and searches for all links.
Step 3: The bot delete not working links.
Step 4: The bot adds the links to a database.
Step 5: The bot goes back to Step 1 with the a found link.



Lets get started making one shall we?

Requirements:

• Brains
• Some php knowledge (Variables, Functions etc.)
• Some HTML knowledge (How to make a link.)
• A webserver (See below)
• A MySQL Database (comes with the webserver below)
• "Simple_HTML_Dom.PHP" (Download: HERE)

 Index: 

• Setting up a webserver
• Concept
• Making the PHP crawler.
• Variations.

Setting up a webserver:
You wanna know how to setup a webserver?
It's super easy! But do you want it on your pc? or on an USB stick?

For the PC download: WAMP WebServer.
Setting it up wont be that hard. Just follow the install instructions and start it.
Now goto: http://localhost/ 
(NO .com .org .net)
Put your ".php" files in: {wamp instal directory}/www/

For USB download: EasyPHP - The portable webserver!
The setup is the same as wamp but make sure your install location is on your USB.

NOTE: Both webserver got MySQL preinstalled!
To connect use:

mysql_connect("localhost", "root", "");

Server: localhost
Username: Root
Password: (none)


Concept:
The concept is very easy. Like said before the bots runs in a loop.
And adds the links to the database. 
Now how are we going to get the HTML source code?
The "Simple_HTML_Dom.php" has all the functions we need!
So let's include it in our html first:

<?php
  include_once('simple_html_dom.php');
?>

Ok! We've included the extension we can now use it!
If we want the source code from a url we need to define the url and load it first:

  <?php
  //---
  include_once('simple_html_dom.php');
  //---
  $url = "http://timvanosch.blogger.com/";
  $html = new simple_html_dom();
  $html->load_file($url);
  //--
  ?>

 
Ok let me explain it:

1. Opening the PHP file: "<?php"
2. Comment (non code)
3. Including the extension
4. Comment (non code)
5. Defining variable "$url". This is the page that we will grab the source from!
6. Defining variable "$html". This is a extension class. (read on)
7. Execute function "load_file" in the "$html" class. This will load "$url" source!
8. Comment (non code)
9. Closing the PHP file: "?>"

We're here. We've succesfully loaded the file in the "$html" variable!.


Making the base PHP file:
We've already made a good base but we want to extend it, So it will echo out the links. 
So we've got the source in "$html", We now need to find all the "<a href="blabla"></a>" tags and cut out the href link.
We're lucky cause "Simple_html_dom.php" already got such a function and looks like this:

$html->find('a');

Ok, This function will return an array with all the "<A>" tags in the source!
To get through all the "<A>" tags quickly we're gonna use "foreach(){}" function.
And I'm gonna use the code from Concept:

<?php
  //---
  include_once('simple_html_dom.php');
  //---
  $url = "http://timvanosch.blogspot.com/";
  $html = new simple_html_dom();
  $html->load_file($url);
  //--
  foreach($html->find("a") as $link)
  {
    echo $link->href."< br />;
  }
?>

 
Now let me explain it:

    9. The foreach will loop and assign a array entry to "$link" till there are no more left.
         So it will start at 0,1,2,3,4,5,6,7,8,9 in the array.
   11. This echos out the href from the "<A>" tag and adds an enter.

Now change "$url" to a site and watch the magic happens.
This is my output: (I have changed the urls a bit for protection!)

http://timvanosch.blogspot.nl/2013/02/earn-money-onlin...
http://bit.ly/Wluavs
http://timvanosch.blogspot.nl/2013/02/earn-money-onlin...
https://plus.google.com/11489535515489321
http://timvanosch.blogspot.nl/2013/02/earn-money-onlin...
http://timvanosch.blogspot.nl/2013/02/earn-money-onlin...
http://www.blogger.com/post-edit.g?blogID=743878789022...
http://www.blogger.com/share-post.g?blogID=74387878902...
http://www.blogger.com/share-post.g?blogID=74387878902...
http://www.blogger.com/share-post.g?blogID=74387878902...
http://www.blogger.com/share-post.g?blogID=743878785902...
http://timvanosch.blogspot.nl/
http://timvanosch.blogspot.com/feeds/posts/default
//www.blogger.com/rearrange?blogID=743878789022871384...
//www.blogger.com/rearrange?blogID=743878789022871384...
//www.blogger.com/rearrange?blogID=743878789022871384...
http://www.blogger.com
//www.blogger.com/rearrange?blogID=743878789022871384...

Nice! We've got results.
So what've you learned ?:

• How to incude extensions.
• How to use extensions.
• How to get source code.
• How to use "foreach(){}"
• How to crawl the web!

Now if you want to make an: 'infinite crawler' just apply your basic php skills and you'll be able to make a loop.
Again goto Concept to see what you have to do for an 'infinite crawler'.
You can put all the urls found on an website in an array or directly into a database.
Then use those urls and crawl them.


Variations:
Ofcourse you can variate much in crawlers. I made one which will show you the found links on a site. You can press these links and it will crawl the pressed link. 
It's like an 'infinite crawler' but then with human pauses in between.

Download: DropBox link to: Crawler_source_code.rar

For the ones that don't trust me:
Jotti - Online virus scanner

Jotti is online virus scanner. It will scan a file with 21 different virus scanners.
I've already uploaded the file on jotti so you can view the results above.

Thank you!
Thanks for reading this post. If you wish to get more tutorials like these subscribe to this blog on the right site. Just enter your e-mail and you will get all the post right to your mail!

Greets, Tim.