PHP Classes

Spider website: Crawl a site and retrieve the the URL of all links

Recommend this page to a friend!
  Info   View files View files (2)   DownloadInstall with Composer Download .zip   Reputation   Support forum (2)   Blog    
Ratings Unique User Downloads Download Rankings
StarStarStar 45%Total: 3,048 This week: 1All time: 1,197 This week: 560Up
Version License PHP version Categories
spider 0.1GNU General Publi...5.0HTML, PHP 5, Searching
Description 

Author

This class can be used to crawl a site and retrieve the the URL of all links.

It can retrieve a page of a site and follow all links recursively to retrieve all the site URLs.

The class can restrict the crawling to URLs with a given extension and avoids accessing pages listed in the site robots.txt file, or pages set with the no index or no follow meta tags.

Picture of Karol Janyst
Name: Karol Janyst <contact>
Classes: 2 packages by
Country: Poland Poland
Age: 36
All time rank: 3849 in Poland Poland
Week rank: 411 Down14 in Poland Poland Down

  Files folder image Files  
File Role Description
Plain text file spider.class.php Class Main class file
Accessible without login Plain text file example.php Example Example file

 Version Control Unique User Downloads Download Rankings  
 0%
Total:3,048
This week:1
All time:1,197
This week:560Up
User Ratings User Comments (2)
 All time
Utility:62%StarStarStarStar
Consistency:78%StarStarStarStar
Documentation:-
Examples:65%StarStarStarStar
Tests:-
Videos:-
Overall:45%StarStarStar
Rank:3248
 
I ran a simple test using this class.
13 years ago (Oliver Lillie)
22%StarStar
It's got great potential, but.
14 years ago (F Philip DeGeorge)
55%StarStarStar