Ruby Web Crawler

This Ruby script reads in a list of links from links.dat, it then picks out the ones it can easily spider and gets a list of URLs from each page listed in links.dat.

Every new URL it finds will be added to newlinks.dat for later spidering by another bot running along side this one.

Related Scripts

  • Url Web Crawler 1.0
    It is basically a program that can make you a search engine. It is a web crawler, has all the web site source code (in ASP, soon to be PHP as well), a...
  • Xceog Web Crawler
    This web crawler is written in Visual Basic. This program is meant for advanced vb developers/programmers. A web crawler is an automated program that ...
  • Web Based Web Crawler
    This version allows your to run the crawler from a Web page, instead of from a command line, and output the results to the Web page.This would be usef...
  • Crawler
    Crawler allows developers to pass an URL to the class, a depth of search and retrieve results from that Web page.The returned results can then be pass...
  • Rcrawl 0.5.1
    Rcrawl is a web crawler written in ruby....
  • Larbin 2.6.3
    Larbin is a web crawler (also called (web) robot, spider, scooter...). It is intended to fetch a large number of web pages to fill the database of a s...
  • Php Web Crawler 1.0.0
    Written in PHP, it can differentiate from normal navigation links and static data like files, image sources and other multimedia files usually found e...
  • Python Web Crawler 1.0.1
    It can differentiate from normal navigation links and static data like files, image sources and other multimedia files usually found embedded in a web...
  • Php Crawler 0.8
    PHP Crawler is a simple website search script for small-to-medium websites. The only requrements are PHP and MySQL, no shell access required....
  • Ruby-web 1.1.0b
    ruby-web is a web-centric distribution of the ruby interpreter, providing many enhancements to the standard ruby libraries to make web programing more...
  • Heritrix 1.12.1
    Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project. Heritrix (sometimes spelled heretrix, or ...
  • Www-crawler-lite 0.003
    It can be used within a mod_perl, CGI or Catalyst-style environment because it does not fork or use threads.The callback-based interface is fast and s...
  • Ruby/asp
    Ruby-ASP provides an Active Server Pages port to the Apache Web Server with Ruby scripting only, and enables developing of dynamic web applications wi...
  • Mod_ruby 1.3.0
    mod_ruby embeds the Ruby interpreter into the Apache web server, allowing Ruby CGI scripts to be executed natively.Ruby scripts will start up much fas...
  • Youtube Crawler
    YouTube Crawler pulls brand spanking new videos from YouTube automatically to you Media Sharing Script! Need an auto-pilot solution to pack your site ...
  • Webtools 4 Larbin 1.0
    Larbin is a Web crawler intended to fetch a large number of Web pages. It should be able to fetch more than 100 millions pages on a standard PC with m...
  • Ruby/dlx 0.9.0 rc1
    No more ruby-C-extensions needed to use the shared C library of your choice in ruby, now you can do it directly in Ruby itself! Ruby/DLX shows how sim...
  • Ruby-debug 0.10.5rc1
    ruby-debug is a faster implementation of the standard debug.rb, included with native Ruby, using a native extension with a new hook Ruby C API .This w...
  • Ruby Lsapi Module 4.1
    Allows developers to run and optimize the LiteSpeed server for running Ruby and Rails apps.By utilizing persistent connection between web server ruby ...
  • Pro-search 0.17.2
    PRO-Search is a crawler of FTP servers, SMB shares, HTTP, dc networks with powerful web search and navigation interface....
DMCA Notice-Privacy Policy
2004 - 2013 DownScripts. All rights reserved.