In the past, I've used tools such as Xenu's Link Sleuth to keep track of links that become outdated or dissappear on domains that I maintain. Now I run linux, and decided to create a program that could run from a crontab. I named this program LinkBoy. It was written in PERL with the aid of the HTTP::Request and LWP::UserAgent packages.

For my site, I run linkboy with these parameters:

./linkboy.pl -icgi-bin -imirror -ilinkboy.php -iiso -o/var/www/htdocs/linkboy.php -w2 http://daga.dyndns.org/

You may type ./linkboy.pl --help for more information about what this is telling it to do.