« robots.txt in the real world
Linux Magazine UK: 10 dollars for 3 issues »

[Osjava] [ANN] Norbert 0.1

August 22nd, 2003 by Hen

Announcing the release of Norbert version 0.1.

Norbert is a parser/handler for the norobots rfc. It parses a site’s
robots.txt file for a particular user-agent and then answers questions on
whether a url is allowed or not according to http://www.robotstxt.org/wc/norobots-rfc.html.

This entry was posted on Friday, August 22nd, 2003 at 10:26 pm and is filed under Tech. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

Comments are closed.