robots.txt — independent software & tools

1 independent robots.txt listing the big search engines bury.

• robots-parser

  npm · Free · by samclarke

  A specification compliant robots.txt parser with wildcard (*) matching support.