robots-exclusion-standard — independent software & tools

1 independent robots-exclusion-standard listing the big search engines bury.

• robots-parser

  npm · Free · by samclarke

  A specification compliant robots.txt parser with wildcard (*) matching support.