user-agent — independent software & tools

1 independent user-agent listing the big search engines bury.

• robots-parser

  npm · Free · by samclarke

  A specification compliant robots.txt parser with wildcard (*) matching support.