crawler — independent software & tools

9 independent crawler listings the big search engines bury.

• AstroCrawl

  github · Free · by Etoileint

  Async web crawler powered by Playwright headless Chromium — JavaScript rendering, AI-powered extraction (OpenAI / Anthropic / Google), CSS rule engine with visual preview, and a full PySide6 desktop GUI. SQLite WAL crash recovery, proxy circuit breaker, RFC 9309 robots.txt, ReDoS-immune regex, 3,822 tests. pip install astrocrawl[full,gui,fast].

• apify

  npm · Free · by GitHub Actions

  The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.

• apify-client

  npm · Free · by GitHub Actions

  Apify API client for JavaScript

• @crawlee/core

  npm · Free · by GitHub Actions

  The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.

• firecrawl-cli

  npm · Free · by GitHub Actions

  Command-line interface for Firecrawl. Scrape, crawl, and extract data from any website directly from your terminal.

• @mendable/firecrawl-js

  npm · Free · by hello_sideguide

  JavaScript SDK for Firecrawl API

• cna-oab-attorney-data-scraper

  github · Free · by isaiasfeerreira

  📊 Scrape attorney data from the OAB CNA Directory with this Python tool, collecting names, OAB numbers, and UFs efficiently and reliably.

• Silent-Snake

  github · Free · by KIingMaxiii6813

  🕵️♂️ Scrape WebApp content efficiently with Silent-Snake, inspired by Screaming Frog for deeper insights and better SEO analysis.

• notion-md-crawler

  npm · Free · by tompenguin

  A library to recursively retrieve and serialize Notion pages with customization for machine learning applications.