Playwright that anti-bots cannot see, so no captchas: same API, anti-detect stealth headless Firefox, undetected fingerprint, bypass bot detection, Python web scraping and browser automation.
- datadome
- perimeterx
- akamai
- antidetect-browser
Repository Index
47,133 repositories indexed
25–48 of 52 · Most stars
Strategies to Gekko trading bot with backtests results and some useful tools.
- gekko
- strategies
- gekko-strategies
- gekko-backtest
Lightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages.
WKZombie is a Swift framework for iOS/OSX to navigate within websites and collect data without the need of User Interface or API, also known as Headless browser. It can be used to run automated tests / snapshots and manipulate websites using Javascript.
Crawly, a high-level web crawling & scraping framework for Elixir.
Command line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern matching. It can also create new or transformed XML/HTML/JSON documents.
- xquery
- xml
- html
- json
Extract structured data from web sites. Web sites scraping.
- golang
- golang-library
- extract-data
- chrome-fetcher
Ruby gem that fetches images and metadata from a given URL. Much like popular social website with link preview.
WarcDB: Web crawl data as SQLite databases.
Scrape any website, article or RSS/Atom Feed with ease!
- scrape
- elixir
- readability
- rss
Send Sir Perceval on a quest to retrieve and gather data from software repositories.
- grimoirelab
- perceval
- data-mining
- data-fetching
An Elixir library for parsing and extracting data from HTML and XML with CSS or XPath selectors.
🌍 Sentiment analysis over real-time social media data, rendering live charts to visualise trends
- sentiment-analysis
- sentiment-visualisation
- data-visualisation
Self-hosted web scraper, crawler and document parser for AI agents: URLs, PDFs with OCR and Office files to clean Markdown/JSON. Firecrawl- and Jina-compatible API, MCP server, agent endpoint, Python/Node SDKs. Single Rust binary.
- ai-agents
- browser-automation
- data-extraction
- firecrawl
Self-hosted, invisible CAPTCHA that detects AI agents driving real browsers. GDPR, WCAG 2.2 AA, and EAA: no cookies, no third-party data sharing, and no visual or audio puzzle. With a false-positive budget enforced in CI. Proof of work, drop-in for Turnstile and reCAPTCHA.
- anti-bot
- behavioral-analysis
- bot-detection
- captcha
Turn any website into clean, structured data: Markdown, metadata and links from one page or a whole site, in every major language.
- crawling
- csharp
- elixir
- ffi
Extract web archive data using Wayback Machine and Common Crawl
- golang
- crawler
- commoncrawl
- wayback-machine
The Frank!Framework is an easy-to-use, stateless integration framework which allows (transactional) messages to be modified and exchanged between different systems.
- java
- system-integration
- open-source
- community-driven
A robust web archive analytics toolkit
- python
- web
- warc
- bigdata
Upload and manage your assets in the iTunes Store using the iTunes Store’s Transporter (iTMSTransporter).
- ruby
- itunes
- app-store
- movies
Python/FastAPI application for monitoring Brazilian public-sector IT opportunities, with automated data collection, job filters, and official-source tracking.
Credential Phish Analysis and Automation
Scrapy project for data capture of vgchartz
Text Generation using Markov Chains fed by 4chan APIs
- scrapper
- data-mining
- markov-chain