Scrape Twitter for Tweets
Repository Index
47,133 repositories indexed
25–48 of 50 · Most stars
Lightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages.
Crawly, a high-level web crawling & scraping framework for Elixir.
Web fetch, search, and crawl for AI agents. Built from scratch in Rust. No keys, no accounts. AGPL v3.
- agpl
- ai-agents
- anti-bot
- crawling
A high performance web crawler / scraper in Elixir.
Command line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern matching. It can also create new or transformed XML/HTML/JSON documents.
- xquery
- xml
- html
- json
A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.
Java API For Chrome and Firefox
:scissors: High performance, multi-threaded image scraper
A Multi-User Selfhosted price tracker for Amazon, Aliexpress, ebay and many more along with custom stores. Get notified when price matches one or more criterias set by the user
- automation
- ecommerce
- filamentphp
- laravel
Extract structured data from web sites. Web sites scraping.
- golang
- golang-library
- extract-data
- chrome-fetcher
The Swiss Army knife to power JavaScript based client solutions ranging from Web apps/explorers over browser extensions to server-side clients like faucets/scrapers.
Semrush SEO AI-powered analysis tool MCP server & CLI for keyword research, domain analytics, backlink analysis
- backlink-analysis
- claude-desktop
- cursor-pro
- semrush
The unofficial HLTV Node.js API
TranslatedAn Agent Skill for scraping the complete comment section of a single Douyin video (top-level comments + nested replies): it copies the page's real requests inside a logged-in Chrome and leaves the signing to the page's SDK
- agent-skill
- claude-code
- comments-scraper
- douyin
DocSearch - Scraper
- algolia
- docsearch
- documentation
- scraper
Extract knowledge from all information sources using gpt and other language models. Index and make Q&A session with information sources.
EveryInfra MCP server & agent plugin: scraper API for TikTok, Instagram, Amazon, YouTube, Xiaohongshu & Douyin. 573 data APIs across 106 platforms, from $0.56/1K calls. For Claude Code, Cursor, Codex.
- ai-agents
- claude-code
- cursor
- mcp
X (Twitter) scraper API & Twitter MCP server to scrape tweets, profiles & followers. Not affiliated with X Corp.
- mcp
- mcp-server
- twitter-api
- x-api
Self-hosted web scraper, crawler and document parser for AI agents: URLs, PDFs with OCR and Office files to clean Markdown/JSON. Firecrawl- and Jina-compatible API, MCP server, agent endpoint, Python/Node SDKs. Single Rust binary.
- ai-agents
- browser-automation
- data-extraction
- firecrawl
Webpage summary extractor using Facebook Open Graph and arc90's readability
- readability
- scraper
- opengraph
Browser extension to simulate browsing behaviour in search engines.
- automation
- audit
- browser-extension
- firefox
Go package for scraping website recipes
- go
- golang
- recipe
- recipes
Self-hosted manga collection manager with automatic metadata, MyAnimeList import, Docker support, and detailed collection statistics.
- database
- isbn
- manga
- myanimelist