Scrape Twitter for Tweets
Repository Index
47,133 repositories indexed
25–48 of 50 · Most stars
Lightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages.
Crawly, a high-level web crawling & scraping framework for Elixir.
Ruby gem for web scraping purposes. It scrapes a given URL, and returns you its title, meta description, meta keywords, links, images...
Web fetch, search, and crawl for AI agents. Built from scratch in Rust. No keys, no accounts. AGPL v3.
- agpl
- ai-agents
- anti-bot
- crawling
A high performance web crawler / scraper in Elixir.
Command line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern matching. It can also create new or transformed XML/HTML/JSON documents.
- xquery
- xml
- html
- json
A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.
Java API For Chrome and Firefox
:scissors: High performance, multi-threaded image scraper
A Multi-User Selfhosted price tracker for Amazon, Aliexpress, ebay and many more along with custom stores. Get notified when price matches one or more criterias set by the user
- automation
- ecommerce
- filamentphp
- laravel
Extract structured data from web sites. Web sites scraping.
- golang
- golang-library
- extract-data
- chrome-fetcher
The unofficial HLTV Node.js API
TranslatedAn Agent Skill for scraping the complete comment section of a single Douyin video (top-level comments + nested replies): it copies the page's real requests inside a logged-in Chrome and leaves the signing to the page's SDK
- agent-skill
- claude-code
- comments-scraper
- douyin
Scrape/Crawl article from any site automatically. Make any web page readable, no matter Chinese or English.
Scrape any website, article or RSS/Atom Feed with ease!
- scrape
- elixir
- readability
- rss
DocSearch - Scraper
- algolia
- docsearch
- documentation
- scraper
A declarative struct-tag-based HTML unmarshaling or scraping package for Go built on top of the goquery library
- golang
- unmarshall
- goquery
- selectors
Self-hosted web scraper, crawler and document parser for AI agents: URLs, PDFs with OCR and Office files to clean Markdown/JSON. Firecrawl- and Jina-compatible API, MCP server, agent endpoint, Python/Node SDKs. Single Rust binary.
- ai-agents
- browser-automation
- data-extraction
- firecrawl
An app to search startup jobs scraped from websites written in Elixir, Phoenix, React and styled-components.
Scrapy project for data capture of vgchartz
Webpage summary extractor using Facebook Open Graph and arc90's readability
- readability
- scraper
- opengraph
Browser extension to simulate browsing behaviour in search engines.
- automation
- audit
- browser-extension
- firefox
Go package for scraping website recipes
- go
- golang
- recipe
- recipes