Scrape domain names from SSL certificates of arbitrary hosts
Repository Index
142,273 visible repositories in the index
481–504 of 627 · Most stars
Firecrawl CLI and agent skills: search, scrape, and interact with the web from your terminal or any coding agent, plus parse, crawl, map, Agent and Monitor. Firecrawl is the context API for AI agents.
- agent-skills
- cli
- firecrawl
- mcp
Turn any website into a compact CLI tailored for AI agents. Browse the web in hundreds of tokens, not tens of thousands.
A Facebook crawler
- scrapy
- crawler
- crawl
Python script, which empowers people with no programming background to generate robust leads on a mass scale. This repo will be compiled of various versatile techniques used in lead generation.
- lead-generation
- python
- scraper
- chromedriver
tool to download comics and manga in pdf/epub/cbr/cbz from a website
- comics-downloader
- comicextra
- go
- windows
狠心开源企业级舆情新闻爬虫项目:支持任意数量爬虫一键运行、爬虫定时任务、爬虫批量删除;爬虫一键部署;爬虫监控可视化; 配置集群爬虫分配策略;👉 现成的docker一键部署文档已为大家踩坑 an enterprise-grade public opinion news crawler project: supports one-click operation of any number of crawlers, scheduled crawler tasks, batch deletion of crawlers; one-click deployment of crawlers
- apscheduler
- python
- scrapy
- scrapy-crawler
Declarative DOM extraction expression evaluator. 👨⚕️
Comic-dl is a command line tool to download manga and comics from various comic and manga sites. Supported sites : readcomiconline.to, mangafox.me, comic naver and many more.
- phantomjs
- python-support
- manga
- debian
Random User-Agent middleware based on fake-useragent
- web-scraping
- python
- scrapy
OSINT Swiss Army Knife
- go
- golang
- osint
- haveibeenpwnd
A library to scrap google images
Twitter data scraping, embedding based image search and more.
OSINT tool - gets data from services like shodan, censys etc. in one app
Kemono pawchive Downloader is a fast, PyQt5 app for archiving content from a wide array of sites, including Kemono, Coomer, Bunkr, Erome, Saint2.su, nhentai, and Discord. It features a Creator Browser, update checker, and supports multi-threaded, multi-part downloads. Sessions can pause, resume, or recover. filters
- coomer-downloader
- coomer-su
- kemono
- kemono-downloader
Twitter Scraper
- twitter-api
- python
- tweets
Comprehensive guide for NetworkChuck Episode 480 - Robin AI Dark Web Scraping Tool. Installation, usage, safety guidelines, and troubleshooting for educational security research.
Ghost Eye Informationgathering Footprinting Scanner and Recon Tool Release. Ghost Eye is an Information Gathering Tool I made in python 3. To run Ghost Eye, it only needs a domain or ip. Ghost Eye can work with any Linux distros if they support Python 3. Author: Jolanda de Koff
- etherape
- information-gathering
- nmap
- ethical-hacking
Moodle-DL downloads course content fast from Moodle (eg. lecture pdfs)
- moodle
- moode-crawler
- crawler
- moodle-downloader
Social media scraping / data collection library for Facebook, Twitter, Reddit, YouTube, Pinterest, and Tumblr APIs
- scraping
- social-media
- python
➖ Stripped down, stable version of firecrawl optimized for self-hosting and ease of contribution. Billing logic and AI features are completely removed. Crawl and convert any website into LLM-ready markdown.
Execute complex automation scripts via remote cloud sessions , featuring integrated residential proxies , automated CAPTCHA solving , and native JavaScript rendering for the toughest dynamic websites.
- headless-browser
- javascript-rendering
- oxylabs
- python
An API to scrape American court websites for metadata.
- scraping
- government
- courts
- pacer
Scrape websites for text by CSS selector.