Lightweight library for scraping web-sites with LLMs
Repository Index
100,822 visible repositories in the index
97–120 of 148 · Most stars
Fast proxy scraper and checker written in Rust. Collects HTTP, SOCKS4 and SOCKS5 proxies from any number of sources, verifies each one really works, and writes JSON and plain text with response time, exit IP, ASN and geolocation. Single binary for Windows, Linux, macOS, Android, FreeBSD, NetBSD, illumos and Solaris, with an interactive TUI.
- proxyscraper
- proxyscrape
- proxy-checker
- proxy
A framework to build Slack apps using Python
- slack
- slack-api
- bolt
- web-api
Watch everything from your terminal.
Broken link checker that crawls websites and validates links. Find broken links, dead links, and invalid URLs in websites, documentation, and local files. Perfect for SEO audits and CI/CD.
- link-checker
- typescript
- broken-links
- 404
An open-source AI Voice Agent that integrates with Asterisk/FreePBX using Audiosocket/RTP/Websocket technology https://demo.agent6789.com
源码级 Chromium 指纹内核,检测站按普通 Chrome 评分。Source-level Chromium fingerprint kernel that passes bot detection tests. Not JS injection — C++ patches compiled into the binary.
- ai-agents
- anti-detect
- anti-detect-browser
- bot-detection
artisan scope: the world's most trusted roasting software
- coffee
- thermocouple
- fuji
- phidgets
:rocket: An open source alternative to searx which provides a modern-looking :sparkles:, lightning-fast :zap:, privacy respecting :disguised_face:, secure :lock: meta search engine
- meta-search-engine
- rust
- search-engine
- fast
Port of Wappalyzer (uncovers technologies used on websites) to automate mass scanning.
- pentesting
- security
- enumeration
- wappalyzer
WKZombie is a Swift framework for iOS/OSX to navigate within websites and collect data without the need of User Interface or API, also known as Headless browser. It can be used to run automated tests / snapshots and manipulate websites using Javascript.
A tool for exporting Telegram group chats into static websites like mailing list archives.
- telegram
- telegram-api
- static-site
- static-site-generator
📰 Newspaper4k a fork of the beloved Newspaper3k. Extraction of articles, titles, and metadata from news websites.
- articles
- articles-data
- crawler
- datasets-preparation
一个零码 , 低码 , AI的微服务接口编排 & 系统集成的强大编排平台,支持Http , Dubbo , WebService等协议的接口编排,支持通过Groovy , JavaScript , Python , Java等多种脚本语言来增强流程,支持使用MySQL,达梦等多种常见数据源。
- microservices-orchestration
- workflow-engine
- third-party-system-integration
- custom-development
Creating Scrapy scrapers via the Django admin interface
Scalable Python web scraping scripts for +80 popular domains
Crawly, a high-level web crawling & scraping framework for Elixir.
Free desktop SEO crawler - open source alternative to Screaming Frog and similar tools. Crawl websites, analyze links, extract SEO data, and export results without subscription fees. Fully customizable and extensible!
- desktop-app
- flask
- free
- open-source
An ergonomic, privacy-aware Rust HTTP Client
- http-client
- http2
- ja3
- ja4
Bastille is an open-source system for automating deployment and management of containerized applications on FreeBSD.
:alien: OneBlog,一个简洁美观、功能强大并且自适应的Java博客
- springboot
- blog
- dblog
- spring-boot
CMS Scanner: Scan Wordpress, Drupal, Joomla, vBulletin websites for Security issues
Make Your Discord Account 24/7 Online!
- discord
- discord-bot
- python
- discord-selfbot
🌐 Make websites accessible for AI agents. Automate tasks online with ease.
- ai-agents
- ai-tools
- browser-automation
- javascript