A Powerful web scraper powered by LLM | OpenAI, Gemini & Ollama
Repository Index
78,351 visible repositories in the index
289–312 of 842 · Most stars
Do you want to LEARN NEW STUFF for FREE? Don't worry, with the power of web-scraping and automation, this script will find the necessary Udemy coupons & enroll you for PAID UDEMY COURSES, ABSOLUTELY FREE!
新浪微博爬虫（Scrapy、Redis）
The process of extracting product data from Amazon using Python, including titles, ratings, prices, images, and descriptions.
- amazon
- amazon-scraper
- python
- web-scraper
使用scrapy,redis, mongodb,graphite实现的一个分布式网络爬虫,底层存储mongodb集群,分布式使用redis实现,爬虫状态显示使用graphite实现
Translated[Captcha Recognition - Training] This project is based on CNN/ResNet/DenseNet+GRU/LSTM+CTC/CrossEntropy to realize verification code identification. This project is only for training the model.
- captcha-recognition
- tensorflow-tutorials
- ocr
- tensorflow
TranslatedMulti-platform content monitoring, collection, and reposting: one web panel to manage Douyin / Xiaohongshu / Kuaishou / WeChat Channels / TikTok
- automation
- content-monitoring
- crawler
- douyin
UI automation library for .Net
Article publishing platform that automatically distributes your articles to various media channels
一款快速、全面、易用的页面信息提取工具，可快速发现和提取页面中的JS、URL和敏感信息。
- jsfinder
- urlfinder
- infofinder
🤖 Top-rated tools to scrape all major public sections from Facebook, Instagram, and Twitter (X) including posts (likes/comments), photos/videos, contact information, followers, following and much more.
- facebook-page-insights
- instagram-followers-scraper
- facebook-ads-metrics
- facebook-contact-info
A tiny, fast Rust CLI that drives a real browser over the Chrome DevTools Protocol — built for coding agents.
- rust
- agent
- browser-automation
- llm
Turn any website into an API your agent can call. Learns from the site's own frontend, heals itself when the site changes.
Watch (parts of) webpages and get notified when something changes via e-mail, on your phone or via other means. Highly configurable.
- python
- webpage
- monitor
- automation
Open Source AI trading agent that operates autonomously across 1000+ markets - Polymarket, Kalshi, Binance, Hyperliquid, Solana DEXs, 5 EVM chains. Scans for edge, executes instantly, manages risk while you sleep. Agent commerce protocol for machine-to-machine payments. Self-hosted. Built on Claude.
🍥 Bilibili 用户爬虫
- bilibili
- python3
- mysql
WebCollector is an open source web crawler framework based on Java.It provides some simple interfaces for crawling the Web,you can setup a multi-threaded web crawler in less than 5 minutes.
A browser testing and web crawling library for PHP and Symfony
Cruise-control is the first of its kind to fully automate the dynamic workload rebalance and self-healing of a Kafka cluster. It provides great value to Kafka users by simplifying the operation of Kafka clusters.
- kafka
- cluster-management
- self-healing
Automatically crawl arXiv papers daily and summarize them using AI. Illustrating them using GitHub Pages.
A powerful browser crawler for web vulnerability scanners
- headless-chrome
- crawler
- chrome-devtools
- golang
An Implementation of NTQQ Protocol, with Pure C#, Derived from Konata.Core
- csharp
- ntqq
- qq-protocol
- qqbot
Track GitHub trending repositories in your favorite programming language by native GitHub notifications!
- github
- trending
- trending-repositories
Make your job hunt easy by automating your application process with this Auto Applier
- python
- selenium
- webscraping
- automation