Python scraper based on AI
- scraping
- scraping-python
- llm
- web-crawler
Repository Index
71,372 visible repositories in the index
1–24 of 35 · Most stars
Elegant Scraper and Crawler Framework for Golang
TranslatedDistributed web crawler admin platform for spiders management regardless of languages and frameworks.
- webcrawler
- scrapy
- crawlab
- spiders-management
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract structured data with LLMs via Pydantic AI, or control the browser in natural language with Stagehand. Works with BeautifulSoup, Parsel, Playwright, and raw HTTP. Download HTML, PDF, images, and other files. Proxy rotation included.
- apify
- automation
- beautifulsoup
- crawler
Web mining module for Python, with tools for scraping, natural language processing, machine learning, network analysis and visualization.
- python
- machine-learning
- natural-language-processing
- web-mining
scrape data from Google Maps. Extracts data such as the name, address, phone number, website URL, rating, reviews number, latitude and longitude, reviews,email and more for each place
- golang
- google-maps-scraping
- web-scraper
- web-scraping
Pholcus is a distributed high-concurrency crawler software written in pure golang
- spider
- crowler
A Chrome DevTools Protocol driver for web automation and scraping.
- cdp
- chrome-headless
- chrome-devtools
- chrome-devtools-protocol
Turn any webpage into structured data using LLMs
TranslatedToday's Hot List (今日热榜), an aggregator website that gets the hot headlines from major popular websites. Written in Go, it uses multiple goroutines to fetch information asynchronously and quickly. Preview: https://mo.fish
- hot
- golang
- spider
- today-s-hot-list
Intelligent proxy pool for Humans™ to extract content from the internet and build your own Large Language Models in this new AI era
- crawler
- python3
- scylla
- proxy-pool
Open-source web scraping API. Turn any website into clean markdown or structured JSON. Anti-detect browser, proxy auto-selection, self-hosted. One command: make up
Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more
- endpoints
- endpoint-discovery
- bugbounty
- crawler
freehire — the open-source search engine for job seekers
- ats
- careers
- hiring
- job-aggregator
A powerful browser crawler for web vulnerability scanners
- headless-chrome
- crawler
- chrome-devtools
- golang
Fast website link checker in Go
Stock market analyzer and predictor using Elasticsearch, Twitter, News headlines and Python natural language processing and sentiment analysis
- stock-market
- stock-prediction
- python
- tweepy
Point it at any web page and it finds the video, transcodes it, and streams it to your TV in real time. It even burns in subtitles.
多播放源自动采集在线影视网站 golang vue
- axios
- elementplus
- film
- gin
Library to scrape and clean web pages to create massive datasets.
- artificial-intelligence
- natural-language-processing
- nlp
- text-mining
Website Cloner - Utilizes powerful Go routines to clone websites to your computer within seconds.
- golang
- go
- cloning
- crawler
Web Scraper in Go, similar to BeautifulSoup
- golang
- go
- webscraper
- webscraping
dhtcrawler is a DHT crawler written in erlang. It can join a DHT network and crawl many P2P torrents. The program save all torrent info into database and provide an http interface to search a torrent by a keyword
Open-source, self-hosted enterprise & site search server built on OpenSearch. Crawls web / file / DB / cloud sources, 20+ languages, REST API, and AI/RAG & semantic search. Apache-2.0.
- java
- elasticsearch
- full-text-search
- lucene