TranslatedSelf-hosted TikTok, Douyin & Bilibili scraper and no-watermark video downloader — async REST API, MCP server for AI agents, CLI and web console, plus offline speech-to-text for downloaded videos. Self-healing identity pool, PostgreSQL archive, one docker compose up.
Repository Index
142,187 visible repositories in the index
1–20 of 20 · Most stars
Collection of scripts to aggregate image data for the purposes of training an NSFW Image Classifier
- nsfw-classifier
- nsfw
- deep-learning
- content-moderation
Web mining module for Python, with tools for scraping, natural language processing, machine learning, network analysis and visualization.
- python
- machine-learning
- natural-language-processing
- web-mining
历史学习项目:Spark 电影推荐、爬虫、Web 与后台。新版持续开发见 cinematch-recommender。
- spark-mllib
- spark-streaming
- ssm-maven
- scrapy
媒体服务器演员头像仓库
Vision utilities for web interaction agents 👀
- ocr
- playwright
- selenium
- webscraping
Get clean data from tricky documents, powered by vision-language models ⚡
- multimodal
- large-language-models
- scrapers
World’s single largest Internet domains dataset
- yacy
- scrapy
- dataset
- search-engines
World Factbook Country Profiles in JSON - Free Open Public Domain Data - No API Key Required ;-)
- factbook
- opendata
- world
- countries
🧹 Python package for text cleaning
- python
- natural-language-processing
- text-cleaning
- text-normalization
Crawl BookCorpus
🔥 This repository contains complete application examples, including websites and other projects, developed using Firecrawl.
- ai
- ai-scraping
- data
- examples
🤖 AI-powered web scraping editor with visual workflow builder. Build, test & deploy web scrapers using natural language. Powered by ScrapeGraphAI & LangGraph.
- ai
- automation
- data-extraction
- docker
➖ Stripped down, stable version of firecrawl optimized for self-hosting and ease of contribution. Billing logic and AI features are completely removed. Crawl and convert any website into LLM-ready markdown.
Complete-Life-Cycle-of-a-Data-Science-Project
- feature-engineering
- unsupervised-learning
- data-science
- deep-learning
A Playwright-based Node.js tool that bypasses search engine anti-scraping mechanisms to execute Google searches. Local alternative to SERP APIs with MCP server integration.
- ai
- google-search
- llm
- mcp-server
Open source web infrastructure for AI. Scrape, crawl, and automate the web, clean markdown, browser sessions, ready for your agents.
OpenClaw-inspired autonomous AI agent built entirely in n8n. Adaptive RAG-powered memory, Skills via MCP templates, Expert Agents with delegated sub-agents, proactive task management, media understanding - self-hosted with one setup script
A dataset of millions of news articles scraped from a curated list of data sources.
Video game review datasets scraped from the Steam website (http://store.steampowered.com/). Data includes review text, number of hours that the reviewer played the game, number of friends, groups, etc.