newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:
- python
- news
- crawler
- crawling
Repository Index
47,133 repositories indexed
1–16 of 16 · Most stars
Download pictures (or videos) along with their captions and other metadata from Instagram.
- instagram-client
- instagram-feed
- downloader
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
- web-scraping
- text-extraction
- nlp
- html2text
SEO Tools for Laravel
- php
- opengraph
- meta-tags
- opengraph-tags
File Attachment toolkit for Ruby applications
- file-upload
- ruby
- rack
- storage
💪 User-generated content for Git-powered websites
- static-site-generator
- jekyll
- hugo
- middleman
:rocket: An open source alternative to searx which provides a modern-looking :sparkles:, lightning-fast :zap:, privacy respecting :disguised_face:, secure :lock: meta search engine
- meta-search-engine
- rust
- search-engine
- fast
Ruby gem for web scraping purposes. It scrapes a given URL, and returns you its title, meta description, meta keywords, links, images...
Pure Clojure Webdriver protocol implementation
- clojure
- webdriver
- firefox
- chrome
A AI Browser run any site JavaScript from Python with the real Firefox engine. Headless SpiderMonkey + Necko for automation and scraping.
- anti-bot
- automation
- firefox
- headless-browser
C++ Reflection Parser / Runtime Skeleton
- cmake
- libclang
- c-plus-plus
- meta
Ruby gem that fetches images and metadata from a given URL. Much like popular social website with link preview.
TranslatedAI subscription and card-shop price comparison platform: say what you need and let AI help you choose. Aggregates top-up service or account offers for ChatGPT / Codex, Claude, Gemini and more, and compares price trends, top-up channels and warranties. AI-powered shopping and price comparison for subscriptions and accounts, with cross-shop offers, price history and purchase details.
- ai
- price-comparison
- self-hosted
- web-scraping
Turn any website into clean, structured data: Markdown, metadata and links from one page or a whole site, in every major language.
- crawling
- csharp
- elixir
- ffi
Upload and manage your assets in the iTunes Store using the iTunes Store’s Transporter (iTMSTransporter).
- ruby
- itunes
- app-store
- movies
Create HTML snippets/embeds from URLs using info from oEmbed, Open Graph, meta tags.