47,133 repositories indexed
1–24 of 29 · Most stars
🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
The fast, flexible, and elegant library for parsing and manipulating HTML and XML.
A little like that j-thing, only in Go.
CLI for common Playwright actions. Record and generate Playwright code, inspect selectors and take screenshots.
Pythonic HTML Parsing for Humans™
jsoup: the Java HTML parser, built for HTML editing, cleaning, scraping, and XSS safety.
SwiftSoup: Pure Swift HTML Parser, with best of DOM, CSS, and jquery (Supports Linux, iOS, Mac, tvOS, watchOS)
Headless Chromium-based web performance metrics collector and monitoring tool
Find web directories without bruteforce
This is a repository for Microsoft Power Automate, Power Apps, and Azure Logic Apps connectors
Ruby gem for web scraping purposes. It scrapes a given URL, and returns you its title, meta description, meta keywords, links, images...
Command line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern matching. It can also create new or transformed XML/HTML/JSON documents.
htmlquery is golang XPath package for HTML query.
Job Queue for Elixir. Clustered or Local. Straight BEAM. Optional Ecto. 💪🍈
The .NET library to build AI agents with 30+ built-in connectors.
PHP Magic Number Detector
High-performance HTML5 parser for Ruby based on Lexbor, with support for both CSS selectors and XPath.
App-inspector is a mobile UI viewer in browser.
Extract knowledge from all information sources using gpt and other language models. Index and make Q&A session with information sources.
A declarative struct-tag-based HTML unmarshaling or scraping package for Go built on top of the goquery library
querySelector that can pierce Shadow DOM roots without knowing the path through nested shadow roots. Useful for automated testing of Web Components. Production use is not advised, this is for test environments/tools such as Web Driver, Playwright, Puppeteer
Playwright PHP library for browser automation: navigation, E2E tests, assertions, screenshots, and so much more!
Self-hosted, invisible CAPTCHA that detects AI agents driving real browsers. GDPR, WCAG 2.2 AA, and EAA: no cookies, no third-party data sharing, and no visual or audio puzzle. With a false-positive budget enforced in CI. Proof of work, drop-in for Turnstile and reCAPTCHA.
:vertical_traffic_light: Block malicious crawlers with Cloudflare Firewall Rules