Repository Index

Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML

• web-scraping
• text-extraction
• nlp
• html2text

Factory Bot ♥ Rails

Run a web page like a program

• cross-platform
• golang
• headless-browser
• purego

querySelector that can pierce Shadow DOM roots without knowing the path through nested shadow roots. Useful for automated testing of Web Components. Production use is not advised, this is for test environments/tools such as Web Driver, Playwright, Puppeteer

• webcomponents
• shadow-dom
• javascript
• puppeteer

Webpage summary extractor using Facebook Open Graph and arc90's readability

• readability
• scraper
• opengraph