SpiderFoot automates OSINT for threat intelligence and mapping your attack surface.
- footprinting
- osint
- threatintel
- python
132,241 visible repositories in the index
1–24 of 111 · Most stars
SpiderFoot automates OSINT for threat intelligence and mapping your attack surface.
Python ProxyPool for web spider
TranslatedA Python web scraping tutorial series, learning Python web scraping from 0 to 1, including browser packet capture and mobile APP packet capture with tools such as fiddler and mitmproxy, the use of the various modules involved in web scraping, such as requests, beautifulSoup, selenium, appium, scrapy, etc., as well as IP proxies, CAPTCHA recognition, using Mysql and MongoDB databases with Python, multi-threaded and multi-process scraping, reverse engineering and cracking of CSS scraping encryption, JS scraping reverse engineering, distributed scraping, practical scraping project examples, etc.
Translated:rainbow:Python3 web crawler practice: Taobao, JD.com, NetEase Cloud Music, Bilibili, 12306, Douyin, Biquge, comic and novel downloads, music and movie downloads, etc.
Translated😮Python simulated login for some large websites, plus some simple web crawlers. I hope they are helpful to you❤️. If you like it, remember to give it a star🌟
Incredibly fast crawler designed for OSINT.
TranslatedPython open-source project "The Road to Self-Taught Programming", with step-by-step tutorials for beginners: AI Lab, treasure videos, data structures, study guides, hands-on machine learning, hands-on deep learning, web crawlers, big-company interview experiences, programmer life, resource sharing.
TranslatedLive stream recording software that supports looped monitoring and recording multiple streamers at once. It supports recording live streams from 40+ platforms, including Douyin, TikTok, Youtube, Kuaishou, Huya, Douyu, Bilibili, Xiaohongshu, pandatv, sooplive, flextv, popkontv, twitcasting, winktv, Baidu, Weibo, Kugou, 17Live, Twitch, Acfun, CHZZK, shopee, and more
TranslatedSina Weibo crawler, uses Python to scrape Sina Weibo data
TranslatedXiaohongshu (Little Red Book) crawler data collection, Xiaohongshu reverse engineering, private messages, live streaming, full-domain operations solution for Xiaohongshu
TranslatedINFO-SPIDER is a crawler toolbox 🧰 that brings many data sources together in one place, designed to help users get their own data back safely and quickly. The tool's code is open source and the process is transparent. Supported data sources include GitHub, QQ Mail, NetEase Mail, Alibaba Mail, Sina Mail, Hotmail, Outlook Mail, JD.com, Taobao, Alipay, China Mobile, China Unicom, China Telecom, Zhihu, Bilibili, NetEase Cloud Music, QQ friends, QQ groups, generating a Moments (WeChat friend circle) photo album, browser browsing history, 12306, cnblogs (博客园), CSDN blog, OSChina blog, Jianshu.
TranslatedThe essence edition of getting started with web crawlers in Python
TranslatedMore and more websites have anti-crawler features: some hide key data in images, and some use inhuman CAPTCHAs. This is a repository of anti-anti-crawler code, built to improve skills by contending with websites of different characteristics (with no malicious intent). (Submissions of websites that are hard to scrape are welcome.) (The project is paused due to work reasons.)
An Efficient ProxyPool with Getter, Tester and Server
TranslatedHands-on 🐍 web crawlers for many kinds of websites and e-commerce data 🕷. Includes 🕸: Taobao products, WeChat official accounts, Dianping, Qichacha, job recruitment sites, Xianyu, Alibaba tasks, Cnblogs, Weibo, Baidu Tieba, Douban movies, Baotu (ibaotu.com), Quanjing (quanjing.com), Douban music, a provincial drug administration, Sohu News, machine learning text collection, FOFA asset collection, Autohome, National Bureau of Statistics, Baidu keyword indexed-page counts, spider pool directories, Toutiao (Today's Headlines), Douban movie reviews, Ctrip, Xiaomi App Store, Anjuke, Tujia homestays ❤️❤️❤️. WeChat crawler showcase project:
:sparkling_heart: High available distributed ip proxy pool, powerd by Scrapy and Redis
Dark Web OSINT Tool
:zap: A distributed crawler for weibo, building with celery and requests.
TranslatedSina Weibo crawler: uses Python to crawl Sina Weibo data and download Weibo images and Weibo videos
持续维护的新浪微博采集工具🚀🚀🚀
新闻网页正文通用抽取器 Beta 版.
🚀🚀🚀feapder is an easy to use, powerful crawler framework | feapder是一款上手简单,功能强大的Python爬虫框架。内置AirSpider、Spider、TaskSpider、BatchSpider四种爬虫解决不同场景的需求。且支持断点续爬、监控报警、浏览器渲染、海量数据去重等功能。更有功能强大的爬虫管理系统feaplat为其提供方便的部署及调度
抖音逆向,抖音爬虫,抖音全部api、私信、直播间监听
Every web site provides APIs.