Repository Index

TranslatedGenerate HD short videos from a topic or keyword with an automated AI workflow.

• python
• tiktok
• ai-video-generator
• content-creation

VoxCPM2: Tokenizer-Free TTS for Multilingual Speech Generation, Creative Voice Design, and True-to-Life Cloning

• audio
• deeplearning
• minicpm
• python

💫 Industrial-strength Natural Language Processing (NLP) in Python

• natural-language-processing
• data-science
• machine-learning
• python

TranslatedSelf-hosted TikTok, Douyin & Bilibili scraper and no-watermark video downloader — async REST API, MCP server for AI agents, CLI and web console, plus offline speech-to-text for downloaded videos. Self-healing identity pool, PostgreSQL archive, one docker compose up.

• tiktok
• douyin
• scraper
• crawler

tiktoken is a fast BPE tokeniser for use with OpenAI's models.

Formerly KrillinAI, VideoTranslation and Dubbing Tool for 120+ Languages. Open-source AI workspace for creators, powered by Codex. Video downloader, video translation, create videos, images, audios, and edits with Agents in one place. For your Social Media like YouTube, TikTok, Instagram, etc.

• video-transcription
• video-translation
• dubbing
• localization

Unsupervised text tokenizer for Neural Network-based text generation.

• neural-machine-translation
• natural-language-processing
• word-segmentation

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

• nlp
• natural-language-processing
• natural-language-understanding
• language-model

Download video and audio from 1000+ sites, transcribe locally with speaker labels, and summarize, translate, or chat with your media using your preferred AI provider.

• downloader
• facebook
• tiktok
• twitter

Minimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization.

CoreNLP: A Java suite of core NLP tools for tokenization, sentence segmentation, NER, parsing, coreference, sentiment analysis, etc.

• natural-language-processing
• nlp
• nlp-parsing
• named-entity-recognition

AutoML library for deep learning

• automl
• neural-architecture-search
• automated-machine-learning
• python

Stanford NLP Python library for tokenization, sentence segmentation, NER, and parsing of many human languages

• python
• nlp
• natural-language-processing
• machine-learning

Open source AI clip generator: turns long videos into viral 9:16 shorts with AI moment detection, face tracking, subtitles and dubbing. Self-host free with Docker (MIT), or use the cloud with GPU speed from $12/mo. MCP server and API for AI agents.

• clip-generator
• open-source
• shorts
• tiktok

🛰️ Track token usage across AI coding agents from your terminal. 🏅 Global leaderboard with trillions of tokens tracked.

• claude
• claude-code
• codex
• cursor

Serve large Qwen models fast on the GPUs you actually own. Qwen3.8-27B on a single 24 GB card with vLLM: 127 tok/s single-user (381 when the answer quotes the prompt), ~1,035 tok/s at 64 concurrent, 150k-262k context. vLLM patches, requant pipeline, benchmarks.

• kv-cache
• llm-inference
• local-llm
• quantization

Official Inference Code of EAGLE-4

• eagle
• gpu
• llm-inference
• sglang

Self-contained Japanese Morphological Analyzer written in pure Go

• japanese
• tokenizer
• nlp-library
• japanese-language

🐋 Run 284B DeepSeek-V4-Flash (<59GiB RAM, 44.4 tok/s pp, 9.1-14.7 tok/s) & 30B MoEs (25-111 tok/s) on a pure 64GB CPU at 256K ctx (0 GPUs). SwiGLU-H2, Stockfish-PVS L3 & 13ms Claude Code restore.

• avx512
• cpu-inference
• hpc
• llama-cpp

Translated斑海豹 (Spotted Seal) — Phocinae-Largha-150M-v1: a 144.3M bilingual decision model. Tiny model. Big decisions.

• bilingual
• calibration
• cpu-inference
• decision-making

Xolotl: run DeepSeek-V4.1-Flash, all 552B, as a local coding agent on one gaming PC (RTX 4090 + 110 GB RAM): ~33 tok/s on agent tasks, ~41 in fast mode, 512K context. Plus GLM-5.3-Flash, Qwen3.8 and LFM3-Omni. MoE experts tiered across VRAM, pinned RAM and NVMe, with Xolotl Direct NVMe->VRAM. OpenAI + Anthropic API.

• cuda
• deepseek
• deepseek-v4
• gguf

CogComp's Natural Language Processing Libraries and Demos: Modules include lemmatizer, ner, pos, prep-srl, quantifier, question type, relation-extraction, similarity, temporal normalizer, tokenizer, transliteration, verb-sense, and more.

• natural-language-processing
• nlp
• big-data
• data-mining

A multilingual command line sentence tokenizer in Golang

• sentence-tokenizer
• tokenizer
• cli
• sentences

The official code of FrameMorrow: Future-Guided Frame Selection with Prospective Tokens for Long-Horizon Video Generation.