47,133 repositories indexed
1–24 of 36 · Most stars
TranslatedChinese word segmentation, part-of-speech tagging, named entity recognition, dependency parsing, constituency parsing, semantic dependency parsing, semantic role labeling, coreference resolution, style transfer, semantic similarity, new word discovery, keyword and keyphrase extraction, automatic summarization, text classification and clustering, pinyin and Simplified/Traditional Chinese conversion, natural language processing
TranslatedJieba (结巴, "stutter") Chinese word segmentation
Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
💫 Industrial-strength Natural Language Processing (NLP) in Python
Large-scale Self-supervised Pre-training Across Tasks, Languages, and Modalities
tiktoken is a fast BPE tokeniser for use with OpenAI's models.
Topic Modelling for Humans
NLTK Source
A very simple framework for state-of-the-art Natural Language Processing (NLP)
Easy-to-use and powerful LLM and SLM library with awesome model zoo.
Minimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization.
Web mining module for Python, with tools for scraping, natural language processing, machine learning, network analysis and visualization.
Stanford NLP Python library for tokenization, sentence segmentation, NER, and parsing of many human languages
Open-source offline translation library written in Python
TranslatedChinese characters to pinyin converter (pypinyin)
VADER Sentiment Analysis. VADER (Valence Aware Dictionary and sEntiment Reasoner) is a lexicon and rule-based sentiment analysis tool that is specifically attuned to sentiments expressed in social media, and works well on texts from other domains.
Easily train your own text-generating neural network of any size and complexity on any text dataset with a few lines of code.
Module for automatic summarization of text documents and HTML pages.
📐 Compute distance between sequences. 30+ algorithms, pure python implementation, common interface, optional external libs usage.
Seamlessly integrate LLMs into scikit-learn.
TranslatedNon-autoregressive System 1 decision engine, designed for structured, typed decision-making scenarios. DeepOpen Multilingual, non-autoregressive System 1 decision engine.
Toolkit for Machine Learning, Natural Language Processing, and Text Generation, in TensorFlow. This is part of the CASL project: http://casl-project.ai/
Bioinformatics'2020: BioBERT: a pre-trained biomedical language representation model for biomedical text mining
A full spaCy pipeline and models for scientific/biomedical documents.