Large-scale Self-supervised Pre-training Across Tasks, Languages, and Modalities
- nlp
- pre-trained-model
- unilm
- minilm
Repository Index
47,133 repositories indexed
1–24 of 26 · Most stars
tiktoken is a fast BPE tokeniser for use with OpenAI's models.
Offline, privacy-first grammar checker. Fast, open-source, Rust-powered
- developer-tools
- english-language
- grammar-checker
- rust
Pretrained models for TensorFlow.js
Easy-to-use and powerful LLM and SLM library with awesome model zoo.
- nlp
- embedding
- bert
- ernie
Multi-layer Recurrent Neural Networks (LSTM, GRU, RNN) for character-level language models in Torch
An open-source framework for detecting, redacting, masking, and anonymizing sensitive data (PII) across text, images, and structured data. Supports NLP, pattern matching, and customizable pipelines.
- python
- pii
- privacy
- data-anonymization
CoreNLP: A Java suite of core NLP tools for tokenization, sentence segmentation, NER, parsing, coreference, sentiment analysis, etc.
- natural-language-processing
- nlp
- nlp-parsing
- named-entity-recognition
Web mining module for Python, with tools for scraping, natural language processing, machine learning, network analysis and visualization.
- python
- machine-learning
- natural-language-processing
- web-mining
Open-source offline translation library written in Python
- python
- machine-translation
- transformers
- translation
Module for automatic summarization of text documents and HTML pages.
- python
- lsa
- textteaser
- html-page
A full spaCy pipeline and models for scientific/biomedical documents.
- scientific-documents
- spacy
- custom-pipes
- nlp
Self-contained Machine Learning and Natural Language Processing library in Go
What's in your data? Extract schema, statistics and entities from datasets
- python
- privacy
- pii
- npi
🚀 Catalyst is a C# Natural Language Processing library built for speed. Inspired by spaCy's design, it brings pre-trained models, out-of-the box support for training word and document embeddings, and flexible entity recognition models.
Selected Machine Learning algorithms for natural language processing and semantic analysis in Golang
- go
- golang
- natural-language-processing
- nlp
No-Frills Functional Programming Lib Augmenting Javascript/Node.js
- functional-programming
- pure-function
- recursion
- javascript
Implementation of various topic models
A 197 KB local semantic verifier for AI agent tool calls. Exit code 0 is not proof.
Common Lisp NLP toolset
R package for stylometric analyses
Apache cTAKES is a Natural Language Processing (NLP) platform for clinical text.
- clinical
- nlp
- bioinformatics
- ai
An NLP library for Uralic languages such as Finnish, Skolt Sami, Moksha and so on. Also supporting some non-Uralic languages such as Spanish, French, Arabic, Swedish, Norwegian, Russian and English. LLMs, FSTs and More!
- nlp-library
- uralic-languages
- finnish
- sami
A flexible and general-purpose ngrams library written in Ruby. Raingrams supports ngram sizes greater than 1, text/non-text grams, multiple parsing styles and open/closed vocabulary models.
- ruby
- ngrams