Repository Index

Large-scale Self-supervised Pre-training Across Tasks, Languages, and Modalities

• nlp
• pre-trained-model
• unilm
• minilm

Unsupervised text tokenizer for Neural Network-based text generation.

• neural-machine-translation
• natural-language-processing
• word-segmentation

An open-source framework for detecting, redacting, masking, and anonymizing sensitive data (PII) across text, images, and structured data. Supports NLP, pattern matching, and customizable pipelines.

• python
• pii
• privacy
• data-anonymization

CoreNLP: A Java suite of core NLP tools for tokenization, sentence segmentation, NER, parsing, coreference, sentiment analysis, etc.

• natural-language-processing
• nlp
• nlp-parsing
• named-entity-recognition

Web mining module for Python, with tools for scraping, natural language processing, machine learning, network analysis and visualization.

• python
• machine-learning
• natural-language-processing
• web-mining

Module for automatic summarization of text documents and HTML pages.

• python
• lsa
• textteaser
• html-page

Go efficient fast multilingual NLP and text segmentation; support English, Chinese, Japanese and others.

• go
• segment
• nlp
• gse

The most accurate natural language detection library for Go, suitable for short text and mixed-language text

• natural-language-processing
• language-detection
• language-recognition
• language-classification

Neural question generation using transformers

• nlp
• nlg
• deep-learning
• transformer

🚀 Catalyst is a C# Natural Language Processing library built for speed. Inspired by spaCy's design, it brings pre-trained models, out-of-the box support for training word and document embeddings, and flexible entity recognition models.

• nlp
• natural-language-processing
• artificial-intelligence
• embeddings

Java API for Natural Language Generation. Originally developed by Ehud Reiter at the University of Aberdeen’s Department of Computing Science and co-founder of Arria NLG. This git repo is the official SimpleNLG version.

• simplenlg
• natural-language-generation
• nlg
• realiser

Parse human names into title, given, middle, family, suffix, nickname and maiden name. Typed, zero-dependency, CJK-aware.

• python
• text-processing
• text-parser
• python-module

RNNLG is an open source benchmark toolkit for Natural Language Generation (NLG) in spoken dialogue system application domains. It is released by Tsung-Hsien (Shawn) Wen from Cambridge Dialogue Systems Group under Apache License 2.0.

• natural-language-processing
• natural-language-generation
• nlg-dataset
• deep-learning

An Analysis Toolkit for Natural Language Generation (Translation, Captioning, Summarization, etc.)

Automatic text summarization

:wink: :cyclone: :strawberry: TextRank implementation in Golang with extendable features (summarization, phrase extraction) and multithreading (goroutine).

• go
• golang
• textrank
• summarization

Modular, fast NLP framework, compatible with Pytorch and spaCy, offering tailored support for French clinical notes.

• nlp
• medical
• text-mining
• clinical-data-warehouse

Rule based Answer Type classification system in Node.js

An NLP library for Uralic languages such as Finnish, Skolt Sami, Moksha and so on. Also supporting some non-Uralic languages such as Spanish, French, Arabic, Swedish, Norwegian, Russian and English. LLMs, FSTs and More!

• nlp-library
• uralic-languages
• finnish
• sami

A Go package for n-gram based text categorization, with support for utf-8 and raw text

A flexible and general-purpose ngrams library written in Ruby. Raingrams supports ngram sizes greater than 1, text/non-text grams, multiple parsing styles and open/closed vocabulary models.

• ruby
• ngrams

Tagify produces a set of tags from a given source. Source can be either an HTML page, a Markdown document or a plain text. Supports English, Russian, Chinese, Hindi, Spanish, Arabic, Japanese, German, Hebrew, French and Korean languages.

• tagify
• keywords
• keyword-extraction
• html

Surface realiser (part of a Natural Language Generation system) using Tree Adjoining Grammar

A tokenizer based on the dictionary and Bigram language models for Go. (Now only support chinese segmentation)

• tokenizer
• segmentation
• golang