Repository Index

Large-scale Self-supervised Pre-training Across Tasks, Languages, and Modalities

• nlp
• pre-trained-model
• unilm
• minilm

Unsupervised text tokenizer for Neural Network-based text generation.

• neural-machine-translation
• natural-language-processing
• word-segmentation

An open-source framework for detecting, redacting, masking, and anonymizing sensitive data (PII) across text, images, and structured data. Supports NLP, pattern matching, and customizable pipelines.

• python
• pii
• privacy
• data-anonymization

An NLP library for building bots, with entity extraction, sentiment analysis, automatic language identify, and so more

• nlp
• nlu
• nodejs
• sentiment-analysis

Go efficient fast multilingual NLP and text segmentation; support English, Chinese, Japanese and others.

• go
• segment
• nlp
• gse

Neural question generation using transformers

• nlp
• nlg
• deep-learning
• transformer

🚀 Catalyst is a C# Natural Language Processing library built for speed. Inspired by spaCy's design, it brings pre-trained models, out-of-the box support for training word and document embeddings, and flexible entity recognition models.

• nlp
• natural-language-processing
• artificial-intelligence
• embeddings

RNNLG is an open source benchmark toolkit for Natural Language Generation (NLG) in spoken dialogue system application domains. It is released by Tsung-Hsien (Shawn) Wen from Cambridge Dialogue Systems Group under Apache License 2.0.

• natural-language-processing
• natural-language-generation
• nlg-dataset
• deep-learning

Implementation of various topic models

The repository host gives no description.

Modular, fast NLP framework, compatible with Pytorch and spaCy, offering tailored support for French clinical notes.

• nlp
• medical
• text-mining
• clinical-data-warehouse

Rule based Answer Type classification system in Node.js

An NLP library for Uralic languages such as Finnish, Skolt Sami, Moksha and so on. Also supporting some non-Uralic languages such as Spanish, French, Arabic, Swedish, Norwegian, Russian and English. LLMs, FSTs and More!

• nlp-library
• uralic-languages
• finnish
• sami

A Go package for n-gram based text categorization, with support for utf-8 and raw text

A natural language date and time parser for Common Lisp

A flexible and general-purpose ngrams library written in Ruby. Raingrams supports ngram sizes greater than 1, text/non-text grams, multiple parsing styles and open/closed vocabulary models.

• ruby
• ngrams

Tagify produces a set of tags from a given source. Source can be either an HTML page, a Markdown document or a plain text. Supports English, Russian, Chinese, Hindi, Spanish, Arabic, Japanese, German, Hebrew, French and Korean languages.

• tagify
• keywords
• keyword-extraction
• html

Ruby gem to calculate statistics from text to determine readability, complexity and grade level of a particular corpus.

• textstat
• flesch-kincaid-grade
• smog
• flesch-reading-ease

A tokenizer based on the dictionary and Bigram language models for Go. (Now only support chinese segmentation)

• tokenizer
• segmentation
• golang

Investigating multilingual language models (BERT) by using them for NER in German and English