Repository Index

Implement a ChatGPT-like LLM in PyTorch from scratch, step by step

• gpt
• large-language-models
• llm
• python

A non-validating SQL parser module for Python

• formatter
• python
• sql
• sql-formatter

Parser Building Toolkit for JavaScript

• javascript
• typescript
• parser-library
• parsing

Self-contained Japanese Morphological Analyzer written in pure Go

• japanese
• tokenizer
• nlp-library
• japanese-language

🌭 Mustard is a Swift library for tokenizing strings when splitting by whitespace doesn't cut it.

• swift
• substrings
• tokenizer

CogComp's Natural Language Processing Libraries and Demos: Modules include lemmatizer, ner, pos, prep-srl, quantifier, question type, relation-extraction, similarity, temporal normalizer, tokenizer, transliteration, verb-sense, and more.

• natural-language-processing
• nlp
• big-data
• data-mining

A multilingual command line sentence tokenizer in Golang

• sentence-tokenizer
• tokenizer
• cli
• sentences

Tokenizer (lexer) for anything

• golang
• tokenizer
• parse
• tokenizing

.NET 7 SDK for OpenAI with a Blazor Server playground

• openai
• openai-api
• sdk
• sdk-dotnet

A simple tokenizer in Ruby for NLP tasks.

• ruby
• nlp
• natural-language-processing
• tokenizer

Naive Bayes text classification implementation as an OmniCat classifier strategy. (#ruby #naivebayes)

• naive-bayes-classifier
• ruby
• text-classification
• sentiment-analysis

A tokenizer based on the dictionary and Bigram language models for Go. (Now only support chinese segmentation)

• tokenizer
• segmentation
• golang

The shamoji (杓文字) is a word filtering package

• go
• tokenizer
• word-filter

Crystal implementation of the Token-Oriented Object Notation(TOON) serialization format

• data-format
• llm
• serialization
• tokenizer