pg_tokenizer Documentation
pg_tokenizer (maintained by Pigsty at pgsty/pg_tokenizer) is a standalone multilingual tokenization extension for PostgreSQL built in Rust (pgrx). It provides customizable text analyzers, character/token filters, and vocabulary models to convert raw text into integer token ID arrays (INT[]) and sparse bm25vector embeddings for vchord_bm25.
Install pg_tokenizer via Pigsty native Linux packages, Docker, Debian/ZIP archives, or cargo-pgrx.
How Text Analyzers and Models combine to turn raw text into token IDs.
Configure character filters, pre-tokenizers (Unicode, Regex, Jieba), stemmers (30 languages), stopwords, synonyms, and n-grams.
Use built-in models (llmlingua2, bert_base_uncased, wiki_tocken, gemma2b), train custom corpus models, or import HuggingFace & Lindera models.
Ready-to-use configurations for English, Chinese (Jieba), Japanese (Lindera), and custom domain corpora.
Complete reference for all tokenizer_catalog SQL functions and TOML configuration options.