Text Analyzers & Filters

A Text Analyzer processes raw text through three sequential stages:

1. Character Filters (character_filters) — Zero or more character-level transformations on the full text string.
2. Pre-tokenizer (pre_tokenizer) — Exactly one splitter that breaks the string into initial tokens.
3. Token Filters (token_filters) — Zero or more token-level transformations, expansions, or filters.

SELECT create_text_analyzer('analyzer1', $$
character_filters = [
    "to_lowercase",
    { unicode_normalization = "nfkd" },
]
pre_tokenizer = "unicode_segmentation"
token_filters = [
    "skip_non_alphanumeric",
    { stopwords = "nltk_english" },
    { stemmer = "english_porter2" },
]
$$);

1. Character Filters (character_filters)

to_lowercase

Converts all characters in the input text to lowercase.

character_filters = ["to_lowercase"]
# or table syntax:
[character_filters.to_lowercase]

unicode_normalization

Applies Unicode Normalization Forms (UAX #15) to standardize composed/decomposed characters, accents, and full-width/half-width variants.

Supported modes:

• "nfc" — Normalization Form C (Canonical Decomposition followed by Canonical Composition)
• "nfd" — Normalization Form D (Canonical Decomposition)
• "nfkc" — Normalization Form KC (Compatibility Decomposition followed by Canonical Composition)
• "nfkd" — Normalization Form KD (Compatibility Decomposition)

[[character_filters]]
unicode_normalization = "nfkd"

2. Pre-Tokenizers (pre_tokenizer)

A text analyzer may specify at most one pre_tokenizer.

unicode_segmentation

Splits text on Unicode word boundaries according to the Unicode Standard Annex #29 (UAX #29) specification (via the Rust unicode-segmentation crate). Ideal for space-delimited and Western languages.

pre_tokenizer = "unicode_segmentation"
# or:
[pre_tokenizer.unicode_segmentation]

regex

Extracts tokens matching a custom regular expression (powered by the Rust regex crate):

[pre_tokenizer]
regex = '''\\w+'''

jieba (Chinese Word Segmentation)

Segments Chinese text into words using the jieba-rs dictionary and Hidden Markov Model (HMM):

[pre_tokenizer.jieba]
mode = "search"
enable_hmm = true

3. Token Filters (token_filters)

Token filters run in the exact order listed in your TOML array.

skip_non_alphanumeric

Drops any token that contains no alphanumeric characters (such as standalone punctuation marks ,, ., ! left behind by word segmentation).

[[token_filters]]
skip_non_alphanumeric = {}
# or in array shorthand:
token_filters = ["skip_non_alphanumeric"]

stemmer (Snowball Stemming — 30 Languages)

Reduces inflected words to their root stem using the Snowball stemming algorithms.

[[token_filters]]
stemmer = "english_porter2"

Supported stemmer values:

stopwords

Removes common stopwords that carry little lexical signal.

• Built-in dictionaries:
  • "lucene_english" — Apache Lucene standard English stopword list
  • "nltk_english" — NLTK standard English stopword list
  • "iso_english" — Stopwords ISO English stopword list
• Custom stopword dictionary: Create your own with create_stopwords(name, text) (one word per line) and reference it by name.

-- Create a custom stopword dictionary (one word per line)
SELECT create_stopwords('my_stopwords', $$
it
is
a
an
the
$$);

SELECT create_text_analyzer('text_analyzer1', $$
pre_tokenizer = "unicode_segmentation"
[[token_filters]]
stopwords = "my_stopwords"
$$);

SELECT apply_text_analyzer('it is a cat', 'text_analyzer1');
-- Output: {cat}

synonym

Normalizes synonymous terms to a single canonical token. Create a synonym dictionary with create_synonym(name, text) where each line contains space-separated synonyms (all words on a line are replaced by the first word on that line):

SELECT create_synonym('my_synonym', $$
postgres pgsql postgresql
index indices
$$);

SELECT create_text_analyzer('text_analyzer2', $$
pre_tokenizer = "unicode_segmentation"
[[token_filters]]
synonym = "my_synonym"
$$);

SELECT apply_text_analyzer('pgsql and postgresql have many indices', 'text_analyzer2');
-- Output: {postgres,and,postgres,have,many,index}

pg_dict (PostgreSQL Full-Text Dictionaries)

Delegates token filtering to any existing PostgreSQL Full-Text Search Dictionary (pg_ts_dict, including Simple, Ispell, Snowball, and Synonym dictionaries; note that Thesaurus dictionaries are not supported because they operate on multi-word phrases rather than single tokens):

[[token_filters]]
pg_dict = "english_stem"

ngram

Splits each token into character n-grams (useful for substring, prefix, CJK without dictionaries, or code/identifier search):

[[token_filters.ngram]]
min_gram = 2
max_gram = 3
preserve_original = true