# vchord_bm25 & pg_tokenizer

> Native BM25 ranking index and Block-WeakAnd full-text search for PostgreSQL (`vchord_bm25` & `pg_tokenizer`).

---

LLMS index: [llms.txt](/llms.txt)

---

**`vchord_bm25`** (maintained by Pigsty at [`pgsty/vchord_bm25`](https://github.com/pgsty/vchord_bm25)) is a native PostgreSQL extension built in Rust (`pgrx`) that implements the **Okapi BM25** ranking algorithm and the **Block-WeakAnd (Block-Max WAND)** Top-K index scan access method inside PostgreSQL.

Paired with **[`pg_tokenizer`](/bm25/tokenizer/)** (maintained by Pigsty at [`pgsty/pg_tokenizer`](https://github.com/pgsty/pg_tokenizer) for customizable multilingual tokenization and vocabulary models) or PostgreSQL's built-in `tsvector` analyzer, `vchord_bm25` brings Elasticsearch/Lucene-class lexical full-text ranking directly into your relational database—and combines seamlessly with **[`vchord`](/docs/)** for hybrid semantic + lexical search.


  - [Getting Started](/bm25/getting-started/) — Understand why BM25 matters, install vchord_bm25 and pg_tokenizer, and run your first ranked full-text query.
  - [Usage \& Hybrid Search](/bm25/usage/) — Build bm25 indexes, run Top-K queries, apply WHERE prefilters, and fuse BM25 with VectorChord dense vectors.
  - [Tokenizers \(pg\_tokenizer\)](/bm25/tokenizer/) — Configure text analyzers, stemmers, stopwords, synonyms, HuggingFace models, Chinese Jieba, and Japanese Lindera.
  - [Reference \& Benchmarks](/bm25/reference/) — Complete SQL types, operators, functions, GUC parameters, version evolution, and BEIR benchmark results.


## Highlights

- **True Okapi BM25 Scoring**: Supports inverse document frequency (IDF), term frequency saturation (\(k_1\)), and quantized 1-byte document length normalization (\(b\) and average document length \(\text{avgdl}\))—solving fundamental relevance limitations of PostgreSQL's native `ts_rank`.
- **Block-WeakAnd Index Acceleration**: Stores compressed posting lists with per-block maximum score upper bounds using delta-bitpacking, skipping non-competitive document blocks during Top-K queries without scanning entire inverted lists.
- **Multilingual Tokenization**: Works with `pg_tokenizer` (`llmlingua2`, `bert_base_uncased`, `wiki_tocken`, `gemma2b`, custom corpus-trained models, Chinese `jieba`, Japanese `lindera`) in `v0.1`–`v0.3`, and supports PostgreSQL's native `tsvector` directly in the next-generation `main` architecture.
- **Native Hybrid Search with VectorChord**: Run lexical BM25 (`vchord_bm25`) and dense vector search (`vchordrq`) inside a single SQL query using Reciprocal Rank Fusion (RRF) or ColBERT MaxSim reranking.

---

Section pages:

- [Getting Started](/bm25/getting-started/): Overview, installation, and quick start guide for vchord_bm25 and pg_tokenizer.
- [Usage & Hybrid Search](/bm25/usage/): Index management, Top-K querying, prefiltering, hybrid search with vchord, and Block-WeakAnd internals.
- [Tokenizers (pg_tokenizer)](/bm25/tokenizer/): Complete guide to pg_tokenizer: text analyzers, character/token filters, pretrained and custom models, and multilingual tokenization.
- [Reference & Benchmarks](/bm25/reference/): Complete SQL reference, GUC parameters, benchmarks, and version history for vchord_bm25 and pg_tokenizer.

---

Backlinks:

- [About](/about/)