Ecosystem Integration: pg_tokenizer - #303
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
There are a few documented formatting consistency issues (including a likely mixed line-ending change in CN nav) that should be corrected before merging to avoid noisy diffs and inconsistent docs style.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Adds bilingual documentation for integrating the pg_tokenizer extension into IvorySQL’s ecosystem docs, including build/install steps, tokenizer/analyzer configuration examples, trigger-based token generation, and an Oracle-mode identifier-case workaround discovered during validation.
Changes:
- Added new English and Chinese ecosystem component pages for
pg_tokenizer(build/install, usage examples, Oracle-mode notes). - Linked
pg_tokenizerinto the English/Chinese ecosystem overview tables. - Added
pg_tokenizerto the English/Chinese navigation trees.
File summaries
| File | Description |
|---|---|
| EN/modules/ROOT/pages/master/ecosystem_components/pg_tokenizer.adoc | New English documentation page for building/installing and using pg_tokenizer, including Oracle-mode caveat. |
| CN/modules/ROOT/pages/master/ecosystem_components/pg_tokenizer.adoc | New Chinese documentation page mirroring the English pg_tokenizer guidance. |
| EN/modules/ROOT/pages/master/ecosystem_components/ecosystem_overview.adoc | Adds pg_tokenizer entry to the English ecosystem components overview table. |
| CN/modules/ROOT/pages/master/ecosystem_components/ecosystem_overview.adoc | Adds pg_tokenizer entry to the Chinese ecosystem components overview table. |
| EN/modules/ROOT/nav.adoc | Adds pg_tokenizer page to the English navigation under ecosystem components. |
| CN/modules/ROOT/nav.adoc | Adds pg_tokenizer page to the Chinese navigation and adjusts the surrounding entries. |
Review details
- Files reviewed: 6/6 changed files
- Comments generated: 4
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| *** xref:master/ecosystem_components/set_user.adoc[set_user] | ||
| *** xref:master/ecosystem_components/pg_tokenizer.adoc[pg_tokenizer] |
There was a problem hiding this comment.
Normalized the Chinese navigation file to a single CRLF newline style.
|
|
||
| [source,ini] | ||
| ---- | ||
| shared_preload_libraries = 'gb18030_2022,liboracle_parser,ivorysql_ora,pg_tokenizer' |
There was a problem hiding this comment.
Added spaces after commas in the preload example.
| |*28*| xref:master/ecosystem_components/zhparser_en.adoc[zhparser] | master branch | PostgreSQL extension for full-text search of Chinese language (Mandarin Chinese). It implements a Chinese language parser base on the | Search engine、keyword extraction | ||
| |*29*| xref:master/ecosystem_components/pgbackrest.adoc[pgBackRest] | 2.58.0 | pgBackRest is a reliable backup and restore solution for PostgreSQL that seamlessly scales up to the largest databases and workloads | Disaster recovery backup, large database backup, off-site/multi-tier disaster recovery | ||
| | *30* | xref:master/ecosystem_components/set_user.adoc[set_user] | REL4_2_0 | PostgreSQL security auditing extension with controlled role switching, supporting allowlists, enforced auditing, and blocking of high-risk operations | Controlled role switching, privilege management, audit logging | ||
| | *31* | xref:master/ecosystem_components/pg_tokenizer.adoc[pg_tokenizer] | 0.1.1 | Configurable in-database text analyzers and tokenizers with multilingual processing and trigger-based token generation | Full-text search, RAG preprocessing, multilingual tokenization |
There was a problem hiding this comment.
Normalized the new index cell to match the existing table format.
|
|
||
| [source,ini] | ||
| ---- | ||
| shared_preload_libraries = 'gb18030_2022,liboracle_parser,ivorysql_ora,pg_tokenizer' |
There was a problem hiding this comment.
Added spaces after commas in the preload example.
|
🚀 IvorySQL-Docs Preview Ready
|
|
🚀 IvorySQL-Docs Preview Ready
|
Summary
Validation
Built pg_tokenizer 0.1.1 from source without extension source changes against the official IvorySQL 5.4 image (PostgreSQL 18.4) on x86_64 Linux.
CREATE EXTENSION pg_tokenizer CASCADE: passed, version 0.1.1ivorysql.identifier_case_switch = normal: passedThe documentation explains why the Oracle-mode setting is required and includes the exact workaround for reproducible testing.
Fixes IvorySQL/IvorySQL#1332