Skip to content

Commit 0e327f0

Browse files
gaoxueyugaoxueyu
authored
Merge pull request #210 from bigplaice/wal2json
add wal2json doc
2 parents 2f7b8e4 + 57b0e78 commit 0e327f0

4 files changed

Lines changed: 274 additions & 0 deletions

File tree

‎CN/modules/ROOT/nav.adoc‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,7 @@
3535
**** xref:master/ecosystem_components/pgaudit.adoc[pgaudit]
3636
**** xref:master/ecosystem_components/pgrouting.adoc[pgrouting]
3737
**** xref:master/ecosystem_components/system_stats.adoc[system_stats]
38+
**** xref:master/ecosystem_components/wal2json.adoc[wal2json]
3839
** IvorySQL架构设计
3940
*** 查询处理
4041
**** xref:master/architecture/dual_parser.adoc[双parser]

‎CN/modules/ROOT/pages/master/ecosystem_components/wal2json.adoc‎

Lines changed: 137 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,137 @@
1+
2+
:sectnums:
3+
:sectnumlevels: 5
4+
5+
= wal2json
6+
7+
== 概述
8+
wal2json 是一个用于 PostgreSQL 逻辑解码的输出插件，这个插件为每个事务生成一个JSON对象。
9+
10+
== 安装
11+
12+
[TIP]
13+
源码安装环境为 Ubuntu 24.04(x86_64)，环境中已经安装了IvorySQL5及以上版本，安装路径为/usr/ivory-5
14+
15+
=== 源码安装
16+
17+
[literal]
18+
----
19+
# 从 https://github.com/eulerto/wal2json/releases/tag/wal2json_2_6 下载 2.6 的源码包 wal2json_2_6.zip
20+
21+
unzip wal2json_2_6.zip
22+
cd wal2json_2_6
23+
24+
# 编译安装插件
25+
make PG_CONFIG=/usr/ivory-5/bin/pg_config
26+
make PG_CONFIG=/usr/ivory-5/bin/pg_config install
27+
28+
----
29+
30+
[TIP]
31+
如果出现找不到xlocale.h的错误，需要手动修改 /usr/ivory-5/include/postgresql/server/pg_config.h
32+
删除或者注释掉 #define HAVE_XLOCALE_H 1 这一行
33+
34+
=== 修改数据库配置文件
35+
36+
修改 postgresql.conf 文件
37+
启用wal_level为logical，并设置最大复制槽数和最大wal发送进程数
38+
39+
[literal]
40+
----
41+
wal_level = logical
42+
max_replication_slots = 10
43+
max_wal_senders = 10
44+
----
45+
46+
pg_hba.conf不需要修改，有以下内容即可（仅限本地链接测试）：
47+
[literal]
48+
----
49+
local replication all trust
50+
host replication all 127.0.0.1/32 trust
51+
host replication all ::1/128 trust
52+
----
53+
54+
重启数据库后配置生效。
55+
56+
== 使用
57+
58+
在第一个终端中执行命令：
59+
[literal]
60+
----
61+
sudo -u ivorysql /usr/ivory-5/bin/bin/pg_recvlogical -d postgres --slot wal2json_slot --create-slot -P wal2json
62+
63+
启动监听，实时输出变更的JSON格式
64+
sudo -u ivorysql /usr/ivory-5/bin/bin/pg_recvlogical -d postgres --slot wal2json_slot --start -o pretty-print=1 -f -
65+
----
66+
67+
在第二个终端中连接数据库：
68+
[literal]
69+
----
70+
/usr/ivory-5/bin/psql -d postgres -p 1521
71+
----
72+
73+
执行下面的SQL语句：
74+
[literal]
75+
----
76+
CREATE TABLE test_cdc (id int primary key, name varchar(50));
77+
INSERT INTO test_cdc VALUES (1, 'test1');
78+
UPDATE test_cdc SET name = 'test1_update' WHERE id = 1;
79+
DELETE FROM test_cdc WHERE id = 1;
80+
DROP TABLE test_cdc;
81+
----
82+
83+
此时在第一个终端上可以看到下面的输出：
84+
[literal]
85+
----
86+
{
87+
"change": [
88+
]
89+
}
90+
{
91+
"change": [
92+
{
93+
"kind": "insert",
94+
"schema": "public",
95+
"table": "test_cdc",
96+
"columnnames": ["id", "name"],
97+
"columntypes": ["integer", "sys.oravarcharbyte(50)"],
98+
"columnvalues": [1, "test1"]
99+
}
100+
]
101+
}
102+
{
103+
"change": [
104+
{
105+
"kind": "update",
106+
"schema": "public",
107+
"table": "test_cdc",
108+
"columnnames": ["id", "name"],
109+
"columntypes": ["integer", "sys.oravarcharbyte(50)"],
110+
"columnvalues": [1, "test1_update"],
111+
"oldkeys": {
112+
"keynames": ["id"],
113+
"keytypes": ["integer"],
114+
"keyvalues": [1]
115+
}
116+
}
117+
]
118+
}
119+
{
120+
"change": [
121+
{
122+
"kind": "delete",
123+
"schema": "public",
124+
"table": "test_cdc",
125+
"oldkeys": {
126+
"keynames": ["id"],
127+
"keytypes": ["integer"],
128+
"keyvalues": [1]
129+
}
130+
}
131+
]
132+
}
133+
{
134+
"change": [
135+
]
136+
}
137+
----

‎EN/modules/ROOT/nav.adoc‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,7 @@
3434
*** xref:master/ecosystem_components/pgaudit.adoc[pgaudit]
3535
*** xref:master/ecosystem_components/pgrouting.adoc[pgrouting]
3636
*** xref:master/ecosystem_components/system_stats.adoc[system_stats]
37+
*** xref:master/ecosystem_components/wal2json.adoc[wal2json]
3738
* IvorySQL Architecture Design
3839
** Query Processing
3940
*** xref:master/architecture/dual_parser.adoc[Dual Parser]

‎EN/modules/ROOT/pages/master/ecosystem_components/wal2json.adoc‎

Lines changed: 135 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,135 @@
1+
2+
:sectnums:
3+
:sectnumlevels: 5
4+
5+
= wal2json
6+
7+
== Overview
8+
wal2json is an output plugin for logical decoding. It generates JSON object for each transaction.
9+
10+
== Installation
11+
12+
[TIP]
13+
The source code installation environment is Ubuntu 24.04 (x86_64), in which IvorySQL 5 or a later version has been installed. The installation path is /usr/ivory-5.
14+
15+
=== Source Code Installation
16+
17+
[literal]
18+
----
19+
# download source code package from: https://github.com/eulerto/wal2json/releases/tag/wal2json_2_6
20+
21+
unzip wal2json_2_6.zip
22+
cd wal2json_2_6
23+
24+
# compile and install the extension
25+
make PG_CONFIG=/usr/ivory-5/bin/pg_config
26+
make PG_CONFIG=/usr/ivory-5/bin/pg_config install
27+
----
28+
29+
[TIP]
30+
If there is error "xlocale.h: No such file or directory" during compilation, user should remove the
31+
line of "#define HAVE_XLOCALE_H 1" from file /usr/ivory-5/include/postgresql/server/pg_config.h.
32+
33+
=== Modify the configuration file
34+
35+
Modify the postgresql.conf file to set wal_level as "logical", and set max_replication_slots/max_wal_senders.
36+
[literal]
37+
----
38+
wal_level = logical
39+
max_replication_slots = 10
40+
max_wal_senders = 10
41+
----
42+
43+
Make sure the following content to be in pg_hba.conf.
44+
[literal]
45+
----
46+
local replication all trust
47+
host replication all 127.0.0.1/32 trust
48+
host replication all ::1/128 trust
49+
----
50+
51+
Then restart the database.
52+
53+
== Use
54+
55+
Open the first terminal and execute command:
56+
57+
[literal]
58+
----
59+
sudo -u ivorysql /usr/ivory-5/bin/pg_recvlogical -d postgres --slot wal2json_slot --create-slot -P wal2json
60+
61+
Start monitoring, output the changes in JSON format and in real time.
62+
sudo -u ivorysql /usr/ivory-5/bin/pg_recvlogical -d postgres --slot wal2json_slot --start -o pretty-print=1 -f -
63+
----
64+
65+
Connect database in the second terminal:
66+
[literal]
67+
----
68+
/usr/ivory-5/bin/psql -d postgres -p 1521
69+
----
70+
71+
Execute the following SQL statement:
72+
[literal]
73+
----
74+
CREATE TABLE test_cdc (id int primary key, name varchar(50));
75+
INSERT INTO test_cdc VALUES (1, 'test1');
76+
UPDATE test_cdc SET name = 'test1_update' WHERE id = 1;
77+
DELETE FROM test_cdc WHERE id = 1;
78+
DROP TABLE test_cdc;
79+
----
80+
81+
The following output will appear in the first terminal:
82+
[literal]
83+
----
84+
{
85+
"change": [
86+
]
87+
}
88+
{
89+
"change": [
90+
{
91+
"kind": "insert",
92+
"schema": "public",
93+
"table": "test_cdc",
94+
"columnnames": ["id", "name"],
95+
"columntypes": ["integer", "sys.oravarcharbyte(50)"],
96+
"columnvalues": [1, "test1"]
97+
}
98+
]
99+
}
100+
{
101+
"change": [
102+
{
103+
"kind": "update",
104+
"schema": "public",
105+
"table": "test_cdc",
106+
"columnnames": ["id", "name"],
107+
"columntypes": ["integer", "sys.oravarcharbyte(50)"],
108+
"columnvalues": [1, "test1_update"],
109+
"oldkeys": {
110+
"keynames": ["id"],
111+
"keytypes": ["integer"],
112+
"keyvalues": [1]
113+
}
114+
}
115+
]
116+
}
117+
{
118+
"change": [
119+
{
120+
"kind": "delete",
121+
"schema": "public",
122+
"table": "test_cdc",
123+
"oldkeys": {
124+
"keynames": ["id"],
125+
"keytypes": ["integer"],
126+
"keyvalues": [1]
127+
}
128+
}
129+
]
130+
}
131+
{
132+
"change": [
133+
]
134+
}
135+
----

0 commit comments

Comments
 (0)