kafka connect从debezium主题自动提取主键

hmmo2u0o  于 2021-06-04  发布在  Kafka
关注(0)|答案(0)|浏览(325)

我能够成功地通过postgres连接debeziumcdc+kafka。我还能够通过jdbc-kafka连接到postgresql数据库,在简单的表书(serial pk,string title,string author)上获取debezium-kafka主题。在这个简单的表上插入、更新、删除都可以正常工作。但是,如果我想为多个postgres表设置一个kafka连接主题(debezium主题),该怎么办呢?我需要1:1转换数据。下面的示例使用相同的数据库,实际上它将被写入其他数据库,所以表名将是一致的。
下面是postgres配置属性文件

name=sink-postgres
connector.class=io.confluent.connect.jdbc.JdbcSinkConnector
value.converter=org.apache.kafka.connect.json.JsonConverter
value.converter.schemas.enable=true
topics=pokladna.public.books
connection.url=jdbc:postgresql://localhost:5434/postgres
connection.user=postgres
connection.password=postgres
dialect.name=PostgreSqlDatabaseDialect
table.name.format=books_kafka
transforms=unwrap
transforms.unwrap.type=io.debezium.transforms.ExtractNewRecordState
transforms.unwrap.drop.tombstones=false
auto.create=true
auto.evolve=true
insert.mode=upsert
delete.enabled=true
pk.fields=id
pk.mode=record_key

我有两个问题:
您知道如何设置pk.mode和pk.fields,以便在构造中的复合主键表中拾取更改吗?
更新并给出答案:简单的复合键就可以了(比如主键(question\ id,tag\ id)),只需添加到pk.fields列名。此外,约束下的键也可以正常工作(例如,constraint tag\u constraint rimary key(question\u id,tag\u id))—您不能对源和目标使用相同的数据库,因为约束名称不同。
如何从kafka消息中自动提取主列名?
下面我附上了debezium主题的输出。我想使用smt提取字段的名称,但它将工作的简单消息asi它显示在例子中。在下面的例子中,我想,我必须平展消息并提取payload.after.id。但是如果每个表中主键的名称不同,我该怎么办呢?

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --to                      
pokladna.public.books --property print.key=true --property key.separator=::::
{"schema":{"type":"struct","fields":[{"type":"int32","optional":false,"field":"id"}],"optional":false,
"name":"pokladna.public.books.Key"},"payload":{"id":8}}
::::
{"schema":{"type":"struct","fields":[{"type":"struct","fields":[{"type":"int32" ,"optional":false,"field":"id"},{"type":"string","optional":false,"field":"title"},{"type":"string","optional":true," field":"primary_author"}],"optional":true,"name":"pokladna.public.books.Value","field":"before"},{"type":"struct","fields":[{"type":"int32","optional":false,"field":"id"},{"type":"string","optional":false,"field":"title"},{"type":"string"                      ,"optional":true,"field":"primary_author"}],"optional":true,"name":"pokladna.public.books.Value",
"field":"after"},{"type":"struct","fields":[{"type":"string","optional":false,"field":"version"},{"type":"string","optional":false,"field":"connector"},{"type":"string","optional":false,"field":"name"},{"type":"int64","optional":false,"field":"ts_ms"},{"type":"string" ,"optional":true,"name":"io.debezium.data.Enum","version":1,
"parameters":{"allowed":"true,last,false"},"default":"false","field":"snapshot"},{"type":"string","optional":false,"field":"db"},{"type":"string","optional":false,
"field":"schema"} ,{"type":"string","optional":false,"field":"table"},{"type":"int64","optional":true,"field":"txId"},
{"type":"int64","optional":true,"field":"lsn"},{"type":"int64","optional":true,"field":"xmin"}],"optional":false,
"name":"io.debezium.connector.postgresql.Source","field":"source"},{"type":"string","optional":false,"field":"op"},{"type":"int64","optional":true,"field":"ts_ms"},{"type":"struct","fields":"type":"string","optional":false,"field":"id"},{"type":"int64", "optional":false,"field":"total_order"},{"type":"int64","optional":false,"field":"data_collection_order"}],"optional"                      :true,"field":"transaction"}],"optional":false,"name":"pokladna.public.books.Envelope"},
"payload":{"before":null,"after":{"id":8,"title":"jabadyba","primary_author":"jajajaja"},"source":{"version":"1.4.0-SNAPSHOT","connector":"postgresql"                      ,"name":"pokladna","ts_ms":1603718235441,"snapshot":"false","db":"postgres","schema":"public",
"table":"books","txId":716,"lsn":30457288,"xmin":null},"op":"c","ts_ms":1603718235812,"transaction":null}}

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题