带有mssql的kafka jdbc连接器仅流100行

368yc8dk  于 2021-06-06  发布在  Kafka
关注(0)|答案(1)|浏览(348)

我正在使用标准的汇合jdbc连接器和microsoft的jdbc驱动程序连接到microsoft sql server数据库。对于一张特殊的table mode=timestamp 给它一个timestamp列或者我把它设置为 mode=bulk ,我只得到100个结果。从来没有比这更多的流。下面是我连接日志的结尾一直在说的:

[2018-09-19 12:51:51,393] INFO WorkerSourceTask{id=MES-0} Source task finished initialization and start (org.apache.kafka.connect.runtime.WorkerSourceTask:199)
[2018-09-19 12:51:51,412] INFO After filtering the tables are: "MES2ERP"."dbo"."BatchDetails" (io.confluent.connect.jdbc.source.TableMonitorThread:175)
[2018-09-19 12:51:51,412] INFO Finished starting connectors and tasks (org.apache.kafka.connect.runtime.distributed.DistributedHerder:868)
[2018-09-19 12:51:51,475] INFO Cluster ID: GMY7vibUSHGp5xRekGDEwg (org.apache.kafka.clients.Metadata:285)
[2018-09-19 12:52:51,069] INFO WorkerSourceTask{id=MES-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:397)
[2018-09-19 12:52:51,070] INFO WorkerSourceTask{id=MES-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:414)
[2018-09-19 12:52:51,077] INFO WorkerSourceTask{id=MES-0} Finished commitOffsets successfully in 7 ms (org.apache.kafka.connect.runtime.WorkerSourceTask:496)
[2018-09-19 12:53:51,082] INFO WorkerSourceTask{id=MES-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:397)
[2018-09-19 12:53:51,082] INFO WorkerSourceTask{id=MES-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:414)

我意识到 batch.max.rows 默认值是100,但是 mode=timestamp ,我应该让表中的所有行都流化,而不是在100之后停止。我希望 bulk 最终也会流式传输所有内容,或者至少每100个重复一次 table.poll.interval.ms ,但它不。。。就这么停了。知道怎么回事吗?
哦,我的table上有成百上千行。
这是我的连接器配置

name=JSON_OCN_MES-BatchDetails
connector.class=io.confluent.connect.jdbc.JdbcSourceConnector
key.converter.schemas.enable=false
key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter.schemas.enable=false
value.converter=org.apache.kafka.connect.json.JsonConverter
connection.url=jdbc:sqlserver://####;databaseName=####;user=####;password=####;
table.whitelist=BatchDetails
mode=bulk
validate.non.null=false
topic.prefix=JSON_OCN_MES-
aiqt4smr

aiqt4smr1#

您可以尝试更改连接器中的默认(100)batch.max.rows设置

"batch.max.rows": "1234",

相关问题