apache kafka - the transaction.id created by flink is wrong - Stack Overflow

admin2025-04-20  0

I am using flink 1.20 with kafka-client 3.4.0. I am using KafkaSink with Exactly Once and checkpoints. I set the transactionIdPrefix and noticed there is a problem in the beginning of using my pipeline.

my transaction.id is auto created by the flink and the same for the client.id. as the flink use the checkpoint id , and the creation of the kafka producer is done before the the first checkpoint successfully triggered ( as "not all required tasks currently running.."), the first transaction.id is kafka-sink-0-1, the second kafka-sink-0-2. I expect the next one to be kafka-sink-0-3 but its kafka-sink-0-1 causing managment.InstanceAlreadyExistsException. the following tarnsaction.id are ok kafka-sink-0-3 ,kafka-sink-0-4....

I know the value kafka-sink-0-X is set by the value of the last checkpoint+1. is there a way to solve this issue?

转载请注明原文地址:http://conceptsofalgorithm.com/Algorithm/1745081460a283883.html

最新回复(0)