Why is the ProducerFencedException not retried?

[FabioBentoLuiz]

Hello,
I am working on a Spring Boot application integrated with Kafka (consuming, processing, and producing), which uses transactions.

After refactoring, we started experiencing ProducerFencedExceptions, which I believe are related to transaction timeouts. The Kafka log looks like this:

[TransactionCoordinator id=0] Completed rollback of ongoing transaction for transactionId my-app8 due to timeout (kafka.coordinator.transaction.TransactionCoordinator); [data-plane-kafka-request-handler-7].

Breaking into smaller batches seems to solve the problem. However, we've noticed that whenever a ProducerFencedException occurs, the produced events are uncommitted (as expected), but the retry configuration (@RetryableTopic) is ignored, so the events do not land on the DLQ.

According to the Spring docs (https://docs.spring.io/spring-kafka/reference/kafka/annotation-error-handling.html#default-eh), the DefaultErrorHandler considers certain exceptions to be fatal and skips retries for such exceptions. However, ProducerFencedException is not listed.

It seems to me that the ProducerFencedException is treated as a "container-level" fatal exception. I am looking for further information and best practices to handle such errors properly in the future.

The listener ack-mode is manual_immediate.

[sobychacko]

Have you tried increasing your transaction timeout - both on the Spring side and on the kafka broker? The fact that smaller chunks work means that something might be slowing things down on the app side, maybe you want to debug things there. In terms of the ProducerFailedException not getting re-tried, I think that makes, since when a producer is fenced, its effectively done and a new producer needs to be created etc. so things might be in flux internally and the container may consider it as non-retryable. Did you try creating a custom error handler and see if you can address this particular exception somehow? In the short term, I would try to increase the timeouts or see what's slowing your system in the first place. Thanks!

[FabioBentoLuiz]

Hi @sobychacko, thank you for your support!

The transactions that timed out were the result of a poor design decision on my part, but a more robust solution is now in place.

What is still bugging me is that the listener somehow committed the offset, so the record was skipped and left unprocessed.

In my application, I see that the ConcurrentMessageListenerContainer is set with an AfterRollbackProcessor as follows:

boolean commitRecovered = true;
AfterRollbackProcessor<?,?> afterRollbackProcessor = new DefaultAfterRollbackProcessor<>(deadLetterPublishingRecover, backoff, kafkaTemplate, commitRecovered);
container.setAfterRollbackProcessor(afterRollbackProcessor);
container.setCommonErrorHandler(null);

The logs also show that the container was clearly not in a healthy state.

2025-06-25T08:05:11.123Z ERROR --- KafkaMessageListenerContainer: Stopping container due to an error.
... almost three minutes later
2025-06-25T08:07:50.900Z ERROR — KafkaMessageListenerContainer: Error while stopping the container.

In this scenario, is it possible for the recovery to fail but for the offset to be committed?