Apache kafka dead letter queue understanding and implementation idea

Kafka Dead Letter Queue

We don’t have the documentation which would be sufficient for implementing the dead letter queue in your application.

In this blog we are going to focus on implementing the dead letter topic/queue for our Spring Boot application.

Let’s create and understand the scenario, say we have an inbound message kafka topic named as “TopicOne” and the consumer group as “TopicOneConsumerGroup” and the received message is sent to the outbound Messaging Queue/ Outbound Topic. Our spring boot application listens to the inbound kafka topic using below given annotation,

@KafkaListener(topics = "TopicOne", groupId = "TopicOneConsumerGroup")

Spring within itself has the ability to identify the failed message processing. Which then forwards the failed messages to the DLT topic by itself.

So when our application starts consuming messages, let’s say out application encountered an exception, then the spring framework by itself identifies the inbound topic from which our application received the inbound message, which is then used by spring to navigate the failed message to Dead Letter Topic / DLT.

For redirecting the failed messages to DLT, spring suffixes/appends the .dlt to the inbound message topic from which we received the message which is now failing. So the spring tries to find the DLT topic on the kafka server which it formed by appending .dlt, it will probably look like this TopicOne.dlt.

In my case, I ensured that I’ve TopicOne.dlt topic created over the kafka server. So after my messages were directed to the DLT topic named TopicOne.dlt.

Now we should have DLT listener in our application which is annotated with @KafkaListener annotation and has properties as below

@KafkaListener(topics = "TopicOne.dlt", groupId = "TopicOneConsumerGroup")

This will automatically start to ingest the failed messages from DLT topic into our application for reprocessing as soon as we start the spring boot application.

But we may not need to automatically start the DLT listener as it is unnecessary to run DLT listener as we will not have failed messages over DLT initially, so for this purpose we can add another property as autoStartUp as false as shown below

@KafkaListener(topics = "TopicOne.dlt", groupId = "TopicOneConsumerGroup", autoStartup = "false")

The default value for autoStartUp is True.

Now, if we specify the property autoStartUp as false, we will need a way to start/activate the listener as it’s not active due to the above mentioned property, hence we will add the identifier property id as shown below

@KafkaListener(id = "DLT_ID", topics = "TopicOne.dlt", groupId = "TopicOneConsumerGroup", autoStartup = "false")

So this id parameter will give an identity to this listener, in our case the value is DLT_ID.

Now we will need to activate this DLT somehow. So for this reason we will have to trigger the DLT listener, as shown below

@Autowired
private KafkaListenerEndpointRegistry kafkaListenerEndpointRegistry;

@GetMapping("/handler/name")
public String startDLT() {
    this.kafkaListenerEndpointRegistry.getListenerContainer("DLT_ID").start();
}Spring within itself has the ability to identify the failed message processing. Which then forwards the failed messages to the DLT topic by itself.

And at some point we should stop the activated DLT handler, we can stop the DLT listener as shown below

@GetMapping("/stop/handler_name")
public String stopDLT() {
    this.kafkaListenerEndpointRegistry.getListenerContainer("DLT_ID").stop();
}

This is a quick guide, please let me know if you have any queries regarding same, or if you find anything misleading.