Apache Kafka is a distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications. Kafka is designed to handle real-time data feeds and is often used for building real-time streaming data pipelines and applications that adapt to the data streams.
When working with Kafka topics, you might encounter the InvalidRetentionMsException. This exception typically surfaces when there is an issue with the retention period configuration of a Kafka topic. The retention period dictates how long Kafka retains messages in a topic before they are eligible for deletion.
Developers may notice that Kafka is not retaining messages for the expected duration, or an error message is logged indicating an InvalidRetentionMsException. This can disrupt data processing workflows that rely on specific retention periods.
The InvalidRetentionMsException is thrown when the retention period set for a Kafka topic is not valid. This could be due to setting a negative value, a value that exceeds the maximum allowed, or a non-numeric value. Kafka requires the retention period to be specified in milliseconds, and it must be a positive integer.
To resolve the InvalidRetentionMsException, you need to ensure that the retention period is set correctly. Follow these steps:
First, check the current retention settings for the topic. You can do this using the Kafka command-line tool:
bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics --entity-name your_topic_name --describe
This command will display the current configuration, including the retention period.
To set a valid retention period, use the following command, replacing your_topic_name with your actual topic name and retention_ms with the desired retention period in milliseconds:
bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics --entity-name your_topic_name --alter --add-config retention.ms=604800000
In this example, 604800000 milliseconds corresponds to 7 days.
After updating the retention period, validate the changes by describing the topic configuration again:
bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics --entity-name your_topic_name --describe
Ensure that the retention.ms value reflects your intended configuration.
For more information on Kafka topic configurations, refer to the official Kafka documentation. Additionally, consider exploring the Kafka Quickstart Guide for a comprehensive overview of setting up and managing Kafka topics.
Let Dr. Droid create custom investigation plans for your infrastructure.
Start Free POC (15-min setup) →
