Apache Kafka is a distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications. Kafka is designed to handle real-time data feeds and is often used for building real-time streaming data pipelines that reliably get data between systems or applications.
When working with Kafka, you might encounter the OffsetOutOfRangeException. This error typically occurs when a consumer tries to fetch data from an offset that is no longer available on the server. This can happen if the offset is either too old and has been deleted due to log retention policies or if it is beyond the current log end offset.
The OffsetOutOfRangeException is thrown when a consumer requests an offset that is outside the range of available offsets for a partition. Each partition in Kafka has a log, and each record in the log is assigned a unique offset. If a consumer tries to read from an offset that has been deleted due to the log retention policy or is beyond the current end of the log, this exception is triggered.
Kafka brokers delete old log segments based on the configured retention policy, which can be time-based or size-based. If a consumer is inactive for a period longer than the retention period, it may attempt to read from an offset that has been deleted.
To resolve the OffsetOutOfRangeException, you can reset the consumer offset to a valid position. Here are the steps to do so:
First, determine which consumer group is encountering the issue. You can use the following command to list all consumer groups:
bin/kafka-consumer-groups.sh --bootstrap-server <broker-host>:9092 --list
Once you have identified the consumer group, describe it to see the current offsets and lag:
bin/kafka-consumer-groups.sh --bootstrap-server <broker-host>:9092 --describe --group <consumer-group>
Reset the offset to the latest or earliest available offset using the following command:
bin/kafka-consumer-groups.sh --bootstrap-server <broker-host>:9092 --group <consumer-group> --topic <topic-name> --reset-offsets --to-latest --execute
Alternatively, you can reset to the earliest offset:
bin/kafka-consumer-groups.sh --bootstrap-server <broker-host>:9092 --group <consumer-group> --topic <topic-name> --reset-offsets --to-earliest --execute
For more information on managing Kafka consumer offsets, you can refer to the Kafka Consumer Configurations and the Kafka Consumer Group Operations documentation.
Let Dr. Droid create custom investigation plans for your infrastructure.
Start Free POC (15-min setup) →
