AWS Kinesis is a platform on AWS to collect, process, and analyze real-time, streaming data. It allows developers to build applications that can continuously ingest and process large streams of data records in real-time. Kinesis is particularly useful for scenarios such as log and event data collection, real-time analytics, and data ingestion for machine learning models.
When using AWS Kinesis, you might encounter the ProvisionedThroughputExceededException. This error indicates that the requested throughput exceeds the shard limit for the stream. It is a common issue when the data rate being sent to the stream is too high for the current configuration.
The ProvisionedThroughputExceededException is triggered when the data rate sent to a shard exceeds its capacity. Each shard in a Kinesis stream has a specific throughput limit: 1 MB/sec or 1,000 records/sec for writes, and 2 MB/sec for reads. When these limits are exceeded, the exception is thrown, indicating that the stream cannot handle the current data load.
This issue typically arises when there is a sudden spike in data traffic or when the stream is not provisioned with enough shards to handle the expected data rate. It can also occur if the application logic inadvertently sends too many records in a short period.
The most straightforward solution is to increase the number of shards in your stream. This can be done using the AWS Management Console, AWS CLI, or AWS SDKs. Here is a command using AWS CLI:
aws kinesis update-shard-count --stream-name YourStreamName --target-shard-count NewShardCount
Replace YourStreamName with your stream's name and NewShardCount with the desired number of shards.
Review your data ingestion strategy to ensure that data is being sent efficiently. Consider batching records together to reduce the number of requests or implementing a backoff strategy to handle spikes in traffic.
Use AWS CloudWatch to monitor your stream's metrics. Pay attention to metrics like IncomingBytes and IncomingRecords to understand your stream's usage patterns. Adjust the number of shards accordingly to prevent future occurrences of this exception.
For more detailed information, you can refer to the AWS Kinesis Documentation and the AWS Kinesis Data Streams page for best practices and optimization tips.
Let Dr. Droid create custom investigation plans for your infrastructure.
Book Demo
