VictoriaMetrics is a fast, cost-effective, and scalable time-series database designed to handle large volumes of data. It is commonly used for monitoring systems, IoT data, and other applications requiring high-performance data ingestion and querying capabilities. VictoriaMetrics supports various data ingestion protocols, including Prometheus remote write, InfluxDB line protocol, and Graphite plaintext protocol.
Data ingestion throttling in VictoriaMetrics is a situation where the system limits the rate of data ingestion. This can manifest as delayed data writes, increased latency, or errors indicating that the ingestion rate is too high. Users may notice that their data is not being ingested as quickly as expected, or they may encounter specific error messages related to throttling.
Throttling typically occurs when the data ingestion rate exceeds the configured limits in VictoriaMetrics. This can happen if the incoming data volume is too high for the current configuration, leading to a bottleneck. The parameter -influx.maxConcurrentInserts is often a key factor in determining the maximum concurrent insert operations allowed, and if this is set too low, it can lead to throttling.
VictoriaMetrics has several configuration options that control the ingestion rate. These include settings for concurrent inserts, memory usage, and other resource limits. If these settings are not properly tuned to match the data volume, throttling can occur.
In addition to configuration limits, system resource constraints such as CPU, memory, and disk I/O can also contribute to throttling. If the system resources are insufficient to handle the data load, it can lead to performance degradation and throttling.
To address data ingestion throttling in VictoriaMetrics, follow these steps:
Begin by monitoring the current data ingestion rates and system performance metrics. Use VictoriaMetrics' built-in monitoring tools or external monitoring solutions to gather insights into the system's behavior. Check for any error messages or logs that indicate throttling issues.
Review and adjust the configuration settings related to data ingestion. Specifically, consider increasing the -influx.maxConcurrentInserts parameter to allow more concurrent insert operations. This can be done by modifying the VictoriaMetrics startup command:
./victoria-metrics -influx.maxConcurrentInserts=1000
Ensure that the new value aligns with the system's capabilities and the expected data volume.
Evaluate the system resources available to VictoriaMetrics. Ensure that there is adequate CPU, memory, and disk I/O to handle the data load. Consider scaling up the hardware or optimizing resource allocation to improve performance.
If increasing the ingestion limits and optimizing resources do not resolve the issue, consider reducing the data volume being ingested. This can be achieved by filtering unnecessary data, aggregating data before ingestion, or adjusting the data collection frequency.
For more information on configuring VictoriaMetrics and troubleshooting common issues, refer to the following resources:
By following these steps and utilizing the available resources, you can effectively manage and resolve data ingestion throttling in VictoriaMetrics.
Let Dr. Droid create custom investigation plans for your infrastructure.
Start Free POC (15-min setup) →
