Prometheus is an open-source systems monitoring and alerting toolkit originally built at SoundCloud. It is now a standalone open source project and maintained independently of any company. Prometheus collects and stores its metrics as time series data, i.e., metrics information is stored with the timestamp at which it was recorded, alongside optional key-value pairs called labels.
Prometheus is designed to be reliable, even when other parts of your infrastructure are broken; it is the go-to system for monitoring and alerting in cloud-native environments.
One common issue users encounter with Prometheus is that it appears not to be storing data. This can manifest as missing metrics in the Prometheus UI or gaps in the data when querying.
When Prometheus is not storing data, you might notice that the graphs in the Prometheus dashboard are empty or that alerts are not being triggered as expected.
The root cause of Prometheus not storing data often lies in storage issues or misconfigured retention settings. Prometheus uses a local on-disk time series database, and if there are issues with disk space or configuration, it can lead to data not being stored.
If the disk is full or there are permissions issues, Prometheus will not be able to write data to its storage. This can happen if the storage volume is not properly mounted or if there are restrictions on the file system.
Prometheus has configurable retention settings that determine how long data is kept. If these settings are misconfigured, data may be deleted prematurely, leading to gaps in the stored data.
First, ensure that the storage volume where Prometheus is writing data has enough space. You can check the disk usage with the following command:
df -h /path/to/prometheus/storage
If the disk is full, consider increasing the storage capacity or cleaning up old data.
Check the retention settings in your Prometheus configuration file (usually prometheus.yml). Look for the --storage.tsdb.retention.time flag, which specifies how long to retain data. Ensure it is set to a reasonable value for your use case:
--storage.tsdb.retention.time=15d
Adjust the retention time as needed and restart Prometheus to apply the changes.
Examine the Prometheus logs for any errors or warnings related to storage. Logs can provide insights into what might be going wrong. Use the following command to view logs:
tail -f /var/log/prometheus/prometheus.log
Look for any messages indicating storage issues or configuration errors.
For more detailed information on configuring Prometheus storage and retention, refer to the official Prometheus Storage Documentation.
If you continue to experience issues, consider reaching out to the Prometheus Users Group for community support.
Let Dr. Droid create custom investigation plans for your infrastructure.
Start Free POC (15-min setup) →
