VMs / EC2 High Connection Refused Rate

The server is refusing a high number of incoming connections.

Understanding Prometheus and Its Purpose

Prometheus is an open-source systems monitoring and alerting toolkit originally built at SoundCloud. It is designed to record real-time metrics in a time-series database, built using a highly dimensional data model. Prometheus is particularly useful for monitoring cloud-based environments like AWS EC2 instances, where it can track various performance metrics and trigger alerts when certain thresholds are breached.

Symptom: High Connection Refused Rate

One of the alerts you might encounter while using Prometheus to monitor your EC2 instances is the "High Connection Refused Rate" alert. This alert indicates that your server is refusing a significant number of incoming connections, which could lead to degraded performance or service outages.

Details About the High Connection Refused Rate Alert

The "High Connection Refused Rate" alert is triggered when the rate of refused connections exceeds a predefined threshold. This can occur due to several reasons, such as insufficient server resources, misconfigured network settings, or an overwhelming number of incoming requests. When this alert is triggered, it is crucial to investigate and resolve the underlying issue promptly to maintain service availability.

Potential Causes of High Connection Refusals

Server resource limitations, such as CPU or memory constraints.
Network misconfigurations, such as incorrect firewall settings.
Application-level issues, such as improper handling of incoming requests.

Steps to Fix the High Connection Refused Rate Alert

To resolve the "High Connection Refused Rate" alert, follow these steps:

1. Check Server Capacity

First, ensure that your server has adequate resources to handle incoming connections. You can use AWS CloudWatch to monitor CPU and memory usage. If resources are maxed out, consider scaling up your instance type or adding more instances to a load balancer.

aws cloudwatch get-metric-data --metric-name CPUUtilization --namespace AWS/EC2 --statistics Average

2. Review Network Configurations

Verify that your network settings are correctly configured. Check your security groups and network access control lists (ACLs) to ensure they allow the necessary traffic. For more information, refer to the AWS Security Groups Documentation.

3. Analyze Application Logs

Examine your application logs to identify any issues with how incoming connections are handled. Look for error messages or patterns that might indicate why connections are being refused. Adjust your application settings or code as needed to better manage incoming requests.

4. Implement Rate Limiting

If your server is overwhelmed by too many requests, consider implementing rate limiting to control the flow of incoming connections. This can help prevent your server from being overloaded and refusing connections. Tools like NGINX rate limiting can be useful for this purpose.

Conclusion

By following these steps, you can effectively diagnose and resolve the "High Connection Refused Rate" alert in your EC2 environment. Regular monitoring and proactive management of server resources and configurations are key to maintaining optimal performance and availability.

Master

VMs / EC2 High Connection Refused Rate

debugging in Minutes

— Grab the Ultimate Cheatsheet

(Perfect for DevOps & SREs)

Most-used commands

Real-world configs/examples

Handy troubleshooting shortcuts

Thankyou for your submission

We have sent the cheatsheet on your email!

Oops! Something went wrong while submitting the form.

VMs / EC2 High Connection Refused Rate

Cheatsheet

(Perfect for DevOps & SREs)

Most-used commands

Thankyou for your submission

We have sent the cheatsheet on your email!

Oops! Something went wrong while submitting the form.

MORE ISSUES

supabase Unauthorized API Access

Detected unauthorized access attempts to the API, indicating potential security threats.

supabase Replication Lag

Significant delay in database replication, which may affect data consistency.

supabase Pod Eviction

Pods are being evicted due to resource constraints or node failures.

supabase Service Dependency Failure

A dependent service is failing, affecting the functionality of the primary service.

supabase Service Latency Spike

Sudden increase in service latency, potentially affecting user experience.

supabase Database Deadlock

Detected deadlocks in the database, which may affect transaction processing.

supabase High I/O Wait

Excessive I/O wait times, indicating potential disk or network bottlenecks.

supabase Node Memory Pressure

A node is under memory pressure, affecting pod scheduling and performance.

supabase High Swap Usage

Excessive swap usage, which may degrade system performance.

supabase Service Restart Loop

A service is continuously restarting, indicating potential configuration or resource issues.

supabase API Rate Limit Exceeded

API requests have exceeded the allowed rate limit, potentially affecting service availability.

supabase Configuration Drift

Detected changes in system configuration that deviate from the desired state.

supabase Node Disk Pressure

A node is experiencing disk pressure, which may affect pod scheduling and performance.

supabase Job Failure

Scheduled jobs or tasks have failed to execute successfully.

supabase Service Unavailable

A service is temporarily unavailable, possibly due to overload or misconfiguration.

supabase High Load Average

The system load average is higher than expected, indicating potential resource saturation.

supabase High Network Traffic

Unusually high network traffic, which may indicate a DDoS attack or misconfigured services.

supabase Backup Failure

Scheduled backups have failed, risking data loss in case of system failures.

supabase High Latency

Increased response times for requests, which may impact user experience.

supabase Unauthorized Access Attempts

Multiple failed login attempts detected, indicating potential security threats.

supabase Pod CrashLoopBackOff

A pod is repeatedly crashing and restarting, indicating issues with the application or configuration.

supabase Certificate Expiry

SSL/TLS certificates are nearing expiration, risking secure communication failures.

supabase Node Not Ready

A node in the cluster is not ready, potentially due to resource constraints or failures.

supabase Service Down

A critical service is not responding, possibly due to crashes or network issues.

supabase High Memory Usage

The memory consumption has surpassed the set limit, which may lead to performance degradation.

supabase Disk Space Low

The available disk space is below the acceptable threshold, risking data write failures.

supabase High CPU Usage

The CPU usage has exceeded the defined threshold, indicating potential over-utilization of server resources.

supabase Database Connection Errors

Frequent connection errors to the database, possibly due to network issues or misconfigurations.

supabase High Error Rate

An increased rate of errors in the application, indicating potential bugs or misconfigurations.

supabase Slow Query Response

Queries are taking longer than expected to execute, affecting application performance.

OpenSearch Index Recovery Failure

An index recovery operation has failed, potentially due to resource constraints or configuration issues.

OpenSearch Cluster Node Joined

A new node has joined the cluster, potentially affecting cluster balance.

OpenSearch Index Read-Only Mode

An index has been set to read-only mode due to disk space issues.

OpenSearch Node Heap Dump Generated

A heap dump has been generated, indicating potential memory issues.

OpenSearch Cluster Node Left

A node has unexpectedly left the cluster.

OpenSearch Node Network Latency High

Network latency between nodes is higher than expected, impacting cluster performance.

OpenSearch Cluster Node Disk Full

A node's disk is full, preventing further data operations.

OpenSearch Cluster State Update Failure

The cluster is unable to update its state due to resource constraints or configuration issues.

OpenSearch Node Disk I/O High

Disk I/O operations on a node are consistently high, impacting performance.

OpenSearch Indexing Throughput Low

The rate of indexing operations is lower than expected.

OpenSearch Search Throughput Low

The rate of search operations is lower than expected.

OpenSearch Node JVM Heap Pressure High

The JVM heap pressure on a node is consistently high, indicating potential memory issues.

OpenSearch Cluster Node Count Low

The number of nodes in the cluster is below the expected count.

OpenSearch Snapshot Failure

A snapshot operation has failed, potentially due to storage issues or configuration errors.

OpenSearch Node Disk Watermark Exceeded

Disk usage on a node has exceeded the high watermark threshold.

OpenSearch Index Shard Size Large

One or more index shards have grown larger than the recommended size.

OpenSearch Snapshot Duration High

Snapshot operations are taking longer than expected to complete.

OpenSearch Cluster Rebalance Failure

The cluster is unable to rebalance shards due to resource constraints or configuration issues.

OpenSearch Pending Tasks High

There is a high number of pending tasks in the cluster, indicating potential bottlenecks.

OpenSearch Cluster Shard Allocation Failure

The cluster is unable to allocate shards due to resource constraints or configuration issues.

OpenSearch Search Latency High

Search queries are taking longer than expected to complete.

OpenSearch Indexing Latency High

Indexing operations are taking longer than expected.

OpenSearch Frequent Garbage Collection

Frequent garbage collection events are occurring, impacting performance.

OpenSearch High JVM Heap Usage

The JVM heap usage is consistently high, leading to potential garbage collection issues.

OpenSearch Node Disk Usage High

The disk usage on one or more OpenSearch nodes is above the threshold.

OpenSearch Node Not Reachable

An OpenSearch node is not reachable or has been removed from the cluster.

OpenSearch Cluster Status Red

One or more primary shards are unassigned in the OpenSearch cluster.

OpenSearch Cluster Status Yellow

One or more replica shards are unassigned in the OpenSearch cluster.

OpenSearch High Memory Usage

The memory usage on the OpenSearch nodes is consistently above the threshold.

OpenSearch High CPU Usage

The CPU usage on the OpenSearch nodes is consistently above the threshold.

ClickHouse ClickHouseHighZooKeeperEphemeralNodeCount

The number of ephemeral nodes in ZooKeeper is too high, which can affect stability.

ClickHouse ClickHouseHighZooKeeperWatchCount

The number of watches in ZooKeeper is too high, potentially affecting performance.

ClickHouse ClickHouseHighZooKeeperNodeCount

The number of nodes in ZooKeeper is too high, which can affect performance.

ClickHouse ClickHouseHighZooKeeperSessionCount

The number of ZooKeeper sessions is too high, potentially overloading the ZooKeeper cluster.

ClickHouse ClickHouseHighZooKeeperRequestErrors

A high number of errors are occurring in requests to ZooKeeper, disrupting coordination.

ClickHouse ClickHouseHighZooKeeperRequestLatency

Requests to ZooKeeper are experiencing high latency, affecting distributed operations.

ClickHouse ClickHouseHighBackgroundTaskQueueSize

The background task queue is too large, potentially delaying important maintenance tasks.

ClickHouse ClickHouseHighMutationQueueSize

The mutation queue size is too large, which can delay data updates.

ClickHouse ClickHouseHighCompactionQueueSize

The compaction queue size is too large, indicating delays in data compaction.

ClickHouse ClickHouseHighPartCountInPartition

A partition has too many parts, which can degrade query performance.

ClickHouse ClickHouseHighReplicaQueueSize

The size of the replication queue is too large, which can delay data synchronization.

ClickHouse ClickHouseHighNetworkErrors

A high number of network errors are occurring, which can disrupt data operations.

ClickHouse ClickHouseHighDiskIOWait

Disk I/O wait times are high, indicating potential bottlenecks in disk operations.

ClickHouse ClickHouseInsertFailureRateHigh

A high rate of insert failures is occurring, which can affect data ingestion.

ClickHouse ClickHouseHighNetworkLatency

Network latency is high, affecting communication between ClickHouse nodes or clients.

ClickHouse ClickHouseQueryFailureRateHigh

A high rate of query failures is occurring, indicating potential issues with queries or server stability.

ClickHouse ClickHouseHighReplicaLag

The lag between replicas and the primary server is too high, risking data consistency.

ClickHouse ClickHouseBackgroundMergesFailing

Background merge operations are failing, which can lead to performance issues.

ClickHouse ClickHouseMergeTreePartCountHigh

The number of parts in a MergeTree table is too high, which can degrade performance.

ClickHouse ClickHouseTableNotReplicated

A table that should be replicated is not being replicated correctly.

ClickHouse ClickHouseZooKeeperSessionExpired

The session with ZooKeeper has expired, potentially disrupting distributed operations.

ClickHouse ClickHouseHighWriteLatency

Write operations are experiencing high latency, which can delay data ingestion.

ClickHouse ClickHouseHighReadLatency

Read operations are experiencing high latency, affecting query performance.

ClickHouse ClickHouseReplicaDown

One or more replicas are not reachable, which can affect data redundancy and availability.

ClickHouse ClickHouseZooKeeperConnectionLoss

The ClickHouse server has lost connection to ZooKeeper, affecting distributed coordination.

ClickHouse ClickHouseHighMemoryUsage

The ClickHouse server is using an unusually high amount of memory, which could lead to performance degradation or crashes.

ClickHouse ClickHouseHighCPUUsage

The CPU usage on the ClickHouse server is consistently high, indicating potential performance issues.

ClickHouse ClickHouseTooManyConnections

The number of connections to the ClickHouse server has exceeded the configured limit.

ClickHouse ClickHouseQueryTimeout

Queries are taking too long to execute and are timing out.

ClickHouse ClickHouseDiskSpaceLow

The disk space on the ClickHouse server is running low, which could prevent new data from being written.

ClickHouse ClickHouseReplicaLag

One or more replicas are lagging behind the primary server, which can lead to stale reads.

Cassandra CassandraClusterWideLatencyHigh

High latency observed across the entire cluster, indicating potential systemic issues.

Cassandra CassandraRepairFailures

Failures occurred during repair operations, potentially affecting data consistency.

Cassandra CassandraNodeLoadImbalance

Uneven data distribution across nodes, leading to load imbalance.

Cassandra CassandraTableCompactionHigh

Compaction tasks for a specific table are taking longer than expected.

Cassandra CassandraHintsDeliveryLatencyHigh

Hint delivery is taking longer than expected, indicating potential network or node issues.

Cassandra CassandraBatchLogReplay

Batch log replay is occurring, indicating potential issues with batch operations.

Cassandra CassandraCQLRequestsHigh

A high number of CQL requests are being processed, potentially overloading the node.

Cassandra CassandraThriftRequestsHigh

A high number of Thrift requests are being processed, potentially overloading the node.

Cassandra CassandraReadRepairFailures

Failures occurred during read repair operations.

Backed by

Resources

Contact

Platform

Connect

Deep Sea Tech Inc. — Made with ❤️ in & 🏢

Doctor Droid