Between 07:40 UTC and 10:31 UTC, we experienced issues affecting user syncing in Atassian Administration. This affected Confluence, Jira Work Management, Jira Service Management, Jira, Trello, and Guard. The issue has been resolved and the service is operating normally.

Between 2025-06-04 14:11 UTC to 20:18 UTC, we experienced delays in delivering emails for Confluence, Jira Work Management, Jira Service Management, Jira, Trello, Atlassian Bitbucket, Guard, Jira Align, Jira Product Discovery, Atlas, Compass. The issue has been resolved and the service is operating normally.

On June 2nd, Trello's search bar functionality was not working correctly. The issue has now been resolved, and the service operates normally for all affected customers.

### **SUMMARY** On May 15, 2025, between 13:55 and 14:18 UTC, Atlassian customers using the Trello product experienced errors or slow loading times when attempting to view their cards and boards. The event was triggered by a database plan cache expiring and high resource usage caused by subsequent database query planning operations. The particular database shard that was impacted held data that was required for every card load. The incident was detected within two minutes by the automated monitoring system and mitigated by increasing resources available to the affected database shard, which put Atlassian systems into a known good state. The total time to resolution was about 23 minutes. ### **IMPACT** The overall impact was between May 15, 2025, 13:55 and May 15, 2025, 14:18 UTC on the Trello product. The incident caused service disruption for all Trello customers. ### **ROOT CAUSE** The issue was caused by a query plan expiring from the database cache, which caused incoming queries to go through a replanning operation. These queries had multiple plans that could satisfy them, and depending on the size of the query, one plan might be significantly more efficient than another. This caused the query planner to perform a great many more replanning operations than usual, which consumed all of the CPU on the server for a brief moment. Once the CPU was consumed, the planning operations themselves began taking too long and therefore required constant replanning in an effort to find more efficient options. This negative feedback loop could not be broken without intervention. ### **REMEDIAL ACTIONS PLAN & NEXT STEPS** We know that outages impact your productivity. While we have a number of testing and preventative processes in place, this specific issue wasn’t identified because it would only occur under very distinct conditions, including the amount of load and the order of database queries. We are prioritizing the following improvement actions to avoid repeating this type of incident: * Review our capacity planning thresholds and ensure that all shards have sufficient overhead to handle unexpected load. * Improve query planner performance by: * Implement hinting for known problematic query shapes to circumvent the query planner. * Investigate long-term generalized solutions to prevent query planner thrashing. Furthermore, we are prioritizing the following additional measures to reduce the impact of any future incidents: * Analyze and reduce single points of failure for loading Trello boards and cards. We apologize to customers whose services were impacted during this incident; we are taking immediate steps to improve the platform’s performance and availability. Thanks, Atlassian Customer Support

### **SUMMARY** On May 5, 2025, between 2:08 p.m. and 4:29 p.m. UTC, some Atlassian customers using Trello were unable to view their boards or cards. The event was triggered by an unexpected error encountered by our infrastructure management tools, which resulted in an incorrect DNS configuration being deployed to a portion of our database. The incident was detected within four minutes by automated monitoring systems and mitigated by identifying the faulty portion of the database and performing a failover, which put Atlassian systems into a known good state. The total time to resolution was about two hours and 21 minutes. ### **IMPACT** The overall impact was on the Trello product on May 5, 2025, between 2:08 p.m. and 4:29 p.m. UTC. The incident caused service disruption to Trello customers whose accounts and boards contained or referenced data on the affected shard of our database. Additionally, some Trello customers would have experienced a service disruption due to our use of load-shedding tools during the incident to strategically block portions of our traffic to aid in recovery. ### **ROOT CAUSE** The day before the incident, on May 4, our infrastructure management tooling encountered an unexpected error when attempting to fetch the networking metadata on a particular host. This led to the host, which was a member of our database cluster, to incorrectly apply the default Operating System DNS configuration. This DNS configuration was not able to resolve internal domains, which led to a partial failure state of the node. The database continued to function normally and there was no immediate customer impact but in the background this incorrect DNS configuration led to the slow buildup of database sessions. These database sessions are usually short-lived and automatically expire when no longer needed, but the DNS misconfiguration prevented this automatic expiration. The database sessions eventually grew to the default maximum on this particular shard. At that point, the shard was unable to generate new sessions, which are required for all basic operations, and the Trello product began experiencing elevated error rates. ### **REMEDIAL ACTIONS PLAN & NEXT STEPS** We know that outages impact your productivity. While we have a number of testing and preventative processes in place, this specific issue wasn’t identified due to the isolated nature of the database session resource and monitoring gaps around this resource and around DNS resolution. We are prioritizing the following improvement actions designed to avoid repeating this type of incident: * Update our infrastructure management tool to use a safe fall-back DNS configuration in the case of unexpected errors. * Expand existing DNS monitoring to include the resolution of internal domains. * Expand existing database session count monitoring to include all database node types. Furthermore, we are prioritizing the following additional measures to reduce the duration of any future incidents: * Evaluate our incident response process to identify actions that can be streamlined for quicker resolution. We apologize to customers whose services were impacted during this incident; we are taking steps designed to improve the platform’s performance and availability. Thanks, Atlassian Customer Support