MySQL Replication Lag

In a replication setup, a noticeable delay between master and slave, leading to outdated data on replicas.
  1. Identify the Replication Lag: Use the command `SHOW SLAVE STATUS\G;` to check the `SecondsBehindMaster` field. This tells you the current lag in seconds.



  1. Check Slave I/O and SQL Running State: In the output of `SHOW SLAVE STATUS\G;`, ensure that both `SlaveIORunning` and `SlaveSQLRunning` are set to `Yes`. If not, there's a problem with replication that needs to be addressed.



  1. Examine Slave Errors: In the same output, look for `LastIOErrno`, `LastIOError`, `LastSQLErrno`, and `LastSQLError` to identify any errors that have occurred during replication.



  1. Determine the Cause of Lag:


- High Write Load: If the master is under heavy write load, this could be causing the lag.
- Network Issues: Check the network bandwidth and health between the master and slave servers using tools like `ping`, `traceroute`, or `mtr`.
- Long-Running Queries: Identify any long-running queries on the slave that might be blocking the replication process using `SHOW PROCESSLIST;`.

  1. Skip Problematic Transactions: If a specific error is causing replication to stop (e.g., a duplicate key error) and it's safe to do so, you can skip the problematic event. Use the following commands carefully:


- Set `global sqlslaveskip_counter=1;`
- Start the slave with `START SLAVE;`.
- Check `SHOW SLAVE STATUS\G;` again to ensure replication has resumed and the lag is decreasing.

  1. Temporary Increase of Slave Workers (for Multi-threaded Replication in MySQL 5.6 and above): If your MySQL version supports multi-threaded replication and the hardware on the slave can handle more threads, increase the number of worker threads temporarily with:


- `STOP SLAVE;`
- `SET GLOBAL slaveparallelworkers = ;`
- `START SLAVE;`.

  1. Monitor Disk Space and I/O Utilization: Use tools like `iotop`, `df`, and `du` to check if disk I/O or space constraints are causing the lag. High I/O wait times or low available disk space might need attention.



  1. Ensure Binary Logs Are Enabled on the Slave (if not already): This allows for the potential use of delayed replication or other advanced recovery techniques.



  1. Address Specific Errors: Based on errors identified in step 3, take action specific to those errors. For example, if there's a network issue, you might need to work with your networking team to resolve it.



  1. Review and Optimize Queries: Use `SHOW FULL PROCESSLIST;` to identify slow-running queries on the master that could be optimized to reduce the load.



  1. Check and Adjust the `innodbflushlogattrx_commit` Parameter: If set to `1` (safe but slow), consider setting it to `2` or `0` for the duration of catching up the slave, but be aware of the durability implications.



  1. Review Replication Filters: If you have replication filters in place (`replicatedodb`, `replicateignoredb`, etc.), ensure they're correctly configured as they might be filtering out necessary data.



Take each step carefully and monitor the impact on the replication lag after each action.

Never debug

MySQL

manually again

Let Dr. Droid create custom investigation plans for your infrastructure.

Book Demo
Automate Debugging for
MySQL
See how Dr. Droid creates investigation plans for your infrastructure.
Book Demo

MORE ISSUES

MySQL Replication Lag
In a replication setup, a noticeable delay between master and slave, leading to outdated data on replicas.
MySQL Query Timeouts
Queries failing due to exceeding the configured timeout limits, often caused by complex or inefficient queries.
MySQL Frequent Crashes or Restarts
The MySQL server crashing unexpectedly or restarting often, possibly due to hardware issues, configuration problems, or resource exhaustion.
MySQL Frequent Deadlocks
Frequent deadlocks occurring during transactions, causing failed queries and rollback operations.
MySQL Full Disk/Storage Issues
Running out of disk space due to large logs, binary logs, or unoptimized data storage, leading to service disruptions.
MySQL InnoDB Buffer Pool Issues
Inefficient use of the InnoDB buffer pool, leading to high disk I/O because of frequent page swapping.
MySQL Table/Index Fragmentation
Fragmented tables or indexes leading to slower performance due to inefficient data access paths. This can often be mitigated by running OPTIMIZE TABLE.
MySQL Disk I/O Bottlenecks
Slow disk performance, often caused by poorly optimized queries, large temporary tables, or inadequate indexing.
MySQL High CPU/Memory Usage
The MySQL server consuming excessive CPU or memory resources, leading to system performance degradation.
MySQL Connection Issues
Hitting the maximum number of connections or connection timeouts due to poor connection pool management or high concurrency.
MySQL Lock Contention
Queries getting stuck due to high contention for row or table-level locks, causing delays or timeouts.
MySQL Slow Queries
Queries taking much longer than expected, especially under load or during complex operations like joins or large scans.
MySQL 1252: Partition does not exist.
MySQL 1251: Key not allowed in partition.
MySQL 1250: Partition key not allowed.
MySQL 1247: Cannot split partition.
MySQL 1249: Cannot update partition index.
MySQL 1248: Cannot update partition schema.
MySQL 1246: Cannot merge partition.
MySQL 1244: Cannot create partition.
MySQL 1245: Cannot drop partition.
MySQL 1243: Statement failed due to partition.
MySQL 1242: Cannot expand partition.
MySQL 1240: Partition is corrupted.
MySQL 1241: Cannot shrink partition.
MySQL 1239: Partition needs repair.
MySQL 1236: Cannot access partition.
MySQL 1238: Cannot repair partition.
MySQL 1237: Partition is full.
MySQL 1234: Cannot unlock partition.
MySQL 1235: Statement not allowed in partition.
MySQL 1232: Cannot delete partition.
MySQL 1231: Cannot update partition.
MySQL 1233: Cannot lock partition.
MySQL 1230: Duplicate partition key.
MySQL 1229: Cannot drop partition.
MySQL 1228: Cannot add partition.
MySQL 1226: Failed to drop partition.
MySQL 1227: Failed to create partition.
MySQL 1222: Too many partitions.
MySQL 1212: Permission denied for key.
MySQL 1225: Duplicate partition name.
MySQL 1224: Partition not empty.
MySQL 1223: Partition key was not found.
MySQL 1221: Cannot delete parent row.
MySQL 1220: Cannot update parent row.
MySQL 1219: Cannot update child row.
MySQL 1216: Cannot delete or update a parent row.
MySQL 1218: Cannot delete child row.
MySQL 1215: Cannot add foreign key constraint.
MySQL 1217: Cannot add child row.
MySQL 1214: Row was updated.
MySQL 1213: Deadlock found when trying to get lock.
MySQL 1208: Failed to drop key.
MySQL 1211: Too many open files.
MySQL 1210: Key already exists.
MySQL 1209: Key not found.
MySQL 1204: Cannot update entry.
MySQL 1206: Error in unique key constraint.
MySQL 1207: Duplicate key found.
MySQL 1205: Error in foreign key constraint.
MySQL 1203: Cannot insert entry.
MySQL 1202: Cannot delete entry.
MySQL 1199: Statement failed due to index.
MySQL 1200: Index is corrupted.
MySQL 1201: Duplicate key value.
MySQL 1198: Error closing file.
MySQL 1193: Statement failed due to table.
MySQL 1197: Failed to open file.
MySQL 1175: Statement not allowed in schema.
MySQL 1196: Too many connections.
MySQL 1194: Table needs repair.
MySQL 1195: Table is corrupted.
MySQL 1192: Statement failed due to view.
MySQL 1191: Statement failed due to procedure.
MySQL 1189: Statement failed due to event.
MySQL 1190: Statement failed due to function.
MySQL 1186: SAVEPOINT cannot be rolled back.
MySQL 1188: Statement failed due to trigger.
MySQL 1187: Statement failed due to SAVEPOINT.
MySQL 1185: SAVEPOINT does not exist.
MySQL 1184: Cannot use SAVEPOINT in transaction.
MySQL 1183: Command not allowed in transaction.
MySQL 1182: Connection was closed.
MySQL 1181: Duplicate transaction.
MySQL 1180: Read-only transaction violation.
MySQL 1179: Transaction is read-only.
MySQL 1176: Statement not allowed in session.
MySQL 1178: Transaction not active.
MySQL 1177: Session not open.
MySQL 1174: Statement not allowed in row.
MySQL 1173: Statement not allowed in table.
MySQL 1172: Statement not allowed in transaction.
MySQL 1170: Cannot open database.
MySQL 1171: Null column in table.
MySQL 1169: Duplicate key.
MySQL 1168: Failed to create file.
MySQL 1167: Out of disk space.
MySQL 1165: Error reading from file.
MySQL 1166: Error writing to file.

Backed by

Platform

AI OpsAlert Grouping & De-DuplicationPlayBooksKubernetes Bot

Resources

DocumentationFun For DevsBlog

Contact

Contact UsAbout UsCareersTerms and ConditionsPrivacy PolicyShipping & and Delivery PolicyCancellation & Refund Policy

Connect

Slack CommunityGithubLinkedInX (Twitter)
Made with ❤️ in Bangalore & San Francisco 🏢

Doctor Droid