MySQL Backups: From strategy to Implementation Mike Frank Senior Product Manager 1
Program Agenda Introduction The 5 Key Steps Advanced Options References 2
Backups are a DBAs Top Priority Be Prepared Automate Test and Practice Be Ready to recover when a data disaster strikes 3
Backups As a DBA you must have a backup strategy Data loss will happen OS crashes Power Failures Data center outages Are you in a cloud? File system crashes Human error Application Error Just a reminder of course - you know all that. 4
MySQL Enterprise Backup Performs Hot Backup Highly efficient and Consistent Performs the backup while the database is running Necessary when The database can t be taken offline Those are Cold Backups The database is large backups can take a long time Export backups are too slow, lock tables, Also lets you perform Warm Backups for non-innodb tables. Table level locks 5
What we ll cover today There s more than we can cover this session The goal today is to get you running scheduled backups regularly. The first key step to protecting your data Later you can explore more powerful work flows and features Partial data moment Compression Task synchronization Hot and Cold support http://www.mysql.com/products/enterprise/backup.html 6
The 5 Easy Steps 7
The 5 Steps 1. Download and Install 2. Setup Backup Directories 3. Creating Full and Incremental Backups 4. Scheduling 5. Recovery Prepare and Restore 8
Step 1 Download and Install 9
Download https://edelivery.oracle.com Login Pick MySQL Database and then OS platform Hit Go 10
Find the download package Your looking for MySQL Enterprise Backup Half way down the page select Specific OS/Version and Format - either TAR or RPM Note for Windows - MySQL Installer for Windows includes MySQL Enterprise Backup 11
Install Uncompress the downloaded file - then ZIPs and Tars Basically you can unzip, untar MEB where ever you want RPM sudo rpm -i package_name.rpm Deb packages sudo dpkg i package_name.rpm Windows Run the MSI installs the product in C:\Program Files\MySQL\MySQL Enterprise Backup 3.8 12
Linux from tar 13
Windows After running MSI installer 14
Once installed set path or note location Default Windows from MSI C:\Program Files\MySQL\MySQL Enterprise Backup 3.8\mysqlbackup.exe Used in 5 step example Default Linux from RPM / DEB /opt/mysql/meb-3.8 If tar or zip will be where you unpacked the files For linux 5 step example used /home/mfrank/meb-3.8.1-linux2.6-x86-64bit/ 15
Step 2 Setup Backup Directories 16
Where do you want the backups to go Make sure you have plenty of space Monitor that space Linux mkdir /MySQLBackupHome chown mysql R /MySQLBackupHome chgrp mysql R /MySQLBackupHome chmod 600 R /MySQLBackupHome Windows mkdir C:\mydatabasebackups\ Make sure only the user that has rights to mysql file can read and write to the directory Right Click properties and Edit permissions for only that user (mysql) Permissions - Modify, Read & Execute, List Folders, Read, Write 17
Step 3 Backup 18
Backup Types Full all the database data Note but not the mysql programs and control files Incremental only the database changes Prerequisite a full backup Can be more than one incremental backups The changes between last full or a prior incremental Partial just part of the database Binary Logs contains each event/transaction http://dev.mysql.com/doc/mysql-enterprise-backup/3.7/en/mysqlbackup.tasks.html http://dev.mysql.com/doc/mysql-enterprise-backup/3.8/en/meb-backup-overview.html http://dev.mysql.com/doc/refman/5.6/en/binary-log.html 19
Keeping it simple You ll want to run backups on a schedule For example here we will show 1 full once week An incremental daily 20
Commands and Scripts You need to have scripts that you schedule In this case a minimum of 2 FULL INCREMENTAL 21
Create Backup User For this example the backup user is mysqlbackup mysql -u root p Mysql> CREATE USER 'mysqlbackup'@'localhost' IDENTIFIED BY '<password>'; mysql> GRANT RELOAD ON *.* TO 'mysqlbackup'@'localhost'; mysql> GRANT RELOAD ON *.* TO 'mysqlbackup'@'localhost'; mysql> GRANT CREATE, INSERT, DROP ON mysql.ibbackup_binlog_marker TO 'mysqlbackup'@'localhost'; mysql> GRANT CREATE, INSERT, DROP ON mysql.backup_progress TO 'mysqlbackup'@'localhost'; mysql> GRANT CREATE, INSERT, DROP ON mysql.backup_history TO 'mysqlbackup'@'localhost'; mysql> GRANT REPLICATION CLIENT ON *.* TO 'mysqlbackup'@'localhost'; mysql> GRANT SUPER ON *.* TO 'mysqlbackup'@'localhost'; mysql> GRANT CREATE TEMPORARY TABLES ON mysql.* TO 'mysqlbackup'@'localhost'; mysql> FLUSH PRIVILEGES; http://dev.mysql.com/doc/mysql-enterprise-backup/3.8/en/mysqlbackup.privileges.html 22
Step 3 a Full Backup 23
FULL Script will Create a time stamped subdirectory in the home backup area Create the backup command Run the backup command 24
Linux FULL Bash Script #!/bin/bash dtstring=`date +%F` new_dir="/mysqlbackuphome/mysqlfullweeklybackup_"$dtstring mkdir $new_dir cd /MySQLBackupHome /home/mfrank/meb-3.8.1-linux2.6-x86-64bit/bin/mysqlbackup --defaults-file=mysqlback.cnf --backup-dir=$new_dir/backup backup &> $new_dir/backup.log echo $? 25
Linux Configuration File - mysqlback.cnf [client] port = 3306 protocol = TCP user = <your backup user> password = <your backup user password> [mysqld] datadir = /var/lib/mysql [mysqlbackup] datadir = /var/lib/mysql innodb_data_home_dir = incremental_base = history:last_backup comments = My main backup - Full once a week, Incrementals daily Note: MEB supports MySQL External Authentication 26
Windows FULL VBScript Set shell = WScript.CreateObject("WScript.Shell") Dim fso Set fso = CreateObject("Scripting.FileSystemObject") new_dir = "C:\MySQLBackupHome\mysqlfullweeklybackup_" & replace(formatdatetime(now,2),"/","") If Not fso.folderexists(new_dir) Then Set f = fso.createfolder( new_dir ) End If backup_command = "cmd.exe /c ""C:\Program Files\MySQL\MySQL Enterprise Backup 3.8\mysqlbackup.exe"" --defaults-file=mysqlback.cnf --backup-dir=" & new_dir & "\backup backup > " & new_dir & "\backup.log 2>&1" Set oexec = shell.exec(backup_command) Do While oexec.status = 0 Loop WScript.Sleep 100 27
Windows Configuration File c:\mysqlbackuphome\mysqlback.cnf [client] port = 3306 socket = MySQL user = <your backup user> password = <your backup user password> [mysqld] datadir = C:\ProgramData\MySQL\MySQL Server 5.6\data\ [mysqlbackup] datadir = C:\ProgramData\MySQL\MySQL Server 5.6\data\ innodb_data_home_dir = Note: MEB supports MySQL External Authentication (Native Windows User) 28
Step 3 a Incremental Backup 29
Linux Incremental Bash Script #!/bin/bash dtstring=`date +%F` new_dir="/mysqlbackuphome/mysqlincrementaldailybackup_"$dtstring rm -Rf $new_dir mkdir $new_dir cd /MySQLBackupHome /home/mfrank/meb-3.8.1-linux2.6-x86-64bit/bin/mysqlbackup --defaults-file=mysqlback.cnf --incremental --backup-dir=$new_dir/backup --incremental-backupdir=$new_dir/backup/inc backup &> $new_dir/backup.log echo $? 30
Windows Incremental VBScript Set shell = WScript.CreateObject("WScript.Shell") Dim fso Set fso = CreateObject("Scripting.FileSystemObject") new_dir = "C:\MySQLBackupHome\mysqlincrementaldailybackup_" & replace(formatdatetime(now,2),"/","") If Not fso.folderexists(new_dir) Then Set f = fso.createfolder( new_dir ) End If Dim backup_command backup_command = "cmd.exe /c ""C:\Program Files\MySQL\MySQL Enterprise Backup 3.8\mysqlbackup.exe"" --defaults-file=mysqlback.cnf --incremental --backup-dir=" & new_dir & "\backup --incremental-backup-dir=" & new_dir & "\backup\inc backup > " & new_dir & "\backup.log 2>&1 Set oexec = shell.exec(backup_command) Do While oexec.status = 0 Loop WScript.Sleep 100 31
Step 4 Scheduling 32
Step 4 a Linux Scheduling 33
Linux CRON Weekly Full Sun at 2:00 am crontab e 0 2 * * 0 /MySQLBackupHome/MyFullBackup.sh Daily Incremental Mon-Sat at 2:00 am crontab e 0 2 * * 1-6 /MySQLBackupHome/MyIncrementalBackup.sh 34
Step 4 b Windows Scheduling 35
Windows Scheduler Create a Basic Task 36
Weekly for Full Daily for Incremental (6 days not including the day the full runs) 37
Select Time, Specify Start a Program 38
Path to your full or incremental script 39
When scripts run MEB details to this log 40
Step 5 Recovery 41
Recovery - Preparing Determine time to recover to If the time is for an incremental Applying an Incremental Backup changes reflected in incremental backup files bring the backup up-to-date Apply those changes for each - oldest to newest For example mysqlbackup --backup-dir=/mysqlbackup/mysqlfullweeklyfullbackup_2013-04-22/backup --incremental-backup-dir =/MySQLBackup/mysqlfulldailyincrementalbackup_2013-04- 23/backup/inc apply-incremental-backup 42
Recovery Restore the data Shutdown the database to be restored Stop the service or run a mysqladmin shutdown Run Copy-Back Copies the datafiles, logs, and other backed up files into place For example mysqlbackup --defaults-file=/etc/my.cnf --backup-dir =/MySQLBackup/mysqlfullweeklyfullbackup_2013-04-22/backup copy-back http://dev.mysql.com/doc/mysql-enterprise-backup/3.8/en/restoring.html 43
Advanced - Backup Getting finer grained for Point in Time Schedule the mysqlbinlog utility to dump all the SQL activity that happened after the binlog position of the last backup Direct the output of the mysqlbinlog command to an output file $ mysqlbinlog --start-position=27183537 /sqldata/binlog.000005 >partial_binlog $ mysqlbinlog /sqldata/binlog.000006 >> partial_binlog $ mysqlbinlog /sqldata/binlog.000007 >> partial_binlog... $ mysqlbinlog /sqldata/binlog.00000n >> partial_binlog 44
Advanced - Recovery Getting finer grained for an exact Point in Time To recover the database to the latest possible time, Pipe the output from mysqlbinlog --start-position=... Directly to mysql This replays all the SQL statements after the last backup. Or to get to a point in time say be before an oops happened Edit the bin log and remove the oops and all statements after it. Then $ mysql < partial_binlog http://dev.mysql.com/doc/refman/5.6/en/point-in-time-recovery.html 45
Advanced Single File Recommended Has been seen to be faster than directory backup Gives you an additional facility to verify the backup Does take a little longer to restore BUT - Can be prepared for restore Extract Apply log Compress back again http://dev.mysql.com/doc/mysql-enterprise-backup/3.8/en/backup-single-file-options.html 46
Advanced Backup Direct to Tape Meb Is a complete tape solution - Direct to Tape using SBT SBT is Oracle s API for backing up and restoring data via media management products use SBT for integration with Oracle Secure Backup use SBT for integration with Symantec Net Backup use SBT for integration with Tivoli Storage Manager Other products that support SBT 47
Advanced Using SBT 48
Advanced - Compression The compression value needs to be determined 1 is normally a good value Unless you have an idle CPU And a very strict disk limitation Compression will save disk but eat up more CPU and will take longer The restore will also be longer 49
Advanced Internal Table backup_history table MySQL Enterprise backup writes status to internal backup table Useful to see detailed history of backups Provide an audit trail to managers and auditors 50
Finally This is just the most basic of scripts You also need to decide on Retention What to save and for how long Schedule and automate this process Its often nice to keep recent backups close for fast recovery Archiving Whether to go to tape MEB also supports direct to tape backups Move files to archive storage tape, HDFS, Cloud - S3. 51
Resources Product Page http://www.mysql.com/products/enterprise/backup.html Latest Release https://blogs.oracle.com/mysqlenterprisebackup/entry/mysql_enterprise_backup_3_8 User Guide http://dev.mysql.com/doc/mysql-enterprise-backup/3.8/en/ Team Blog https://blogs.oracle.com/mysqlenterprisebackup/ Questions, Ideas, http://www.mysql.com/about/contact/ Phone: USA=+1-866-221-0634; Outside USA = +1-208-338-8100 Email: mike.frank@oracle.com 52