InnoDB: A journey to the core II. Jeremy Cole and Davi Arnaut

Transcription

1 InnoDB: A journey to the core II Jeremy Cole and Davi Arnaut

2 bit.ly/innodb_journey_ii

3 Jeremy Making MySQL Awesome at Google Worked at MySQL Contributor since years in the MySQL community Code, documentation, research, bug reports Yahoo, Proven Scaling, Gazillion, Twitter

4 Davi MySQL Internals development at LinkedIn Worked at MySQL Designed and built Twitter MySQL Long time Open Source contributor: Apache, Linux kernel, etc.

5 About this work... blog.jcole.us/innodb Not intended to be comprehensive Not authoritative (it is based on research) One of the best sources of documentation for InnoDB formats Approach: 1. Read the C and C++ sources 2. Implement in Ruby 3. Refactor and correct until reasonable 4. Document

6 Resources github.com/jeremycole/innodb_ruby groups.google.com/forum/#forum/innodb_ruby blog.jcole.us/innodb/

7 Overview of InnoDB Storage

8 High-level Overview Caching Buffer Pool Page Cache Adaptive Hash Indexes Buffer Pool LRU Buffer Pool Flush List Data Dictionary Cache Additional Mem Pool Log Buffer Transaction System Log Group iblogfile0 iblogfile1 iblogfile2 Storage ibdata1 space 0 IBUF_HEADER IBUF_TREE TRX_SYS RSEG_HDR DICT_HDR Doublewrite Buffer Block 1 (64 pages) Block 2 (64 pages) UNDO_LOG Data Dict. SYS_TABLES SYS_COLUMNS SYS_INDEXES SYS_FIELDS Tables with file_per_table A.ibd B.ibd C.ibd

9 High-level Overview Caching Buffer Pool Page Cache Adaptive Hash Indexes Buffer Pool LRU Buffer Pool Flush List Data Dictionary Cache Additional Mem Pool Log Buffer Transaction System Log Group iblogfile0 iblogfile1 iblogfile2 Storage ibdata1 space 0 IBUF_HEADER IBUF_TREE TRX_SYS RSEG_HDR DICT_HDR Doublewrite Buffer Block 1 (64 pages) Block 2 (64 pages) UNDO_LOG Data Dict. SYS_TABLES SYS_COLUMNS SYS_INDEXES SYS_FIELDS Tables with file_per_table A.ibd B.ibd C.ibd

10 Terminology and Concepts

11 Things we re not covering Binary Log: Log of statements or row images used for replication and completely unrelated to InnoDB s transactional logging (even though they are binary too). General/Error Logs: Informational logs produced by the server.

12 Redo What people mean when they say InnoDB log(s). Two or more redo logs (ib_logfile0, ib_logfile1, ) preallocated and used in a circular fashion. Information necessary to redo (or re-apply) changes to data stored in InnoDB. Used to reconstruct changes if necessary (crash recovery).

13 Log Sequence Number (LSN) A 64-bit unsigned integer representing a point in time in the redo log system, counted since log initialization. (At initial install time.) An ever-increasing number similar to the number of bytes logged.

14 Redo logging in practice The redo log is a Write-Ahead Log (WAL) a technique used to ensure data integrity in case of a crash. Logs of a change must be written to disk before the actual changes they represent. Every data change is written to the redo log buffer (as a redo record) as the table data is modified. The redo log buffer is written to disk (but not synced) when each page modification is complete.

15 Redo logging in practice The redo log is synced to disk up to a the page s modification LSN before a page can be flushed to disk. Depending on innodb_flush_log_at_trx_commit, the log may be synced to disk at transaction commit.

16 Checkpoint A point in time, represented by an LSN value (checkpoint LSN). Checkpoint means that changes to the table data for every change made prior to the checkpoint LSN have been flushed. Once a checkpoint has been completed, redo logs prior to the checkpoint are no longer needed.

17 Undo Information necessary to: undo (or revert) changes to data stored in InnoDB, and can be used to rollback transactions. Used to implement multi-versioning: users can see a consistent view ( snapshot ) of the database.

18 Rollback Pointer (ROLL_PTR) A rollback segment number, page number, and page offset pointing to a specific undo log record containing the previous version of a record. Used to walk backwards through record versions (history) for any record. Used for multiversioning (to re-create old versions of any record). Used for transaction rollback.

19 Transaction ID (TRX_ID) A 64-bit unsigned integer representing the point at which the transaction started. Incremented with each transaction. Written to each record in clustered indexes. Maximum value written to the system tablespace TRX_SYS page.

20 Transaction Serialization Number (TRX_NO) A 64-bit unsigned integer representing the maximum TRX_ID at the time of commit. Written to undo log header on commit. Used for purge of old record versions.

21 Undo logging in practice With every change to the database, undo logs store the previous version of the data. Every clustered (PK) index record has a pointer to the previous version of the record (called the rollback pointer ). Every undo record stores a rollback pointer to its previous version. Every undo log change must also be redo logged

22 What happens when you UPDATE

23 Transaction start When the transaction is first started: 1. A transaction ID (TRX_ID) is assigned and may be written to the highest transaction ID field in the TRX_SYS page. A record of the TRX_SYS page modification is redo logged if the field is updated. 2. A read view is created based on the assigned TRX_ID.

24 Record modification Each time the UPDATE modifies a record: 1. Undo log space is allocated. 2. Previous values from record are copied to undo log. 3. Record of undo log modifications are written to redo log. 4. Page is modified in buffer pool; rollback pointer is pointed to previous version written in undo log. 5. Record of page modifications are written to redo log. 6. Page is marked as dirty (needs to be flushed to disk).

25 What about other users/transactions? Once records are modified, even if uncommitted, other transactions will see the changes. Depending on their isolation level, each transaction will: Use the latest version of the record from the index page. Find the latest committed version of the record. Find the version of the record corresponding to their read view. Any transaction needing a record version other (older) than the latest version from the index page must use undo logs to reconstruct the record at a previous version.

26 Transaction commit When the transaction is committed (implicitly or explicitly): 1. Undo log page state is set to purge (meaning it can be cleaned up when it s no longer needed). 2. Record of undo log modifications are written to redo log. 3. Redo log buffer is flushed to disk (depending on the setting of innodb_flush_log_at_trx_commit).

27 Background flush The background flush thread runs continuously to: 1. Find the oldest dirty pages and add them to a flush batch. 2. Ensure the redo log is written and flushed up to the newest LSN in the batch. 3. Write a block of up to 128 pages to the double write buffer (and wait for sync). 4. Write each page to its final destination in a tablespace file.

28 Inclusion in checkpoint A checkpoint is generated automatically (periodically) to: 1. Ensure everything older than that point has been flushed to the tablespace files (and if not, flush it immediately). 2. Write a checkpoint to the redo log header. (From this point on the redo log before the checkpoint is no longer needed.)

29 Background purge The background purge thread runs continuously to: 1. Find the oldest undo logs that are no longer needed in each rollback segment, and: 2. Actually remove any delete-marked records from indexes. 3. Free the undo log pages. 4. Prune the history lists.

30 What really happens in the redo log?

31 Undo initialization (at first write in transaction) lsn type size space page offset UNDO_HDR_CREATE MLOG_2BYTE MLOG_2BYTE MLOG_2BYTE MULTI_REC_END 1

32 Undo initialization (at first write in transaction) type page offset UNDO_HDR_CREATE UNDO_LOG MLOG_2BYTE UNDO_LOG TRX_UNDO_PAGE_HDR + TRX_UNDO_PAGE_START MLOG_2BYTE UNDO_LOG TRX_UNDO_PAGE_HDR + TRX_UNDO_PAGE_FREE MLOG_2BYTE UNDO_LOG TRX_UNDO_SEG_HDR + free + TRX_UNDO_LOG_START MULTI_REC_END

33 Undo log and record update lsn type size space page offset UNDO_INSERT COMP_REC_UPDATE_IN_PLACE

34 Transaction commit lsn type size space page offset MLOG_2BYTE MLOG_4BYTE MLOG_4BYTE MLOG_4BYTE MLOG_2BYTE MLOG_4BYTE MLOG_2BYTE MLOG_4BYTE MLOG_2BYTE MLOG_4BYTE MLOG_2BYTE MLOG_4BYTE MLOG_8BYTE MLOG_2BYTE MULTI_REC_END 1

35 Transaction commit lsn type page offset MLOG_2BYTE UNDO_LOG TRX_UNDO_STATE MLOG_4BYTE RSEG_HDR TRX_RSEG_UNDO_SLOTS + 1 * TRX_RSEG_SLOT_SIZE MLOG_4BYTE RSEG_HDR TRX_RSEG_HISTORY_SIZE MLOG_4BYTE UNDO_LOG TRX_UNDO_HISTORY_NODE + FLST_PREV (page) MLOG_2BYTE UNDO_LOG TRX_UNDO_HISTORY_NODE + FLST_PREV (offset) MLOG_4BYTE UNDO_LOG TRX_UNDO_HISTORY_NODE + FLST_NEXT (page) MLOG_2BYTE UNDO_LOG TRX_UNDO_HISTORY_NODE + FLST_NEXT (offset) MLOG_4BYTE RSEG_HDR TRX_RSEG_HISTORY + FLST_FIRST (page) MLOG_2BYTE RSEG_HDR TRX_RSEG_HISTORY + FLST_FIRST (offset) MLOG_4BYTE UNDO_LOG TRX_UNDO_HISTORY_NODE + FLST_PREV (page) MLOG_2BYTE UNDO_LOG TRX_UNDO_HISTORY_NODE + FLST_PREV (offset) MLOG_4BYTE RSEG_HDR TRX_RSEG_HISTORY + FLST_LEN MLOG_8BYTE UNDO_LOG TRX_UNDO_TRX_NO MLOG_2BYTE UNDO_LOG TRX_UNDO_DEL_MARKS MULTI_REC_END

36 Transaction commit lsn type page description MLOG_2BYTE UNDO_LOG TRX_UNDO_STATE MLOG_4BYTE RSEG_HDR TRX_RSEG_UNDO_SLOTS + 1 * MLOG_4BYTE RSEG_HDR TRX_RSEG_HISTORY_SIZE MLOG_4BYTE UNDO_LOG TRX_UNDO_HISTORY_NODE + FLST_PREV (page) MLOG_2BYTE UNDO_LOG TRX_UNDO_HISTORY_NODE + FLST_PREV (offset) MLOG_4BYTE UNDO_LOG TRX_UNDO_HISTORY_NODE + FLST_NEXT (page) MLOG_2BYTE UNDO_LOG TRX_UNDO_HISTORY_NODE + FLST_NEXT (offset) MLOG_4BYTE RSEG_HDR TRX_RSEG_HISTORY + FLST_FIRST (page) MLOG_2BYTE RSEG_HDR TRX_RSEG_HISTORY + FLST_FIRST (offset) MLOG_4BYTE UNDO_LOG TRX_UNDO_HISTORY_NODE + FLST_PREV (page) MLOG_2BYTE UNDO_LOG TRX_UNDO_HISTORY_NODE + FLST_PREV (offset) MLOG_4BYTE RSEG_HDR TRX_RSEG_HISTORY + FLST_LEN MLOG_8BYTE UNDO_LOG TRX_UNDO_TRX_NO MLOG_2BYTE UNDO_LOG TRX_UNDO_DEL_MARKS MULTI_REC_END Set undo page state to active. Write undo page number to rollback segment slot. Increase total history size by 1. Set up history list node-to-node links. Add UNDO_LOG page to beginning of history list. Link page from previous link on previous page which was bumped from the beginning of the history list. Increase length of history list by 1. Write transaction serialization number (TRX_NO) to undo page. Update delete-marks flag in undo page.

37 Physical Structure of Undo Logs

38 Basic overview of InnoDB page structure Default page size is 16 KiB Every page has a FIL header (38 bytes) and trailer (8 bytes) The FIL header contains information to determine structure of the rest of the page On a 16 KiB page, there are 16,338 bytes of usable space

39 Basic Page Overview 0 38 FIL Header (38) Other headers and page data, depending on page type. Total usable space: 16,338 bytes FIL Trailer (8)

40 0 16 KiB 32 KiB 48 KiB 64 KiB ibdata1 File Overview FSP_HDR: Filespace Header / Extent Descriptor IBUF_BITMAP: Insert Buffer Bookkeeping INODE: Index Node Information SYS: Insert Buffer Header INDEX: Insert Buffer Root TRX_SYS: Transaction System Header SYS: First Rollback Segment SYS: Data Dictionary Header Fixed Page Number Allocations More pages... Page 64 Page 128 Page 192 Double Write Buffer Block 1 (64 pages) Double Write Buffer Block 2 (64 pages) More pages...

41 Basics of undo log structure Undo logs are stored in UNDO_LOG pages in the system tablespace. The transaction system has up to 128 rollback segments (previous 1, now 128). Each rollback segment has 1024 slots for transactions. Each rollback segment slot points to an undo log segment header on the first undo log page of the segment. Undo log pages may be re-used as they are no longer needed.

42 TRX_SYS Overview FIL Header (38) Transaction ID (8) TRX_SYS FSEG Entry (10) Rollback Segment 0: Space (4) Rollback Segment 0: Page (4)... Rollback Segment 127: Space (4) Rollback Segment 127: Page (4) (Empty Space: bytes) Master Log Info (112) (Empty Space: 888 bytes) Binary Log Info (112) (Empty Space: 980 bytes) Doublewrite Buffer Info (38) (Empty Space: 154 bytes) FIL Trailer (8)

43 SYS_RSEG_HEADER Overview FIL Header (38) Rollback Segment Header (34) Undo Segment Slot 0 (4)... Undo Segment Slot 1023 (4) (Empty Space: bytes) FIL Trailer (8)

44 Rollback Segment Header Max Size (4) History Size (4) History List Base Node (16) Rollback Segment FSEG Entry (10)

45 UNDO_LOG Overview FIL Header (38) UNDO Page Header (18) UNDO Segment Header (26) Undo Records FIL Trailer (8)

46 Undo Segment Header State (2) Last Log Offset (2) Undo Segment FSEG Entry (10) Undo Segment Page List Base Node (16) Undo Page Header Undo Page Type (2) Latest Log Record Offset (2) Free Space Offset (2) Undo Page List Node (12)

47 Undo Record N-2 N N+2 N+3 Previous Record Offset (2) Next Record Offset (2) Type + Extern Flag + Compilation Info (1) Undo Number (1-11) Table ID (1-11) (Variable length undo record data.)

48 Physical Structure of Redo Logs

49 Log Group Structure Logical Record 2 Record 3 Record 5 Record 1 Record 4 Record 6 Record 8 Record 9 Record 7 Physical Log File Header Block Header Data Block Trailer Block Header Data Block Trailer Log File Header Block Header Data Block Trailer Block Header Data Block Trailer ib_logfile0 ib_logfile1

50 Log File 0 Overview Log File Header Block Checkpoint 1 Block (Unused Block) Checkpoint 2 Block Log Header Log blocks, each 512 bytes.

51 Log Checkpoint Log File Header Log Group ID (4) Start LSN (8) Log File Number (always 0) (4) Created By (32) Checkpoint Number (8) Checkpoint LSN (8) Buffer Size (4) Archived LSN (8) Log Group Array (unused) (256) Checksum 1 (4) Checksum 2 (4) FSP Free Limit (4) FSP Magic Number (4)

52 Log File 1+ Overview Log File Header Block (Unused Block) (Unused Block) (Unused Block) Log Header Log blocks, each 512 bytes.

53 Log Block 0 12 Flush Flag (1 bit) + Block Number (4) Data Length (2) First Record Offset (2) Checkpoint Number (4) Log records of variable length. Total space available: 496 bytes. 512 Checksum (4)

54 Log Record Overview Single Record Flag (1 bit) + Type (1) Space ID (4) Page Number (4) Record payload, variable length based on type. (Some types have no payload.) Record types DUMMY_RECORD and MULTI_REC_END do not write Space ID or Page Number.

55 Log Record MLOG_nBYTE MLOG_1BYTE, MLOG_2BYTE, MLOG_4BYTE Single Record Flag (1 bit) + Type (1) Space ID (4) Page Number (4) Page Offset (2) Value (4)

56 Log Record MLOG_COMP_REC_INSERT Short Record Info Single Record Flag (1 bit) + Type (1) Space ID (4) Page Number (4) Number of Columns (2) Number of Unique Columns (2) Field Type Info (variable) Previous Record Offset (2) Record Length + Short Record Bit (1-5) Record Info and Status Bits (1) Length of Record Header (1-5) Mismatch Index (to previous record) (1-5) Record Data (variable)

57 Crash Recovery

58 When is crash recovery used? When you crash and restart its namesake. After restoring from backups (LVM snapshot, xtrabackup). Restarting after fast shutdown.

59 Check for unclean shutdown Redo logs and system tablespace are opened. Checkpoints are read to find highest checkpoint number. Redo logs are scanned starting from latest checkpoint. Crash recovery is initiated if any redo records are found.

60 Create file-per-table space ID mapping Open all.ibd files in directories within the data directory. Read their space ID from the page 0 (FSP_HDR) header. Populate the space ID to table name mapping.

61 Torn page recovery All 128 pages from the doublewrite buffer are examined: Each target page from the tablespace is read. If the header and trailer LSN do not match or the page checksum is invalid, the page is restored from the doublewrite buffer. If the doublewrite buffer version of the page is also corrupt, the server will assert (crash).

62 Rollback of uncommitted transactions Transaction system (rollback segments) is initialized. Transactions corresponding to active undo logs are resurrected. Redo log records are applied since the latest checkpoint. Active transactions are rolled back (using undo logs).

63 Hot Backup

64 Making a hot backup 1. Stream redo logs from last checkpoint, saving everything. 2. Copy all space files (ibdata*,.ibd) page by page and verify checksum (each page must not be corrupted, but doesn t need to be the most recent version). 3. Copy all.frm files. 4. Stop streaming redo logs.

65 Restoring from a hot backup 1. Copy all InnoDB and MySQL files into place. 2. Apply all collected redo logs from hot backup process. 3. Perform crash recovery (reverts back to most recent checkpoint in complete log, not beginning of hot backup). 4. Done

66 Problems and Inefficiencies

67 MySQL Bug #69477 InnoDB: Use of large externally-stored fields makes crash recovery lose data Blobs are written in a single write to the redo logs. If a blob is more than ~10% of the size of the redo logs, it can overwrite the most recent checkpoint. If a crash occurs during this state, data loss occurs.

68 Old read views in transactions Transactions with old read views will: 1. Delay purging of potentially huge amounts of undo log, consuming disk space in the system tablespace. 2. Have to traverse many row versions to find usable versions for their read view. (And be very slow because of it.) This is one of the main reasons that performance degrades for big queries in a non-linear fashion (closer to exponential).

69 Unlimited undo per transaction User opens a transaction. User starts an infinite loop updating a single row. Every intermediate row version is kept The newest version is what s actually in the index page, even if it s not committed yet. All other users will skip over millions of uncommitted intermediate row versions to get to a usable row version for their read views. Slooooooooow.

70 Whaaaa? mysql> select * from test.t; a b row in set (0.89 sec)

71 >> rec.string => "(a=1) (b= )" >> rec.roll_pointer => {:is_insert=>false, :rseg_id=>76, :undo_log=>{:page=>18383, :offset=>6404}} >> rec.each_undo_record.each_with_index { u, i pp u.undo_record; puts; break if i >= 1 } {:page=>18383, :offset=>6404, :trx_id=> , :roll_ptr=> {:is_insert=>false, :rseg_id=>75, :undo_log=>{:page=>18382, :offset=>6404}}, :key=>[{:name=>"a", :type=>"int UNSIGNED", :value=>1}], :row=>[{:name=>"b", :type=>"int UNSIGNED", :value=>999999}]} {:page=>18382, :offset=>6404, :trx_id=> , :roll_ptr=> {:is_insert=>false, :rseg_id=>74, :undo_log=>{:page=>18381, :offset=>6404}}, :key=>[{:name=>"a", :type=>"int UNSIGNED", :value=>1}], :row=>[{:name=>"b", :type=>"int UNSIGNED", :value=>999998}]}

72 Q & A