MDEV-39092 Copy Aria data and logs as part of backup by mariadb-andrzejjarzabek · Pull Request #4971 · MariaDB/server

mariadb-andrzejjarzabek · 2026-04-22T08:28:29Z

An interim solution with some room for optimization:

All DDL is blcoked while Aria is being backed up
All table caches purged when Aria backup starts (including for non-Aria tables)
Writes to non-transactional, but not to transactional tables are blocked when table files are being backed up
All commits blocked when Aria log files are being backed up

CLAassistant · 2026-04-22T08:28:38Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

btr_search_drop_page_hash_index() is being invoked on a non-file page (state < FREED). Everywhere else, the more readable !is_read_fixed() or !is_io_fixed() assertions are safe to use.

buf_page_t::set_freed(), buf_page_t::flush(), buf_page_t::write_fix_try(), buf_page_t::write_unfix_try(): Use a compare-and-exchange loop to set or clear a write-fix. While set_freed() and flush() are protected by a page latch, write_fix_try() and write_unfix_try() are not. innodb_backup_batch_wait(): Look up any pages that we are about to back up. For any dirty pages, invoke buf_page_t::write_fix_try() to try to set a fake "write fix" lock-free. If the page is currently write-fixed, acquire and release a page latch to wait wait for the write to complete.

buf_page_create_low(): Invoke buf_page_t::set_freed() to clear a fake "write fix".

log_t::set_archive(false, thd): Ensure that the sequence bit value 0 will be expected on crash recovery and backup. log_t::circular_recovery_from_0(): Accessor for log_sys.circular_recovery_from_sequence_bit_0. buf_flush_wait(), log_checkpoint_low(): Ensure that a checkpoint will be written to reset log_sys.circular_recovery_from_sequence_bit_0. log_t::write_checkpoint(): Reset circular_recovery_from_sequence_bit_0 whenever applicable. We used to blindly reset it in log_t::set_archive(). (cherry picked from commit a848493)

buf_page_t::set_reinit(): Use std::atomic::compare_exchange_weak to prevent races with write_fix_try() and write_unfix_try().

(cherry picked from commit 1370d5a)

…RENAME fil_name_process(): Simplify the logic. If no matching tablespace is found but file_name_t::create_lsn had been set in response to parsing a FILE_CREATE record, try to apply FILE_RENAME to deferred_spaces. fil_delete_apply(): A wrapper for fil_space_free(). When recovering a log in innodb_log_archive=ON format, we must apply FILE_DELETE records in order to avoid a future clash with FILE_CREATE or FILE_RENAME. (cherry picked from commit 7d106c2)

The doublewrite buffer in the system tablespace is only useful for crash recovery in case a data page had been incompletely written by the time the server was killed. If the server is killed during a backup, the backup will be incomplete and unusable anyway. Furthermore, the page range locking makes page writes and backup mutually exclusive.

file_name_t::page0_lsn: Keep track of the last applied recv_sys_t::parse_page0() so that a multi-batch recovery will not reset the file to a smaller size. (cherry picked from commit 0c4039b)

On top of the provisional Aria backup solution provisionally incorporated into MDEV-14992, the following improvements have been made: Aria data and index files are copied under DDL-locked lock level instead of commit-locked, making the backup operation less disruptive. Only log files are copied in the commit-locked phase. Writes to non-transactional Aria tables are blocked in the DDL-locked phase, while writes to transactional tables are written to the log file, allowing consistent point-in-time backup at the time of acquiring the commit lock. Data, index and log files are now copied as a "step" action rather than "end phase" action, allowing them to be copied in parallel using the CONCURRENT option. Non-Aria files, including common SQL-layer metadata and files from other storage engines are copied by the SQL layer rather than the Aria plugin. Note these files are at this time not copied concurrently when the concurrent option is used.

Non-resilient engine files need to be flushed before they are copied to backup at a time when they cannot be further written to. The method of purging table caches is not sufficient for this purpose, as it doesn't flush tables that are in use at the time of the purge. Although the backup lock ensures that they cannot be written to, they can still be opened for reading, in which case the purge does not flush them and the files may be copied incomplete. Instead of purging table caches directly, we call flush_tables(), which in addition to purging, also flushes tables using the HA_EXTRA_FLUSH handler call. This flusing is based on the type of table, only affecting "non-transactional" user tables, which exclude InnoDB tables and transactional Aria tables. The flushing has also been moved from Aria plugin to general SQL code.

The rules for acquiring table locks are different for system tables than for user tables. System tables only become blocked for writing at the MDL_BACKUP_WAIT_COMMIT lock level. At least for statistics tables, which are non-transactional, this means that if they are copied at the same time as the user tables, the files may be copied torn resulting in a corrupt backup. The solution needs to be to flush these files and copy them in BACKUP_PHASE_NO_COMMIT. The actual solution is to flush all system tables at the beginning of that phase and to copy all Aria tables in "mysql" schema also in that phase. This somewhat suboptimal in that some tables in "mysql" are copied under MDL_BACKUP_WAIT_COMMIT, where they could be copied under a lower lock level; however the impact of this is limited by the small number and typically small size of these tables and the trade-off is the (also small) cost of determining their table category and transactionality.

mariadb-andrzejjarzabek requested a review from dr-m April 22, 2026 08:28

gkodinov added the MariaDB Corporation label Apr 23, 2026

mariadb-andrzejjarzabek force-pushed the MDEV-39092 branch 6 times, most recently from 4ef94be to c143ff2 Compare April 25, 2026 19:02

dr-m reviewed Apr 28, 2026

View reviewed changes

mariadb-andrzejjarzabek force-pushed the MDEV-39092 branch from efbc62b to c77278b Compare May 4, 2026 14:00

dr-m force-pushed the MDEV-14992 branch from bcbda03 to e0d850e Compare May 5, 2026 12:06

mariadb-andrzejjarzabek force-pushed the MDEV-39092 branch 2 times, most recently from e89625e to 06fd556 Compare May 17, 2026 19:19

dr-m force-pushed the MDEV-14992 branch from e0d850e to 0c52540 Compare May 18, 2026 09:40

mariadb-andrzejjarzabek force-pushed the MDEV-39092 branch 2 times, most recently from 4d6a19c to d86a6a1 Compare May 20, 2026 05:50

dr-m force-pushed the MDEV-14992 branch from bdec600 to 45e7902 Compare May 20, 2026 06:52

mariadb-andrzejjarzabek force-pushed the MDEV-39092 branch 2 times, most recently from c6f5119 to 9ebb23e Compare May 20, 2026 20:24

dr-m reviewed May 21, 2026

View reviewed changes

Comment thread sql/sql_backup.cc Outdated

mariadb-andrzejjarzabek force-pushed the MDEV-39092 branch 2 times, most recently from 8565956 to 04e3bc2 Compare May 21, 2026 10:53

mariadb-andrzejjarzabek force-pushed the MDEV-39092 branch 2 times, most recently from 14ca552 to c9429eb Compare May 29, 2026 09:47

dr-m force-pushed the MDEV-14992 branch from d595ce0 to c0e48fc Compare May 29, 2026 14:17

mariadb-andrzejjarzabek force-pushed the MDEV-39092 branch 2 times, most recently from d35cd47 to 824afeb Compare June 1, 2026 15:15

mariadb-andrzejjarzabek marked this pull request as ready for review June 1, 2026 15:16

mariadb-andrzejjarzabek marked this pull request as draft June 1, 2026 15:17

dr-m added 7 commits August 6, 2026 16:50

fixup! cb5a923

23eccec

fixup! bfed32b

e2e5c5e

fixup! 23eccec

402622c

btr_search_drop_page_hash_index() is being invoked on a non-file page (state < FREED). Everywhere else, the more readable !is_read_fixed() or !is_io_fixed() assertions are safe to use.

MDEV-40667 Infinite loop on SET GLOBAL innodb_log_archive=ON

75833f9

squash! 402622c

3803826

buf_page_create_low(): Invoke buf_page_t::set_freed() to clear a fake "write fix".

mariadb-andrzejjarzabek force-pushed the MDEV-39092 branch from 1146eea to 1ef48c2 Compare August 10, 2026 11:56

dr-m and others added 17 commits August 10, 2026 16:21

squash! 3803826

cc3dc04

buf_page_t::set_reinit(): Use std::atomic::compare_exchange_weak to prevent races with write_fix_try() and write_unfix_try().

Catch std::bad_alloc

3b0d067

- Find the script even in non-linux environment

1733bf7

(cherry picked from commit 1370d5a)

Merge InnoDB_backup::logs to InnoDB_backup::queue

e6a4b7f

Introduce InnoDB_backup::mutex to reduce log_sys.latch contention

293f016

fixup! 62ae3b1

38d0d83

fixup! 38d0d83

756a8c4

Fix an off-by-one error

ac5617b

fixup! 756a8c4

cfe5b80

MDEV-40756 Incorrect multi-batch recovery of file size

a74da0d

file_name_t::page0_lsn: Keep track of the last applied recv_sys_t::parse_page0() so that a multi-batch recovery will not reset the file to a smaller size. (cherry picked from commit 0c4039b)

fixup! e16b1b4

3bf15cd

Clean up innodb_backup_batch_wait()

a707184

fixup! 293f016

a2020ca

mariadb-andrzejjarzabek force-pushed the MDEV-39092 branch from 1ef48c2 to 6c6299a Compare August 18, 2026 07:54

mariadb-andrzejjarzabek force-pushed the MDEV-39092 branch from 6c6299a to a4a20ea Compare August 19, 2026 14:51

dr-m force-pushed the MDEV-14992 branch 2 times, most recently from e5a9d2d to b625be2 Compare August 24, 2026 13:59

Navigation Menu

Uh oh!

Uh oh!

MDEV-39092 Copy Aria data and logs as part of backup - #4971

MDEV-39092 Copy Aria data and logs as part of backup#4971
mariadb-andrzejjarzabek wants to merge 35 commits into
MariaDB:MDEV-14992from
mariadb-andrzejjarzabek:MDEV-39092

mariadb-andrzejjarzabek commented Apr 22, 2026 •

edited

Loading

Uh oh!

CLAassistant commented Apr 22, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

7 participants

Uh oh!

Uh oh!

Conversation

mariadb-andrzejjarzabek commented Apr 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

CLAassistant commented Apr 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

7 participants

mariadb-andrzejjarzabek commented Apr 22, 2026 •

edited

Loading

CLAassistant commented Apr 22, 2026 •

edited

Loading