Percona XtraBackup moved to https://jira.percona.com/projects/PXB

error: failed to read metadata

Asked by David Childers

Hello, everyone -

I inherited a backup system that uses xtrabackup running from a script that is run by cron.

Here's some information about the system we are running on:
[root@<hostname> logs]# uname -srvpio
Linux 2.6.32-220.17.1.el6.x86_64 #1 SMP Wed May 16 00:01:37 BST 2012 x86_64 x86_64 GNU/Linux

I really don't know very much about xtrabackup

The backup system was working just fine up until just a couple of hours ago.

Something must have changed and I am working to find out what changed.

In the mean time, I thought I would ask if anyone here has seen anything like the following error:

innobackupex: Waiting for ibbackup (pid=9435) to suspend
innobackupex: Suspend file '/data/backups/incremental/15/xtrabackup_suspended'

xtrabackup: Error: cannot open /data/backups/incremental/14/xtrabackup_checkpoints
xtrabackup: error: failed to read metadata from /data/backups/incremental/14/xtrabackup_checkpoints
innobackupex: Error: ibbackup child process has died at /usr/bin/innobackupex line 374.

I've included the full text of the latest backup session below.

Any hints, ideas, suggestions or leads of any kind would be greatly appreciated.

Thank you in advance!

/David C.
-----------------------------------------------------------------------
InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oy
and Percona Inc 2009-2012. All Rights Reserved.

This software is published under
the GNU GENERAL PUBLIC LICENSE Version 2, June 1991.

121018 16:00:01 innobackupex: Starting mysql with options: --password=xxxxxxxx --user='bkpuser' --unbuffered --
121018 16:00:01 innobackupex: Connected to database with mysql child process (pid=9485)
121018 16:00:07 innobackupex: Connection to database server closed
IMPORTANT: Please check that the backup run completes successfully.
           At the end of a successful backup run innobackupex
           prints "completed OK!".

innobackupex: Using mysql Ver 14.14 Distrib 5.5.25, for Linux (x86_64) using readline 5.1
innobackupex: Using mysql server version Copyright (c) 2000, 2011, Oracle and/or its affiliates. All rights reserved.

innobackupex: Created backup directory /data/backups/incremental/16
121018 16:00:07 innobackupex: Starting mysql with options: --password=xxxxxxxx
 --user='bkpuser' --unbuffered --
121018 16:00:07 innobackupex: Connected to database with mysql child process (p
id=9512)
121018 16:00:09 innobackupex: Connection to database server closed

121018 16:00:09 innobackupex: Starting ibbackup with command: xtrabackup_55 --defaults-group="mysqld" --backup --suspend-at-end --target-dir=/data/backups/incremental/16 --incremental-basedir='/data/backups/incremental/15' --parallel=4
innobackupex: Waiting for ibbackup (pid=9519) to suspend
innobackupex: Suspend file '/data/backups/incremental/16/xtrabackup_suspended'

xtrabackup: Error: cannot open /data/backups/incremental/15/xtrabackup_checkpoints
xtrabackup: error: failed to read metadata from /data/backups/incremental/15/xtrabackup_checkpoints
innobackupex: Error: ibbackup child process has died at /usr/bin/innobackupex line 374.
+ '[' 2 -eq 0 ']'
+ echo '\nIncremental Backup failed at 16\n'
\nIncremental Backup failed at 16\n
+ echo -e 'Incremental Backup failed at 16: '
+ mail -s 'Please check the status log.' dchilders@<domain.tld> -- -r bkpuser@localhost
+ '[' 0 -eq 1 ']'
+ exit 1

Question information

Language:
English Edit question
Status:
Solved
For:
Percona XtraBackup moved to https://jira.percona.com/projects/PXB Edit question
Assignee:
No assignee Edit question
Solved by:
Alexey Kopytov
Solved:
Last query:
Last reply:
Revision history for this message
Best Alexey Kopytov (akopytov) said : 		#1

David,

These are the key parts of the output:

121018 16:00:09 innobackupex: Starting ibbackup with command: xtrabackup_55 --defaults-group="mysqld" --backup --suspend-at-end --target-dir=/data/backups/incremental/16 --incremental-basedir='/data/backups/incremental/15' --parallel=4
...
xtrabackup: Error: cannot open /data/backups/incremental/15/xtrabackup_checkpoints
xtrabackup: error: failed to read metadata from /data/backups/incremental/15/xtrabackup_checkpoints

So the script passes a path to the previous incremental backup directory to innobackupex, but it doesn't exist. Because the previous incremental backup failed. Which in turn failed, because it was trying to use /data/backups/incremental/14 as a path to the previous backup which apparently did not succeed as well.

So it looks like at some point one of the incremental backups failed for whatever reasons. And then the script did not handle it correctly and incremented the counter for a failed backup. So all subsequent backups have failed, because they were trying to use non-existing backups as the 'previous' one.

Revision history for this message
David Childers (dchilders) said : 		#2

Ah - OK - and that now makes sense.

When I checked on the situation this morning, I found that the incremental backups are now completing successfully again.

This is because the full backup happened at 2AM (local time) which wound up recreating the directories.

Now the incrementals can find the path they need.

But why do you suppose the error didn't simply say 'path not found' or something like that instead of "failed to read metadata"?

The metadata terminology threw me off the trail. I was thinking there may have been some file system corruption or something like that.

Thank you VERY much for your help! I appreciate it!

/David C.

Revision history for this message
Alexey Kopytov (akopytov) said : 		#3

When creating an incremental backup, XtraBackup wants a path to the previous backup (either a full or an incremental one) to figure out where to start (i.e. the LSN of the last update contained in the previous backup). That is, it reads the metadata of the last backup and gets the LSN value from the xtrabackup_checkpoints file.

Revision history for this message
David Childers (dchilders) said : 		#4

Very good. When the term metadata is used in that context, the error message makes complete sense.

Thank you for helping me to understand.

/David C.