(MODULES-10023) Fix multiple xtrabackup regressions (#1245)

* fix xtrabackup target directory and related regressions

* fix handling of incremental backups with xtrabackup

* MODULES-10023 add parameter $incremental_backups to parent class

* MODULES-10023 add acceptance tests for xtrabackup

* MODULES-10023 add xtrabackup documentation

* MODULES-10023 fix acceptance tests

* MODULES-10023 fix xtrabackup tests on CentOS 6

* MODULES-10023 fix xtrabackup tests on Debian 8, Ubuntu 14.04, Scientific 6

* MODULES-10023 fix xtrabackup tests on SLES, RHEL 8 and RHEL/CentOS 5

* MODULES-10023 fix xtrabackup tests on RHEL/CentOS 5

* MODULES-10023 disable new xtrabackup tests on EOL operating systems

* (maint) exclude oracle6/centos6 platforms on running the feature tests

Co-authored-by: David Swan <david.swan@puppet.co.uk>
Co-authored-by: sheena <sheena@puppet.com>

Author: 3 people

README.md

@@ -14,6 +14,7 @@
     * [Install Percona server on CentOS](#install-percona-server-on-centos)
     * [Install MariaDB on Ubuntu](#install-mariadb-on-ubuntu)
     * [Install Plugins](#install-plugins)
+    * [Use Percona XtraBackup](#use-percona-xtrabackup)
 4. [Reference - An under-the-hood peek at what the module is doing and how](REFERENCE.md)
 5. [Limitations - OS compatibility, etc.](#limitations)
 6. [Development - Guide for contributing to the module](#development)
@@ -392,6 +393,69 @@ mysql::server::db:
 ### Install Plugins
 
 Plugins can be installed by using the `mysql_plugin` defined type. See `examples/mysql_plugin.pp` for futher examples.
+
+### Use Percona XtraBackup
+
+This example shows how to configure MySQL backups with Percona XtraBackup. This sets up a weekly cronjob to perform a full backup and additional daily cronjobs for incremental backups. Each backup will create a new directory. A cleanup job will automatically remove backups that are older than 15 days.
+
+```puppet
+yumrepo { 'percona':
+  descr    => 'CentOS $releasever - Percona',
+  baseurl  => 'http://repo.percona.com/release/$releasever/RPMS/$basearch',
+  gpgkey   => 'https://www.percona.com/downloads/RPM-GPG-KEY-percona https://repo.percona.com/yum/PERCONA-PACKAGING-KEY',
+  enabled  => 1,
+  gpgcheck => 1,
+}
+
+class { 'mysql::server::backup':
+  backupuser        => 'myuser',
+  backuppassword    => 'mypassword',
+  backupdir         => '/tmp/backups',
+  provider          => 'xtrabackup',
+  rotate            => 15,
+  execpath          => '/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin',
+  time              => ['23', '15'],
+}
+```
+
+If the daily or weekly backup was successful, then the empty file `/tmp/mysqlbackup_success` is created, which makes it easy to monitor the status of the database backup.
+
+After two weeks the backup directory should look similar to the example below.
+
+```
+/tmp/backups/2019-11-10_full
+/tmp/backups/2019-11-11_23-15-01
+/tmp/backups/2019-11-13_23-15-01
+/tmp/backups/2019-11-13_23-15-02
+/tmp/backups/2019-11-14_23-15-01
+/tmp/backups/2019-11-15_23-15-02
+/tmp/backups/2019-11-16_23-15-01
+/tmp/backups/2019-11-17_full
+/tmp/backups/2019-11-18_23-15-01
+/tmp/backups/2019-11-19_23-15-01
+/tmp/backups/2019-11-20_23-15-02
+/tmp/backups/2019-11-21_23-15-01
+/tmp/backups/2019-11-22_23-15-02
+/tmp/backups/2019-11-23_23-15-01
+```
+
+A drawback of using incremental backups is the need to keep at least 7 days of backups, otherwise the full backups is removed early and consecutive incremental backups will fail. Furthermore an incremental backups becomes obsolete once the required full backup was removed.
+
+The next example uses XtraBackup with incremental backups disabled. In this case the daily cronjob will always perform a full backup.
+
+```puppet
+class { 'mysql::server::backup':
+  backupuser          => 'myuser',
+  backuppassword      => 'mypassword',
+  backupdir           => '/tmp/backups',
+  provider            => 'xtrabackup',
+  incremental_backups => false,
+  rotate              => 5,
+  execpath            => '/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin',
+  time                => ['23', '15'],
+}
+```
+
 ## Reference
 
 ### Classes

manifests/backup/mysqlbackup.pp

@@ -27,6 +27,7 @@
   $postscript               = false,
   $execpath                 = '/usr/bin:/usr/sbin:/bin:/sbin',
   $optional_args            = [],
+  $incremental_backups      = false,
 ) inherits mysql::params {
 
   mysql_user { "${backupuser}@localhost":

manifests/backup/mysqldump.pp

@@ -28,6 +28,7 @@
   $optional_args            = [],
   $mysqlbackupdir_ensure    = 'directory',
   $mysqlbackupdir_target    = undef,
+  $incremental_backups      = false,
 ) inherits mysql::params {
 
   unless $::osfamily == 'FreeBSD' {

manifests/backup/xtrabackup.pp

@@ -51,9 +51,18 @@
   }
 
   if $incremental_backups {
+    # Warn if old backups are removed too soon. Incremental backups will fail
+    # if the full backup is no longer available.
+    if ($backuprotate.convert_to(Integer) < 7) {
+      warning(translate('The value for `backuprotate` is too low, it must be set to at least 7 days when using incremental backups.'))
+    }
+
+    # The --target-dir uses a more predictable value for the full backup so
+    # that it can easily be calculated and used in incremental backup jobs.
+    # Besides that it allows to have multiple full backups.
     cron { 'xtrabackup-weekly':
       ensure  => $ensure,
-      command => "/usr/local/sbin/xtrabackup.sh --target-dir=${backupdir} ${additional_cron_args}",
+      command => "/usr/local/sbin/xtrabackup.sh --target-dir=${backupdir}/$(date +\\%F)_full ${additional_cron_args}",
       user    => 'root',
       hour    => $time[0],
       minute  => $time[1],
@@ -62,13 +71,23 @@
     }
   }
 
+  # Wether to use GNU or BSD date format.
+  case $::osfamily {
+    'FreeBSD','OpenBSD': {
+      $dateformat = '$(date -v-sun +\\%F)_full'
+    }
+    default: {
+      $dateformat = '$(date -d "last sunday" +\\%F)_full'
+    }
+  }
+
   $daily_cron_data = ($incremental_backups) ? {
     true  => {
-      'directories' => "--incremental-basedir=${backupdir} --target-dir=${backupdir}/$(date +\\%F_\\%H-\\%M-\\%S)",
+      'directories' => "--incremental-basedir=${backupdir}/${dateformat} --target-dir=${backupdir}/$(date +\\%F_\\%H-\\%M-\\%S)",
       'weekday'     => '1-6',
     },
     false => {
-      'directories' => "--target-dir=${backupdir}",
+      'directories' => "--target-dir=${backupdir}/$(date +\\%F_\\%H-\\%M-\\%S)",
       'weekday'     => '*',
     },
   }

manifests/params.pp

@@ -38,7 +38,7 @@
   $client_dev_package_provider = undef
   $daemon_dev_package_ensure   = 'present'
   $daemon_dev_package_provider = undef
-  $xtrabackup_package_name     = 'percona-xtrabackup'
+  $xtrabackup_package_name_default = 'percona-xtrabackup'
 
 
   case $::osfamily {
@@ -55,8 +55,12 @@
         /^(RedHat|CentOS|Scientific|OracleLinux)$/: {
           if versioncmp($::operatingsystemmajrelease, '7') >= 0 {
             $provider = 'mariadb'
+            if versioncmp($::operatingsystemmajrelease, '8') >= 0 {
+              $xtrabackup_package_name_override = 'percona-xtrabackup-24'
+            }
           } else {
             $provider = 'mysql'
+            $xtrabackup_package_name_override = 'percona-xtrabackup-20'
           }
           if versioncmp($::operatingsystemmajrelease, '8') >= 0 {
             $java_package_name   = 'mariadb-java-client'
@@ -158,6 +162,7 @@
       $mycnf_owner         = undef
       $mycnf_group         = undef
       $server_service_name = 'mysql'
+      $xtrabackup_package_name_override = 'xtrabackup'
 
       if $::operatingsystem =~ /(SLES|SLED)/ {
         if versioncmp( $::operatingsystemmajrelease, '12' ) >= 0 {
@@ -237,6 +242,10 @@
       } else {
         $php_package_name = 'php5-mysql'
       }
+      if  ($::operatingsystem == 'Ubuntu' and versioncmp($::operatingsystemrelease, '16.04') < 0) or
+          ($::operatingsystem == 'Debian') {
+        $xtrabackup_package_name_override = 'percona-xtrabackup-24'
+      }
 
       $python_package_name = 'python-mysqldb'
       $ruby_package_name   = $::lsbdistcodename ? {
@@ -548,6 +557,12 @@
     },
   }
 
+  if defined('$xtrabackup_package_name_override') {
+    $xtrabackup_package_name = pick($xtrabackup_package_name_override, $xtrabackup_package_name_default)
+  } else {
+    $xtrabackup_package_name = $xtrabackup_package_name_default
+  }
+
   ## Additional graceful failures
   if $::osfamily == 'RedHat' and $::operatingsystemmajrelease == '4' and $::operatingsystem != 'Amazon' {
     fail(translate('Unsupported platform: puppetlabs-%{module_name} only supports RedHat 5.0 and beyond.', {'module_name' => $module_name}))

manifests/server/backup.pp

@@ -48,6 +48,8 @@
 #   Dump stored routines (procedures and functions) from dumped databases when doing a `file_per_database` backup.
 # @param include_triggers
 #   Dump triggers for each dumped table when doing a `file_per_database` backup.
+# @param incremental_backups
+#   A flag to activate/deactivate incremental backups. Currently only supported by the xtrabackup provider.
 # @param ensure
 # @param time
 #   An array of two elements to set the backup time. Allows ['23', '5'] (i.e., 23:05) or ['3', '45'] (i.e., 03:45) for HH:MM times.
@@ -88,6 +90,7 @@
   $provider                 = 'mysqldump',
   $maxallowedpacket         = '1M',
   $optional_args            = [],
+  $incremental_backups      = true,
 ) inherits mysql::params {
 
   if $prescript and $provider =~ /(mysqldump|mysqlbackup)/ {
@@ -120,6 +123,7 @@
       'execpath'                 => $execpath,
       'maxallowedpacket'         => $maxallowedpacket,
       'optional_args'            => $optional_args,
+      'incremental_backups'      => $incremental_backups,
     }
   })
 }