Backup and site migration

This chapter discusses various methods for creating a backup of a Drupal website, and how to migrate a complete site to a new location. It also discusses how to use the CLI to mirror all the files that make up a Drupal website to another web server.

Table of contents

• Introduction
• Backup and Migrate
  • Setting up local backup directories for Backup and Migrate
  • Backup profiles
  • Making a local manual backup
  • Setting up a local scheduled backup
  • Offsite backup
  • Creating a minimal backup profile for Backup and Migrate
  • Restoring with mysql
  • Restoring with drush
• Using mysqldump for backing up the database
• Backing up the files
• Migrating a site
  • Perparing for migration
  • Migrate the database
  • Migrate the files
  • Finalizing migration
• Merging sites
• Final word

Drupal projects mentioned in this chapter: Backup and Migrate, Enabled Modules.

Introduction

Whether you are hosting your site yourself, or you use a hosting service, you need to have a robust backup system in place. If you are hosting your site yourself, there will be nobody else that will take care of backups. Most hosting services offer backup as part of the standard plan, and you may opt in on this provided you trust your provider. This chapter only describes how to set up and manage your own backups.

Something known as the “3-2-1” backup practice is recommended. This is an easy-to-remember acronym for a keeping at least three (3) copies of your data, store two (2) backup copies on different storage media, with one (1) of them located offsite. A common approach is to have the following three backup locations:

1. Production database.
2. Local SQL-dumps of the production database.
3. Offsite archive of timestamped SQL-dumps of the production database.

The “3-2-1” backup practice ensures that a local copy is available for fast roll-back for non-mechanical failures (e.g. a hacker compromises or a bug corrupts the database), while an offsite backup ensures that the data does not disappear if your producation location breaks down mechanically, e.g. by fire or flood, so that all data stored on that location are lost.

Obviously, the more backups you have, the less chance you have to lose all of them at once. But, the “3-2-1” backup practice is a well-proven and cost-efficient way of securing the data against loss.

A working Drupal website is made up of the following components:

• Database content (entities, etc.).
• File content (media assets).
• Configuration (variables, content types, fields, views, etc.).
• Code and its environment (modules, libraries, themes and translations).

To be able to restore a site from backup, all components must be recreated from a saved copy, or the original copies must remain in place.

To migrate a site to another web server (physical or virtual) the components that make up the source website must be copied to the target website and put together to mirror the configuration of the source website.

For an introduction to Drupal backups, see the chapter in the Drupal administration guide chapter titled Backing up a site. Also see the documentation for the Drupal Backup and Migrate project, Backup using the CLI.

This chapter will describe how to make a backup of a Drupal website and restore that at the same locaton, and how to migrate a Drupal website to another location. It will focus on using the contributed project Backup and Migrate for both tasks. The use of some CLI tools for these tasks will also be described.

Backup and Migrate

This contributed project integrates the tasks of creating manual and scheduled backups into the Drupal site itself. It has some benefits compared to the basic mysqldump CLI command:

1. It recognises ephemeral tables and truncates them automatically (manual intervention required if a tablename prefix is used).
2. It has built-in scheduling with “smart” delete.
3. It can be used from the CLI with the help of drush.

To use Backup and Migrate, you must first download the project from Drupal.org, and install it as you would normally install a contributed Drupal project. See: Installing modules for general information about installing a module project.

However, if you have drush available, the simplest way to install a Drupal project is to to use drush and the CLI. Navigate below the webroot on your web server and give the following command:

$ drush en backup_migrate -y

Backup and Migrate lets you back up your site using different methods, such as:

1. Download (produces an SQL dump that you can download and save).
2. Local backups directory (requires you to configure a private file system path).
3. Offsite destination (requires you to install and configure an offsite destination).

In this chapter, we are not going to use the download method, but the two other methods shall be described.

Setting up local backup directories for Backup and Migrate

How to set the private file system path has been described elsewhere, in the section describing how to set up a download method. But just to make sure, navigate to Configuration » Media » File system and check that the field for the “Private file system path” has been filled in with a valid path.

Then to set up Backup and Migrate to back up to directories below the private file systems path, navigate to Configuration » System » Backup and Migrate » Settings » Destinations and make sure that all the destinations specified exists.

By default, the predefined destinations are “Manual Backups Directory” and “Sceduled Backups Directory”.

Backup profiles

If you navigate to Configuration » System » Backup and Migrate » Settings » Settings profiles, you can override and create new backup settings profiles.

The profile named “Default settings” has sensible defaults. If you want to see what they are click on Override. To inspect and change a custom profile (you need to create it first, using “Add Settings Profile”), click on Edit.

If you alter a profile you must click on Save settings profile to save your changes.

Making a local manual backup

The contents of the site's database are dynamically created by the users of the site. On an active site, the content will be changing frequently.

To create a manual backup using Backup and Migrate, navigate to Configuration » System » Backup and Migrate, select the desired options from the rolldown-menus.

When done selecting options press Backup now.

If the Token module is enabled, the token [site:name] will be used to create the name of the backup-file, otherwise the text version will be used. You may change this by altering the profile under the “Settings” tab, but the default settings are usually adequate.

Setting up a local scheduled backup

To set up scheduled backups, click on the Schedules tab and then on the link Add Schedule. The dialogue shown below will appear.

You should give the schedule a name. You can keep most of the default settings, but if disk space is not unlimited, it is recommended that you tick “ Automatically delete old backups” and select the “Smart Delete” option.

When done, press Save schedule.

Offsite backup

In addition to having local backups at hand, you probably want to also store backups offsite, so that if your local backup is corrupted or physically destroyed, you still have backups.

Backup and Migrate has three built-in offsite destinations:

1. Amazon AWS S3
2. Email
3. FTP Directory

In addition, if you visit the project page of Backup and Migrate, you'll find link to companion contributed projects that extends the project to let you set up other offsite destinations, such as Dropbox, HPCloud and OpenCloud. This chapter will only discuss offsite backup based upon Amazon AWS S3. However due to current limittaions, we will not at present make use of the built-in support for this cloud service in Backup and migrate.

Amazon AWS S3

Amazon Web Services Simple Storage Service (AWS S3) provides cloud storage. You can use AWS S3 to store and retrieve any amount of data at any time, from anywhere on the web. It is well suited for storing backup files offsite.

The contributed projects Backup and Migrate, AWS SDK for PHP and AmazonS3 all advertises support for AWS S3. However, at the time of writing (April 2020), their support for AWS S3 is obsolete, as they are not compatible with AWS SDK PHP version 3. This version of the SDK uses Amazon AWS S3 signature SigV4. This signature has been required in all AWS S3 regions launched after 2013, and will also required for new buckets created after June 24, 2020 in all regions. For details, see Amazon: SigV2 Deprecation. Signature SigV4 is not yet supported by any Drupal 7 project.

I leave the instructions for installing in place below, for those that wants to try it. However, if you (like me) do not want to use this library, you may want to continue reading about the stopgap described below that section.

To be able to use AWS S3 as a backup destination, you need to clone a PHP library located at https://github.com/tpyo/amazon-s3-php-class from github, install it in site/all/libraries under your webroot, and check out the version tagged v0.5.1 (more recent commits do not work). The following sequence of commands shall do it:

$ git clone https://github.com/tpyo/amazon-s3-php-class.git s3-php5-curl
$ mv s3-php5-curl/ drupalroot/sites/all/libraries/
$ cd drupalroot/sites/all/libraries/s3-php5-curl/
$ git checkout v0.5.1
Note: checking out 'v0.5.1'.

You are in 'detached HEAD' state. You can look around, make experimental
…
HEAD is now at 121318e Version bump

This library uses uses Amazon AWS S3 signature SigV2 and currently works with with a few regions, including “US East (N. Virginia)”. It does not work with newer locations such as “EU (Frankfurt)”. Instead, it shows this warning:

User warning: S3::getBucket(): [AccessDenied] Access Denied in S3::__triggerError()
  (line 373 of …/sites/all/libraries/s3-php5-curl/S3.php).

For more information, see this issue @ Drupal.org.

As a stopgap (until Backup and Migrate supports all AWS S3 regions), I do the following:

1. I use Backup and Migrate to back up the database as a local scheduled backup.
2. I use the Unix CLI (i.e. S3cmd) to copy this backup-file to a AWS S3 bucket.
3. I also copy all new files (e.g. attachments, media assets) to an AWS S3 bucket, using a similar command.

The pair of commands below does the copying. You should of course replace the italics in these commands with real paths that exists on your site, and real buckets you have created.

$ s3cmd sync --skip-existing sqldump-directory s3://bucket/sqldump-directory/
$ s3cmd sync --skip-existing files-directory s3://bucket/files-directory/

To automate this, use the Unix command cron to run this at regular intervals.

Creating a minimal backup profile for Backup and Migrate

The Backup and Migrate default setting will by default truncate the standard tables containing temporary data when backing up the database.

But for an absolutely minimal backup, you should create a special profile that truncates the data in the following tables:

To exclude the data from these tables, navigate to Configuration » System » Backup and Migrate » Settings. Click on the link “Add Settings Profile”, and give the profile a name (e.g. “Minimal”) so you can recognise it. Then under Default database backup options, select the tables you don't want included in the backup from the menu as show below. Some of these may already be excluded in the default profile.

When done, click on Save settings profile.

Some low cost hosting plans only allows a single database. Multiple sites are separated by setting a different table prefix for each site, rather having a separate database for each site. Backup and Migrate is not aware of this, and by default makes a backup of the entire database, not just the tables that are part of a single Drupal site. If you use this module to backup a Drual site that share a database with other sites, the module's default settings will put tables belonging to all the sites into the dump. You still get a usable backup, but it will be a lot bigger than it need to be. To isolate the tables from a single site, you must create a custom Backup and Migrate backup profile to exclude the tables belonging to other sites, and truncate the tables containing temporary data from the site you're backing up. (There is an active issue in the project's issue queue to automatically filter on the default prefix if one is set, but it is not worked on.)

The Backup and Migrate module may run out of execusion time when restoring if the SQL dump is large.

Restoring with mysql

While Backup and Migrate is great for creating SQL dumps, restoring sometimes take a very long time. Sometimes, this is due to a damaged backup. For instance, once a restore of a modest 3 Mbyte SQL dump failed with “Fatal error: Out of memory (allocated 2097152) (tried to allocate 20480 bytes) in …/html/includes/cache.inc on line 449”. You may also get this error: “The operation timed out. Try increasing your PHP max_execution_time setting.” Luckily, restoring using the CLI as described below works fine.

Backup and Migrate may use a very long execution time during a restore, or fail due to running out of time or menory. Bailing out halfway in a rollback tend to leave the site broken, and the GUI can no longer be used. If this happens, it is possible to use gunzip to unzip the SQL-dump (mysql requires text) archived by Backup and Migrate, and insert it into the database using the MySQL monitor mysql. Always, after importing a database into your Drupal site, use drush to make dure the database is up-to-date. E.g.:

$ gunzip sqldump.sql.gz
$ mysql -u dbuser -p dbname < sqldump.sql
$ drush updb

I've experienced that this takes more than one hour for an 80 Mbyte SQL-dump on one of my servers, but the rollback works.

See also Mike Crittenden: Exporting and importing big Drupal databases.

Restoring with drush

There are drush commands to interact with Backup and Migrate. These are handy if your site's database is so FUBAR that you are no longer able to restore using the GUI.

To list available backup files:

$ drush bam-backups

To restore a particular backup:

$ drush bam-restore db manual my-backup.mysql.gz

The first argument to drush is the command (“bam-restore” is to restore, using Backup and Migrate as the underlying service). The second argument is the destination (“db” is the default database). The third argument is the source (“manual” is the manual backups directory), The fourth and last argument is the name of the backup file.

See all the drush commands that exists for interacting with Backup and Migrate at drushcommands.com.

The following is not a good sign:

$ drush bam-backups
The drush command 'bam-backups' could not be found. …

It means the site is so FUBAR that drush is unable to view the site. In that case, in the CLI navigate to the directory above the webroot and move the files that make up the site to a directory named html_site to protect them from being zapped by the install:

$ mv html/ html_site/

Now, reinstall Drupal to create a clean install. You may create a new database for this, or reuse the existing one. If you chose the latter option, the present one will be dropped.

After the clean install of Drupal has completed, move the the clean install to a directory named html_clean, move old site (i.e. html_site) back to its original position:

$ mv html/ html_clean/
$ mv html_site/ html/ 

If the settings.php from the old site is different from the settings.php from the clean install, decide wich one to use. E.g. to replace it with the one from the clean install, do:

$ mv html_clean/site/default/settings.php html/site/default/

Try to restore again. You will have to enable Backup and Migrate and configure the path to the private system or the location of offsite before you shall be able to do this.

If the restore is successful, fine! If it breaks, you need to revert to the clean install and try to restore from older backups until you find one that works.

Using mysqldump for backing up the database

If you use the CLI and mysqldump to create a backup of the content of the database, it is recommended that you first clear the cache of the site you are migrating to reduce the size of the dump. To do this manually from the Drupal GUI:

• Clear all caches: Configuration » Performance log messages » Clear all caches.
• Truncate log files: Reports » Recent log messages » Clear log messages.

Then use the CLI:

$ mysqldump -u dbuser -p dbname > example.sql
Enter password:

You can also do everything from the CLI with drush and mysqldump. The sequence of commands should be something like this:

$ drush --root=/var/www/example.com/html --uri=example.com cc all
$ mysqldump -u dbuser -p dbname > example.sql
Enter password:

If you have multiple sites in a single database, and you want to first clear all the caches:

$ drush --root=/var/www/example1.com/html --uri=example1.com cc all
$ drush --root=/var/www/example2.com/html --uri=example2.com cc all
…
$ mysqldump -u dbuser -p dbname > examples.sql
Enter password:

On the other hand, if you have multiple sites is a single database, but only want the tables from a single website in the dump, you must tell mysqldump what tables to include. To do this you first produce a list of the tables you want to back up. If the website example1.com uses the table prefix is “prefix1_”, you can extract the list of table from the CLI:

$ sudo mysql -u root -p -N information_schema \
  -e "select table_name from tables \
  where table_schema = 'databasename' and table_name like 'prefix1_%'" > example1.txt
Enter password:

Note that to run the MySQL monitor as root, you may need to be root in the CLI.

Edit the file and get all the databases onto one line. Then the following will produce the SQL-dump of all tables belonging to example1.com:

$ cd /var/www/example.com/html
$ drush cc all
$ mysqldump -u dbuser -p dbname `cat example1.txt` > example1.sql
Enter password:

This produces a local backup of your database.

If you want to copy the backup offsite to a backup server. You can do that with the CLI using the command scp. Example:

$ scp examples.sql user@example.com:/var/backup

Backing up the files

To create a backup of the files, you may use the CLI and tar to bundle all the files into a tarball, and copy that to a backup server. For example, you may do the following (assuming webroot is /var/www/html):

$ cd /var/www
$ tar -czf allfiles.tar.gz html/
$ scp allfiles.tar.gz user@example.com:/var/backup

However. the non-asset files on a Drupal site consists of the Drupal core, contributed exensions (modules and themes), custom extensions, libararies and translations. On a production site, this is usually stable, and only changes when one of the components is updated. All these components can be restored by downloading a fresh copy from their source.

For a small production site running plain vanilla Drupal, it is usually suffiscient to keep a record of all files that make up the site and where they are located. You can do this with the following CLI-command to record the entire file tree (provided your webroot is /var/www/html):

$ tree /var/www/html > tree.txt
$ scp tree.txt user@example.com:/var/backup

The media assets are files that are uploaded by the site's users. To back up these files, you may mirror them on a backup server. To keep the backup small, you may want to flush the image styles cache prior to creating the tarball:

$ cd /var/www/html/sites/default
$ drush image-flush
$ tar -czf assets.tar.gz files/
$ scp assets.tar.gz user@example.com:/var/backup

Or, to save even more space, you may only back up the directories containing non-cached media assets (analyse your assets to determine what those are).

To restore the site, download fresh copies of all non-asset files from their source and all media assets from backup, then use the tree command again to produce tree2.txt, and then use the diff CLI-command to verify that you've been reproduced the exact same files at the destination site in the same location as the original.

Migrating a site

As you've probably guessed from the name, the Drupal Backup and Migrate package can also be used to migrate a Drupal site to a another web server. In Drupal 7 the non-file contents and the site configuration are stored in the database, while file content (media assets), code and its environment are stored in the file system. In Drupal 8, the site configuration is stored in the file system. For both versions, both the database and the files must be migrated in order to successfully migrate a site.

Backup and Migrate works at the database table level. It works fine for migrating an entire site. It cannot be used for migrating just part of a site, such as its content. Since there are so many interdepen­dencies between tables it is not possible to cleanly extract datasets in a way that allows them to be re-imported.

By default, Backup and Migrate only backs up the database. Version 3 of the module can be configured to also back up the file system. If the site has a lot of media assets, this tends to produce a very large backup file. For this reason, I prefer to handle file migration with CLI tools.

To migrate a site, you need to perform the following steps:

1. Prepare for migration.
2. Make an SQL-dump of the database of the source site, transfer it to the target site, and itsert into the target site database.
3. Transfer all the files that make up the source site to the target site, and restore them to the same location in the file tree below the webroot as they had on the source site.

Perparing for migration

To prepare a site for migration, you should first clean the environment of any excess baggage. Revert to the default theme and language, upgrade everything to the latest stable version, and disable, uninstall and delete anything you no longer need. Here is a checklist of things to do:

1. Put the site in maintenance mode to prevent it being changed by users.
2. Revert to the default theme (Bartik).
3. Revert to the default language (English).
4. Disable, uninstall and delete all modules that you do not use. The project Enabled Modules can be used to quicly identify those that are disabled.
5. Upgrade the core and contributed projects to their latest stable version.
6. Delete all inactive users.
7. Delete all nodes that are no longer relevant.
8. Delete content types that are no longer relevant.
9. Delete orphan files.
10. Flush the image cache (drush image-flush).

You need to create a database on the target to hold your database import. How you do this is described in Set up a database for Drupal. Make a note of the name of the database credentials (i.e. the name of database, the name of the database user, and the password assocaiated with this database user).

Migrate the database

To migrate the database, you create an SQL-dump of the database on your source site, and import it into the empty database you've created of your target site.

The recommended procedure relies on using the Backup and Migrate project to create the SQL-dump. However, if the database is MySQL or MariaDB, it is also possible to use the CLI-command mysqldump to create the SQL-dump. Both methods are described below.

Migrate the database with Backup and Migrate

Create a manual local SQL-dump of the source database as described in the section Making a local manual backup.

Create a new (empty) Drupal site at the target server. This can be standard Drupal installation. Here is a step-by-step description:

1. Create a new Drupal instance at the target server. Use the “Standard” profile. How you do this is decribed in the chapter Installing Drupal.
2. Log in as the super administrator, and put the site into maintenance mode.
3. Configure the path to the private file system (see Download method).
4. Install and enable the Backup and Migrate project on the target site (see Adding an extension to your site).

To import the database into the target, you may either restore it from a file on the target server, download it from an offsite destination or upload it from your desktop computer.

To restore from a file on the target server, place the backup pair (i.e. the files with names ending with .mysql.gz and .info) in the directory private://backup_migrate/manual/ on the target site, navigate to Configuration » System » Backup and Migrate » Saved backups, and then click on the link restore to the right of relevant backup.

To upload the backup from your desktop computer, first make sure the backup file is there, then navigate to Configuration » System » Backup and Migrate » Restore and finally browse the file system on your desktop computer to locate the backup file. After uploading, press “Restore now”.

After restoring the database, you may have to manually edit settings.php to correct the database credentials array and the $base_url setting.

After the database have been migrated into a minimal Drupal site, there will be a mismatch between the file system configuration stored in the database and the actual configuration. The theme may break, and there will be lot of complaints on the screen and from the watchdog about missing files. This is normal and no cause for concern. Things will return to normal after the file system also have been migrated (see below) and caches cleared.

Migrate the database with mysqldump

First, create a SQL-dump of the source database as described in the section Backing up the database with mysqldump.

Then, to import the SQL-dump into the target's database, transport the SQL-dump to the target web server (if it is different from the source) and use the following commands to import and update the database afterwards:

$ mysql -u dbuser -p dbname < sqldump.sql
$ drush updb

The dbuser and dbname should be those of the newly created Drupal site (set up as the last step of preparing for migration).

Migrate the files

If both the source and the target is on the same file system, just recursively copy the webroot from one vhost root to another. For example (assuming the webroot of the source is /var/www/source.com/html and the webroot of the destination is /var/www/destination.com/html):

$ cp -R /var/www/source.com/html /var/www/destination.com/

If both the source and the target is on different file systems, the CLI with the command rsync will copy all files in the file tree root given as an argument. Provided you are logged in on the mirror server (i.e. the backup or the target server), the following rsync command will make a copy of all the files below the webroot in the source server (assuming the webroot is /var/www/html, and that all italics are replaced with relevant strings):

$ rsync -r user@example.com:/var/www/html /var/www

To just sync the files that lives below the sites directory, use one of the following:

$ rsync -r user@example.com:/var/www/html/sites /var/www/html
$ rsync -r user@example.com:/var/www/example.com/html/sites /var/www/example.com/html

The first is for a single website, the second for a whost.

Note that to have a fully working mirror site, you also need to copy the database from the source site to the database on the mirror. You do this by following the instructions already given in the section Migrate the database.

If rsync is not available (many sites have firewall-settings that block ssh), you can migrate the files by first creating a backup of the files by using one of the methods described above.

After copying the tarball to the target destination, unpack it to the new webroot. For example:

$ cd /var/www
$ tar -xvzf allfiles.tar.gz

You may need to change the name of the topmost directory if this has not the same name as was configured for the source web server.

After getting the files in place, edit the site's settings.php to use the correct database credentials to access the migrated databse.

Finalizing migration

After everything has been migrated to the target site, you may turn on the custom theme and additional languages, the clear all caches. Your destination site should now be an exact copy of the source site. Navigate to Reports » Status report to verify that everything is OK (and fix the errors if it is not).

When everything is OK, take the target site out of maintenance mode and start using it.

Merging sites

The above recipe for migraton creates a new, empty database to receive the migrated data. If it is not empty, the data already present will be deleted. This makes this recipe unsuitable for merging sites, for instance the use case where a live production server gets new features and content from a test server while retaining its existing contents.

Final word

You can automate all the operations here with either Unix cron (CLI) or with by using the schedule feature of Backup and Migrate.

Backup and Migrate allows for both manual and scheduled backups. It also lets you set up an offsite destination for backup storage.

If you set up scheduled backups to be stored on the local file system for the site you are backing up, you need to arrange for having it copied (by scp or similar) offsite. How to write the scripts to automate this process is left as an exercise to the reader.

Backup files often contain personal data. Make sure your backup-files are not accessible to outsiders. If you're backing up to the server, you should test to see if your backup files are publicly accessible. Storing backup files on a public file system is a very common cause of information disclosure.

---

Last update: 2020-04-14.