Backup and site migration

This chapter discusses various methods for creating a backup of a Drupal website, and how to migrate a complete site to a new location. It also discusses how to use the CLI to mirror all the files that make up a Drupal website to another web server.

Table of contents

• Introduction
• Backup and migrate
  • Setting up local backup directories for Backup and migrate
  • Backup profiles
  • Making a local manual backup
  • Setting up a local scheduled backup
  • Offsite backup
  • Creating a minimal backup profile for Backup and migrate
  • Restoring with mysql
  • Restoring with drush
• Using mysqldump for backing up the database
• Backing up the files
• Migrating a site
  • Perparing for migration
  • Migrate the database
  • Migrate the files
  • Finalizing migration
• Merging sites
• Final word

Drupal projects mentioned in this chapter: Backup and migrate, Enabled Modules.

Introduction

Whether you are hosting your site yourself, or you use a hosting service, you need to have a robust backup system in place. If you are hosting your site yourself, there will be nobody else that will be making backups. Most hosting services offer backup as part of the standard plan, and you may opt in on this provided you trust your provider. This chapter only describes how to set up and manage your own backups.

For an introduction to Drupal backups, see the chapter in the Drupal administration guide chapter titled Backing up a site. Also see the documentation for the Drupal Backup and Migrate project, Backup using the CLI.

A working Drupal website is made up of the following components:

• Database content (entities, etc.).
• File content (media assets).
• Configuration (variables, content types, fields, views, etc.).
• Code and its environment (modules, libraries, themes and translations).

To be able to restore a site from backup, all components must be recreated from a saved copy, or from original sources.

The best practice is to store the copy off-site, so that the backup copy does not disappear if the site breaks down mechanically, is compromised by hackers, or otherwise becomes damaged.

To migrate a site to another web server (physical or virtual) the components that make up the source website must be copied to the target website and put together to mirror the configuration of the source website.

Backup and migrate

This contributed project integrates the tasks of creating manual and scheduled backups into the Drupal site itself. It has some benefits compared to the basic mysqldump CLI command:

1. It recognises ephemeral tables and truncates them automatically (manual intervention required if a tablename prefix is used).
2. It has built-in scheduling with “smart” delete.
3. It can be used from the CLI with the help of drush.

To use Backup and Migrate, you must first download the project from Drupal.org, and install it in your Drupal website. See: Using composer to download and update files for information about installing a module project with composer.

The recommended way to install a Drupal project is to to use composer and the CLI. Navigate to the siteroot (the directory containing your site's composer.json and give the following pair of commands (replace 4.1 with the latest stable version number):

$ composer require 'drupal/backup_migrate:^4.1'
$ drush en backup_migrate

There is a major bug for version 5.0.0-RC2reported in this issue: Unable to restore backup from the Saved Backups tab. Until that is fixed, for D9, use the patched version that is in the local repo on do20.

Backup and Migrate lets you back up your site to two different destinations:

1. Download (produces an SQL dump that you can download and save)
2. Local manual backups directory (require you to configure a private file system path)

In this tutorial, we are going to use Backup and Migrate to backup the database to a local backups directory that lies below the path to the private file system.

Setting up local backup directories for Backup and migrate

[Screenshots and descriptions may still be D7-based.]

How to set the private file system path has been described elsewhere, in the section describing how to set up a download method. But just to make sure, navigate to Configuration » Media » File system and check that the field for the “Private file system path” has been filled in with a valid path.

Then to set up Backup and Migrate to back up to directories below the private file systems path, navigate to Configuration » System » Backup and Migrate » Settings » Destinations and make sure that all the destinations specified exists.

By default, the predefined destinations are “Manual Backups Directory” and “Sceduled Backups Directory”.

Backup profiles

If you navigate to Configuration » System » Backup and Migrate » Settings » Settings profiles, you can override and create new backup settings profiles.

The profile named “Default settings” has sensible defaults. If you want to see what they are click on Override. To inspect and change a custom profile (you need to create it first, using “Add Settings Profile”), click on Edit.

If you alter a profile you must click on Save settings profile to save your changes.

Making a local manual backup

The contents of the site's database are dynamically created by the users of the site. On an active site, the content will be changing frequently.

To create a manual backup using Backup and Migrate, navigate to Configuration » System » Backup and Migrate, select the desired options from the rolldown-menus.

When done selecting options press Backup now.

If the Token module is enabled, the token [site:name] will be used to create the name of the backup-file, otherwise the text version will be used. You may change this by altering the profile under the “Settings” tab, but the default settings are usually adequate.

Setting up a local scheduled backup

To set up scheduled backups, click on the Schedules tab and then on the link Add Schedule. The dialogue shown below will appear.

You should give the schedule a name. You can keep most of the default settings, but if disk space is not unlimited, it is recommended that you tick “ Automatically delete old backups” and select the “Smart Delete” option.

When done, press Save schedule.

Offsite backup

In addition to having local backups at hand, you probably want to also store backups offsite, so that if your local backup is corrupted or physically destroyed, you still have backups. Best practice is to have at least have three different copies of your data. i.e.:

1. Production database.
2. Local SQL-dumps of the production database.
3. Offsite archive of timestamped SQL-dumps of the production database.

Backup and Migrate was originally designed to integrate with a cloud service named NodeSquirrel for offsite backup. However, it was discontinued at November 1, 2019. Integration with it was removed from Backup and Migrate in December 2019.

Below are pointers to an alternative cloud services that may be used for the offsite backup archive: AWS S3.

AWS S3

Amazon Web Services Simple Storage Service (AWS S3) provides cloud storage. You can use AWS S3 to store and retrieve any amount of data at any time, from anywhere on the web. It is well suited for storing backup files offsite.

There exists modules to integrate Drupal and AWS S3 (see, for example, the projects named S3 File System and AmazonS3). However, for backup purposes, I've created my own tools based upon the CLI.

I've started to collect my notes about these tools in this chapter: AWS S3 (it is somewhat sketchy at the moment).

Creating a minimal backup profile for Backup and migrate [D8]

The Backup and migrate default setting will by default truncate some standard tables containing temporary data when backing up the database.

But for an absolutely minimal backup, you should create a special profile that truncates the data in the following tables:

These additional tables was truncated in D7. Unclear weather they exist in D8:

• {accesslog} – access log generated by statistics module (disabled by default);
• {aggregator_item} – if aggregator is enabled: loses RSS aggregator items;
• {captcha_sessions} – logs CAPTCHA sessions, seems safe to truncate;
• {ctools_css_cache} – if ctools is enabled;
• {ctools_object_cache} – if ctools is enabled;
• {syslog} – system events log generated by syslog module (disabled by default);

To exclude the data from these tables, navigate to Configuration » Development » Backup and Migrate Settings. Click on the link “Add Settings Profile”, and give the profile a name (e.g. “Minimal”) so you can recognise it. Then in the menu under “Exclude data from these tables”, select the tables you want truncated in the backup. Some of these may already be truncated in the default profile.

When done, click on Save.

Some low cost hosting plans only allows a single database. Multiple sites are separated by setting a different table prefix for each site, rather having a separate database for each site. Backup and Migrate is not aware of this, and by default makes a backup of the entire database, not just the tables that are part of a single Drupal site. If you use this module to backup a Drual site that share a database with other sites, the module's default settings will put tables belonging to all the sites into the dump. You still get a usable backup, but it will be a lot bigger than it need to be. To isolate the tables from a single site, you must create a custom Backup and Migrate backup profile to exclude the tables belonging to other sites, and truncate the tables containing temporary data from the site you're backing up. (There is an active issue in the project's issue queue to automatically filter on the default prefix if one is set, but it is not worked on.)

The Backup and Migrate module may run out of execusion time when restoring if the SQL dump is large.

Restoring with mysql

While Backup and Migrate is great for creating SQL dumps, restoring sometimes take a very long time. Sometimes, this is due to a damaged backup. For instance, once a restore of a modest 3 Mbyte SQL dump failed with “Fatal error: Out of memory (allocated 2097152) (tried to allocate 20480 bytes) in …/html/includes/cache.inc on line 449”. You may also get this error: “The operation timed out. Try increasing your PHP max_execution_time setting.” Luckily, restoring using the CLI as described below works fine.

Backup and Migrate may use a very long execution time during a restore, or fail due to running out of time or menory. Bailing out halfway in a rollback tend to leave the site broken, and the GUI can no longer be used. If this happens, it is possible to use gunzip to unzip the SQL-dump (mysql requires text) archived by Backup and Migrate, and insert it into the database using the MySQL monitor mysql. E.g.:

$ gunzip sqldump.sql.gz
$ mysql -u dbuser -p dbname < sqldump.sql

I've experienced that this takes more than one hour for an 80 Mbyte SQL-dump on one of my servers, but the rollback works.

Restoring with drush

There are drush commands to interact with Backup and migrate. These are handy if your site's database is so FUBAR that you are no longer able to restore using the GUI.

To list available backup files:

$ drush bam-backups

To restore a particular backup:

$ drush bam-restore db manual my-backup.mysql.gz

The first argument to drush is the command (“bam-restore” is to restore, using Backup and migrate as the underlying service). The second argument is the destination (“db” is the default database). The third argument is the source (“manual” is the manual backups directory), The fourth and last argument is the name of the backup file.

See all the drush commands that exists for interacting with Backup and migrate at drushcommands.com.

The following is not a good sign:

$ drush bam-backups
The drush command 'bam-backups' could not be found. …

It means the site is so FUBAR that drush is unable to view the site. In that case, in the CLI navigate to the siteroot and move the files that make up the site to a directory named html_site to protect them from being zapped by the install:

$ mv html/ html_site/

Now, reinstall Drupal to create a clean install. You may create a new database for this, or reuse the existing one. If you chose the latter option, the present one will be dropped.

After the clean install of Drupal has completed, move the the clean install to a directory named html_clean, move old site (i.e. html_site) back to its original position:

$ mv html/ html_clean/
$ mv html_site/ html/ 

If the settings.php from the old site is different from the settings.php from the clean install, decide wich one to use. E.g. to replace it with the one from the clean install, do:

$ mv html_clean/site/default/settings.php html/site/default/

Try to restore again. You will have to enable Backup and migrate and configure the path to the private system or the location of off-site before you shall be able to do this.

If the restore is successful, fine! If it breaks, you need to revert to the clean install and try to restore from older backups until you find one that works.

Using mysqldump for backing up the database

If you use the CLI and mysqldump to create a backup of the content of the database, it is recommended that you first clear the cache of the site you are migrating to reduce the size of the dump. To do this manually from the Drupal GUI:

• Clear all caches: Configuration » Performance log messages » Clear all caches.
• Truncate log files: Reports » Recent log messages » Clear log messages.

Then use the CLI:

$ mysqldump -u dbuser -p dbname > example.sql
Enter password:

You can also do everything from the CLI with drush and mysqldump. The sequence of commands should be something like this:

$ drush --root=/var/www/example.com/html --uri=example.com cc all
$ mysqldump -u dbuser -p dbname > example.sql
Enter password:

If you have multiple sites in a single database, and you want to first clear all the caches:

$ drush --root=/var/www/example1.com/html --uri=example1.com cc all
$ drush --root=/var/www/example2.com/html --uri=example2.com cc all
…
$ mysqldump -u dbuser -p dbname > examples.sql
Enter password:

On the other hand, if you have multiple sites is a single database, but only want the tables from a single website in the dump, you must tell mysqldump what tables to include. To do this you first produce a list of the tables you want to back up. If the website example1.com uses the table prefix is “prefix1_”, you can extract the list of table from the CLI:

$ sudo mysql -u root -p -N information_schema \
  -e "select table_name from tables \
  where table_schema = 'databasename' and table_name like 'prefix1_%'" > example1.txt
Enter password:

Note that to run the MySQL monitor as root, you may need to be root in the CLI.

Edit the file and get all the databases onto one line. Then the following will produce the SQL-dump of all tables belonging to example1.com:

$ cd /var/www/example.com/html
$ drush cc all
$ mysqldump -u dbuser -p dbname `cat example1.txt` > example1.sql
Enter password:

This produces a local backup of your database.

If you want to copy the backup offsite to a backup server. You can do that with the CLI using the command scp. Example:

$ scp examples.sql user@example.com:/var/backup

Backing up the files

To create a backup of the files, you may use the CLI and tar to bundle all the files into a tarball, and copy that to a backup server. For example, you may do the following (assuming webroot is /var/www/html):

$ cd /var/www
$ tar -czf allfiles.tar.gz html/
$ scp allfiles.tar.gz user@example.com:/var/backup

However. the non-asset files on a Drupal site consists of the Drupal core, contributed exensions (modules and themes), custom extensions, libararies and translations. On a production site, this is usually stable, and only changes when one of the components is updated. All these components can be restored by downloading a fresh copy from their source.

For a small production site running plain vanilla Drupal, it is usually suffiscient to keep a record of all files that make up the site and where they are located. You can do this with the following CLI-command to record the entire file tree (provided your webroot is /var/www/html):

$ tree /var/www/html > tree.txt
$ scp tree.txt user@example.com:/var/backup

The media assets are files that are uploaded by the site's users. To back up these files, you may mirror them on a backup server. To keep the backup small, you may want to flush the image styles cache prior to creating the tarball:

$ cd /var/www/html/sites/default
$ drush image-flush
$ tar -czf assets.tar.gz files/
$ scp assets.tar.gz user@example.com:/var/backup

Or, to save even more space, you may only back up the directories containing non-cached media assets (analyse your assets to determine what those are).

To restore the site, download fresh copies of all non-asset files from their source and all media assets from backup, then use the tree command again to produce tree2.txt, and then use the diff CLI-command to verify that you've been reproduced the exact same files at the destination site in the same location as the original.

Migrating a site

As you've probably guessed from the name, the Drupal Backup and Migrate package can also be used to migrate a Drupal site to a another web server. In Drupal 7 the non-file contents and the site configuration are stored in the database, while file content (media assets), code and its environment are stored in the file system. In Drupal 8, the site configuration is stored in the file system. For both versions, both the database and the files must be migrated in order to successfully migrate a site.

Backup and Migrate works at the database table level. It works fine for migrating an entire site. It cannot be used for migrating just part of a site, such as its content. Since there are so many interdepen­dencies between tables it is not possible to cleanly extract datasets in a way that allows them to be re-imported.

By default, Backup and Migrate only backs up the database. Version 3 of the module can be configured to also back up the file system. If the site has a lot of media assets, this tends to produce a very large backup file. For this reason, I prefer to handle file migration with CLI tools.

To migrate a site, you need to perform the following steps:

1. Prepare for migration.
2. Make an SQL-dump of the database of the source site, transfer it to the target site, and itsert into the target site database.
3. Transfer all the files that make up the source site to the target site, and restore them to the same location in the file tree below the webroot as they had on the source site.

Perparing for migration

To prepare a site for migration, you should first clean the environment of any excess baggage. Revert to the default theme and language, upgrade everything to the latest stable version, and disable, uninstall and delete anything you no longer need. Here is a checklist of things to do:

1. Put the site in maintenance mode to prevent it being changed by users.
2. Revert to the default theme (Bartik).
3. Revert to the default language (English).
4. Disable, uninstall and delete all modules that you do not use. The project Enabled Modules can be used to quicly identify those that are disabled.
5. Upgrade the core and contributed projects to their latest stable version.
6. Delete all inactive users.
7. Delete all nodes that are no longer relevant.
8. Delete content types that are no longer relevant.
9. Delete orphan files.
10. Flush the image cache (drush image-flush).

You need to create a database on the target to hold your database import. How you do this is described in Set up a database for Drupal. Make a note of the name of the database credentials (i.e. the name of database, the name of the database user, and the password assocaiated with this database user).

Migrate the database

To migrate the database, you create an SQL-dump of the database on your source site, and import it into the empty database you've created of your target site.

The recommended procedure relies on using the Backup and migrate project to create the SQL-dump. However, if the database is MySQL or MariaDB, it is also possible to use the CLI-command mysqldump to create the SQL-dump. Both methods are described below.

Migrate the database with Backup and Migrate

Create a manual local SQL-dump of the source database as described in the section Making a local manual backup.

Create a new (empty) Drupal site at the target server. This can be standard Drupal installation. Here is a step-by-step description:

1. Create a new Drupal instance at the target server. Use the “Standard” profile. How you do this is decribed in the chapter Installing Drupal.
2. Log in as the super administrator, and put the site into maintenance mode.
3. Configure the path to the private file system (see Download method).
4. Install and enable the Backup and Migrate project on the target site (see Adding an extension to your site).

To import the database into the target, you may either restore it from a file on the target server, download it from an offsite destination or upload it from your desktop computer.

To restore from a file on the target server, place the backup pair (i.e. the files with names ending with .mysql.gz and .info) in the directory private://backup_migrate/manual/ on the target site, navigate to Configuration » System » Backup and Migrate » Saved backups, and then click on the link restore to the right of relevant backup.

To upload the backup from your desktop computer, first make sure the backup file is there, then navigate to Configuration » System » Backup and Migrate » Restore and finally browse the file system on your desktop computer to locate the backup file. After uploading, press “Restore now”.

After restoring the database, you may have to manually edit settings.php to correct the database credentials array and the $base_url setting.

After the database have been migrated into a minimal Drupal site, there will be a mismatch between the file system configuration stored in the database and the actual configuration. The theme may break, and there will be lot of complaints on the screen and from the watchdog about missing files. This is normal and no cause for concern. Things will return to normal after the file system also have been migrated (see below) and caches cleared.

Migrate the database with mysqldump

First, create a SQL-dump of the source database as described in the section Backing up the database with mysqldump.

Then, to import the SQL-dump into the target's database, transport the SQL-dump to the target web server (if it is different from the source) and use the following commands:

$ mysql -u dbuser -p dbname < sqldump.sql

The dbuser and dbname should be those of the newly created Drupal site (set up as the last step of preparing for migration).

Migrate the files

If both the source and the target is on the same file system, just recursively copy the webroot from one vhost root to another. For example (assuming the webroot of the source is /var/www/source.com/html and the webroot of the destination is /var/www/destination.com/html):

$ cp -R /var/www/source.com/html /var/www/destination.com/

If both the source and the target is on different file systems, the CLI with the command rsync will copy all files in the file tree root given as an argument. Provided you are logged in on the mirror server (i.e. the backup or the target server), the following rsync command will make a copy of all the files below the webroot in the source server (assuming the webroot is /var/www/html, and that all italics are replaced with relevant strings):

$ rsync -r user@example.com:/var/www/html /var/www

Note that to have a fully working mirror site, you also need to copy the database from the source site to the database on the mirror. You do this by following the instructions already given in the section Migrate the database.

If rsync is not available (many sites have firewall-settings that block ssh), you can migrate the files by first creating a backup of the files by using one of the methods described above.

After copying the tarball to the target destination, unpack it to the new webroot. For example:

$ cd /var/www
$ tar -xvzf allfiles.tar.gz

You may need to change the name of the topmost directory if this has not the same name as was configured for the source web server.

After getting the files in place, edit the site's settings.php to use the correct database credentials to access the migrated databse.

Finalizing migration

After everything has been migrated to the target site, you may turn on the custom theme and additional languages, the clear all caches. Your destination site should now be an exact copy of the source site. Navigate to Reports » Status report to verify that everything is OK (and fix the errors if it is not).

When everything is OK, take the target site out of maintenance mode and start using it.

Merging sites

The above recipe for migraton creates a new, empty database to receive the migrated data. If it is not empty, the data already present will be deleted. This makes this recipe unsuitable for merging sites, for instance the use case where a live production server gets new features and content from a test server while retaining its existing contents.

Final word

You can automate all the operations here with either Unix cron (CLI) or with by using the schedule feature of Backup and Migrate.

Backup and Migrate allows for both manual and scheduled backups. It also lets you set up an offsite destination for backup storage.

If you set up scheduled backups to be stored on the local file system for the site you are backing up, you need to arrange for having it copied (by scp or similar) offsite. How to write the scripts to automate this process is left as an exercise to the reader.

Backup files often contain personal data. Make sure your backup-files are not accessible to outsiders. If you're backing up to the server, you should test to see if your backup files are publicly accessible. Storing backup files on a public file system is a very common cause of information disclosure.

---

Last update: 2020-03-02.