Automated tests

by Gisle Hannemyr

Drupal 7 supports automating tests. This chapter will show how to write create a basic test-suite as part of a custom module, and how to run the test suite and evaluate its output.

Introduction
Invariants
Setting up
First example
Second example
Tearing down
Test resources
Tests sending mail
Final word

Introduction

The Drupal 7 core testing framework lets you build automated tests by extending one of the following two classes: DrupalWebTestCase or DrupalUnitTestCase.

The DrupalWebTestCase should be used for most tests. On every test method you have, the whole of Drupal will be reset, all necessary modules will be enabled, the cache cleared and rebuilt. This is a slow and complex process, so each test will take some time.

Drupal considers anything that doesn't do anything with Drupal to be a unit test. So if you are testing something that doesn't need to access the database, entities or output, you can use DrupalUnitTestCase, which is faster and more lightweight.

The source code of the project that is used as example in this chapter can be downloaded from the git repository at git.drupal.org with this CLI command:

git clone --branch 7.x-1.x http://git.drupal.org/sandbox/gisleh/2226115.git inf5272

Before starting to write test cases, you should familiarize yourself with Drupal SimpleTest coding standards.

Invariants

The Drupal test framework is based upon invariants. In programming, an invariant is a condition that must hold everytime the program executes through a specific position.

The Drupal test framework perform invariant checking by means of assertions. I.e. you assert that a certain invariant exists in a specific condition at a certain point in the program's execution.

Note that Drupal's automated testing does not actually tell you whether your code is corret or not. All it does it to let you specify invariants and check whether they hold.

Setting up

If you enable the core Testing module, you enable a simple test environment that allows you to create test files to use automated functions to test your module.

The test environment created by the Testing module uses a built-in virtual web browser to walk the installation through a series of tests defined in a test-file. This chapter shall explain how to set up a test file.

note Each of test you set up will run in a completely new Drupal instance, created from scratch for the test. Only the default Drupal core modules are enabled. None of the site configuration and none of the users exists. If your test sequence requires a user, you'll have to create one (just as you would if you were setting up a manual testing environment from scratch).

First example

In this example, we shall create a simple custom module that do very little, and then create a test suite for this module.

The test suite run some very basic tests in order to demonstrate how the Drupal's test framework works. It should be said that the tests used in the demonstrator are not meaningful as tests. They are just there to demonstrate the framework. The machinename of the module shall be testexample.

First, as always when we create a custom module, we need to create a .info-file for it. Below is the content of testexample.info.

name = Testexample
description = An example of using Drupal's automated test.
files[] = testexample.test
core = 7.x

Please note that while testexample.module will be loaded automatically, the assignment of the name of the files containing the planned test class to the files property is required to load the class into Drupal's dynamic code registry.

Next, let us create testexample.module. Since this is simple demonstration, it will only make itself visible by means of hook_help(), but not do anything else. Its content is shown below:

/**
 * Implements hook_help().
 */
function testexample_help($path, $arg) {
  switch ($path) {
    case 'admin/help#testexample':
      $output = '<p>'
        . t('A simple module to demonstrate Drupal\'s automated test.')
        . '</p>';
      $output .= '<p>' . t('The module has no modifiable settings.') . '</p>';
      return $output;
  }
}

Now we can start creating the test class for this module. To keep things simple, this going to be an extremely basic test class, just to demonstrate the mechanics of automated testing. The test class is first going to check that the result of summing two numbers is what one should expect, and the loop through a list of strings to check whether each starts with the letter “t”.

The test suite we plan to write does not interact with Drupal, so it will be written as a subclass of the DrupalUnitTestCase class. The skeleton of the planned test class looks like this:

You need to create a file that contains the test class. It should be named with the module's machine name followed by .test, in this case testexample.test.

class Exampletest extends DrupalUnitTestCase {
  // Placeholder for class variable holding the elements to test.
  // Placeholder for getInfo().
  // Placeholder for test functions.
  // Placeholder for custom assert method.
}

Now we're to exapand each of the placeholder items.

We start by just setting up an array with some elements to test. Since this is just an example, we let it be the integers 1 to 5, spelled out with letters.

  // Set up an array of elements (stings) to test.
  var $testelements = array('one', 'two', 'three', 'four', 'five');

The next thing we need to implement is a static method named getInfo() that provides the testing framework with some iformation about our test. Three pieces of information are required:

The name of the test.
A description of the test.
The group this test is a part of.

This is how this method may look:

  /**
   * Implements getInfo().
   */
  public static function getInfo() {
    return array(
      'name' => t('Unit tests.'),
      'description' => t('Run some basic tests to show demo unit testing.'),
      'group' => t('Inf5272'),
    );
  }

Next, add the test methods. The name of each method must start with the string “test” to be recognised by Drupal.

Having two public test functions with the same name may break stuff.
Re Notify version 1.4 upgrade.

The first method will check the sum of adding two numbers, and instantiate a pre-defined assert method to check that the result is what we expect. The second will loop through the elements we want to test, and instantiate the assert method for each element.

This is how these two method may look:

  /**
   * Test that the result of adding two numbers is what we expect.
   */
  function testAddTwoNumbers() {
    $result = 2 + 2;
    $this->assertEqual(4, $result, 'The result should be four.');
  }

  /**
   * Test that elements starts with a 't'.
   */
  function testStartsWithT() {
    // Loop through the elements and instantiate the assert method.
    foreach ($this->testelements as $element) {
      $this->assertStartsWithT($element);
    }
  }

For the last test method, we shall use a custom assert method that will check whether the elements starts with a “t”. It makes use of the pre-defined assertTrue to provide a TRUE or FALSE test result.

  /**
   * Custom assert method. Checks that element starts with a 't'.
   *
   * @param $element
   *   The element to check the assertion for.
   * @return
   *   None.
   */
  function assertStartsWithT($element) {
    $this->assertTrue('t' == $element[0], 'Element "' . $element
      . '" starts with a \'t\'.');
  }

Now, we're ready to pull it all together. This is how is how the class Exampletest may look like:

class Exampletest extends DrupalUnitTestCase {

  // Set up an array of testelements to go through.
  var $testelements = array('one', 'two', 'three', 'four', 'five');

  /**
   * Implements getInfo().
   */
  public static function getInfo() {
    return array(
      'name' => t('Unit tests.'),
      'description' => t('Run some basic tests to show demo unit testing.'),
      'group' => t('Inf5272'),
    );
  }

  /**
   * Test that the result of adding two numbers is what we expect.
   */
  function testAddTwoNumbers() {
    $result = 2 + 2;
    $this->assertEqual(4, $result, 'The result should be four.');
  }

  /**
   * Test that elements starts with a 't'.
   */
  function testStartsWithT() {
    // Loop through the elements and instantiate the assert method.
    foreach ($this->testelements as $element) {
      $this->assertStartsWithT($element);
    }
  }

  /**
   * Custom assert method. Checks that element starts with a 't'.
   *
   * @param $element
   *   The element to check the assertion for.
   * @return
   *   None.
   */
  function assertStartsWithT($element) {
    $this->assertTrue('t' == $element[0], 'Element "' . $element
      . '" starts with a \'t\'.');
  }

}

After installing and enabling the demonstrator module, we should now be able to navigate to Configuration » Development » Testing and see the test installed as one of the test available, as shown on the screen dump below.

Tick the checkbox to select what tests to run.

To run the test, tick the checkbox to the left of it, and click “Run tests”. The result from running the tests in the Exampletest class is shown below:

Result of running Exampletest.

Three tests passed, and three tests failed. The “bugs” that produces the failures has been introduced deliberately in this example, to demonstrate how failures look. The reason for the failed tests is that the words “one”, “four” and “five” do not start with a “t”, thereby violating the asserting that all words should start with a “t”.

Second example

In the second example, we're going to test creation of an article node. This time, we need to extend the class DrupalWebTestCase, since we're going to interact with a Drupal entity.

There is nothing different about the static method named getInfo(). We just change the information to match the new test.

What is new in this example is the setUp() method. In this method, we install Drupal, enable the testexample module, create a user, grant that user a permission, and log the user in on the site that just has been created. You should always use the setUp() method we have to create and enable everything that shall be needed during the lifetime of the test class instance.

Then in testCreateArticle(), an article is created an posted. The asserr method just checks that the standard text that appears after an article has been created appears.

class Secondtest extends DrupalWebTestCase {
  protected $user;

  /**
   * Implements getInfo().
   */
  public static function getInfo() {
    return array(
      'name' => t('Web tests.'),
      'description' => t('Test ability to create an article.'),
      'group' => t('Inf5272'),
    );
  }

  /**
   * Set-up Drupal and login user.
   */
  public function setUp() {
    // Enable any modules required for the test. This should be an array of
    // module names. 
    parent::setUp(array('testexample'));
    // Create and log in our user. Give permissions.
    $this->user = $this->drupalCreateUser(array(
      'create article content',
      ));
    $this->drupalLogin($this->user);
  }

  /**
    * Tests creation of an article node.
    */
  public function testCreateArticle() {
    // Create node to edit.
    $edit = array();
    $edit['title'] = $this->randomName(8);
    $edit["body[und][0][value]"] = $this->randomName(16);
    $this->drupalPost('node/add/article', $edit, t('Save'));
    $this->assertText(t('Article @title has been created.',
      array('@title' => $edit['title'])));
  }

}

Unlike the previous example, there are no deliberate bugs in this, so no failures should be reported.

See also For an alternative version of this second example, see: Testing (simpletest) Tutorial on Drupal.org. Rather than relying on the core article node type, this tutorial shows how to create new content type called simpletest_example. The tutorial will explain how to test the simpletest_example content type to ensure it functions properly.

note Note that you must not call the function variable_set() in your public setUp() function before invoking parent::setUp(). At that point, the database for the test instance is not yet created, so the call will interact with the production database.

Tearing down

There is also a sister method to setUp() called tearDown(). This method is executed after each test is run.

It can be used to undo some of the tasks done by the tests. However, you usually do not need this. By using the SimpleTest API functions, however, most of this is handled for you automagically.

public function tearDown() {
  // Teardown tasks go here.
  parent::tearDown();
}

Test resources

The Drupal debug function works with tests, and will place its output in the test result log. Example:

$_debug = format_date(REQUEST_TIME, 'custom', 'Y-m-d H:i:s');
debug($_debug, 'REQUEST_TIME');

The following documentation is helpful when designing tests:

API functions: This is a cheat sheet of some functions that are helpful during testing. These include controlling the internal testing browser, creating users with specific permissions, and simulating clicking on links that appear in the page.
Assertions: This is a library on the pre-defined assertions that are available for use in the Drupal testing framework.

See also In addition to the Drupal core test framework discussed in this chapter, there are projects to provide JavaScript-based front-end automated testing (FAT), such as Test Swarm. There also exists projects that integrates Drupal with third party FAT, such as Selenium.

Tests sending mail

The testing module will default to using the TestingMailSystem. This does not send any mail, but gives access to the contents of mails with this method.

$mails = $this->drupalGetMails();
$lastmail = count($mails) - 1;

Debugging tests

[For techniques involving passing setting persistent variables in the datebase, passing options with GET, check out anonymous_publishing.test.]

To look at the current page for debugging, you can use:

$pagecontent = $this->drupalGetContent();
debug($pagecontent);

Final word

Please note that for every test method you have, the testing module will re-install the whole Drupal installation, enable all the modules specified, clear the cache and rebuild everything. That is a slow and complex process, so think about the number of test methods you create.

One way to limit the number, but still have the tests grouped into multiple functions, is have a single testMethod() that calls multiple other functions which are not prefixed with the string “test”. For example, verifyStuff(), verifyOtherStuff().

These shall share the same environment, so make sure that they neither conflict nor rely on each other.

If you're the maintainer of a contributed module, enable it for automated testing. That will automatically run all tests for every patch that you upload to the issue queue. This serves as a first guard against patches written by others breaking the module.

Other closing points:

Automated tests are never complete, there is always something that is not tested. So manual testing is necessary too, although you can focus on the things that are not automatically tested, which saves a lot of time.
Tests can be wrong too, make sure to verify that the test is correct when you have test failures.

Last update: 2014-04-19 [gh].