Automated tests

by Gisle Hannemyr

The PHPUnit testing framework is used by symfonic Drupal. This chapter provides an introduction to making use of GitLab's CI testing framework for PHPUnit tests, and how to debug tests.

Table of contents

Introduction

[This was previously about running tests locally. This has been deprecated in favout of GitLab TI tests. This needs a a major rewrite.]

DO: GitLab CI

Setting up to use the GitLab CI

To set up a project for GitLab CI testing, all you need to do it to copy .gitlab-ci.yml to the root of your project and add it to Git. (No changes are necessary.) When you push to the repo, automated tests are enabled. There are three stages:

  1. Build: Use Composer to build the project
  2. Validate: PHPCS, using four different validators.
  3. Test: The actual PHPUnit tests

Even if there are no PHPUnit tests created, you will get feedback from the first two stages.

In the Notify module project, there is a minimal functional test named LoadTest.php. It can be used as a bare bones PHPUnit test. Just remember to replace the three instances of "notify" with the short name of your project.

The PHPUnit framework

Drupal comes with automated tests that help prevent regressions and promoting code quality.

The PHPUnit core testing framework is used by Symfony-based Drupal supports the following types of PHP-based tests:

Tests based upon PHPUnit are created extending one of the following classes:

Sources: PHPUnit: PHPUnit Manual, Drupal.org: PHPUnit in Drupal, Atlantic.bt: A Full Guide to PHPUnit in Drupal 8, Drupal.org: Types of tests.

Socketwrench: Writing Automated Tests in Drupal 8:

  1. Test types and set up
  2. Functional tests
  3. Unit tests
  4. Kernel tests

The BrowserTestBase should be used for most tests. On every test method you have, the whole of Drupal will be reset, all necessary modules will be enabled, the cache cleared and rebuilt. This is a slow and complex process, so each test will take some time.

Drupal considers anything that doesn't do anything with Drupal to be a unit test. So if you are testing something that doesn't need to access the database, entities or output, you can use DrupalUnitTestCase, which is faster and more lightweight.

In addtition, for JS there are these frameworks:

They are not yet described by me.

Invariants

The Drupal test framework is based upon invariants. In programming, an invariant is a condition that must hold everytime the program executes through a specific position.

The Drupal test framework perform invariant checking by means of assertions. I.e. you assert that a certain invariant exists in a specific condition at a certain point in the program's execution.

Note that Drupal's automated testing does not actually tell you whether your code is corret or not. All it does it to let you specify invariants and check whether they hold.

Testing a simple UnitTestCase

In this example, we shall create a very simple PHPUnit test for the custom Hello World extension introduced in a previous chapter.

It should be said that the test used in this demostrator is not meaningful. It is used there to demonstrate the framework. The unit test is not going to interact with Drupal. All it will do is to check that the result of summing two numbers is what one should expect. The test will not interact with Drupal, so it will be written as a subclass of the “UnitTestCase” class.

Start out by creating folder named “tests” in the module's root folder, and inside it a folder named “src”. Since this is a unit test, there should be a subdirectory named “Unit”. The PHP file containing need to have a name ending with “Test”. Here is the path from the module's root folder: “tests/src/Unit/HelloWorldTest.php”. Its contents may be this:

<?php

namespace Drupal\Tests\hello_world\Unit;

use Drupal\Tests\UnitTestCase;

/**
 * Our main test class for hello_world.
 */
class HelloWorldTest extends UnitTestCase {

  public function testAddTwoNumbers() {
    $result = 2 + 2;
    $this->assertEquals($result, 4);
  }
  
}

To run this specific unit test, use:

$ vendor/bin/phpunit web/modules/custom/hello_world/tests/src/Unit/HelloWorldTest.php
PHPUnit 9.5.21 by Sebastian Bergmann and contributors.

Testing Drupal\Tests\hello_world\Unit\HelloWorldTest
.                                                                   1 / 1 (100%)

Time: 291 ms, Memory: 6.00MB

OK (1 test, 1 assertion)

If an assertion fails, the following will appear:

FAILURES!
Tests: 1, Assertions: 1, Failures: 1.

Running a single functional test

Functional tests interact with the website through browser that is part of the testing framework. For this reason, functional tests have to be invoked with a user that is the web server user. You can either configure Apache to run as your own system user or run tests as the web server user. The command below let you do the latter. Source: Drupal.org: Permission problems.

To become the web-server user:

$ sudo -u www-data bash

noteIf you get this warning: “User warning: mkdir(): Permission Denied”, you are not running the tests as the web server user.

To run functional tests you extend the BrowserTestBase class. If the test produces errors, you may want to capture these in a text file named phpunit.txt:

$ vendor/bin/phpunit \
  web/modules/custom/hello_world/tests/src/Functional/HelloWorldFunctionalTest.php \
  > phpunit.txt

To view the putput in the browser, convert it to an HTML-report with this command:

$ phpunitreport.pl

This produces this file in the siteroot: phpunit.html. You may set up a shortcut to it in the admin toolbar.

To delete phpunit.txt in the siteroot directory, use:

$ phpunitclean.sh

Running a single kernel test

To run kernel tests you extend either the KernelTestBase or the EntityKernelTestBase class./p>

Running all tests

PHPUnit is able to locate and run tests under the directory given as argument. Use the first line to run all tests that exists for custom extensions. Use the second line to run ll the tests that exist for the contributed extension Notify.

$ vendor/bin/phpunit web/modules/custom
$ vendor/bin/phpunit web/modules/constrib/notify

Assertions

https://phpunit.readthedocs.io/en/9.5/assertions.html

assertCount()

assertCount($expectedCount, $haystack[, string $message = ''])

The $haystack is something countable, typically an array.

Reports an error identified by $message if the number of elements in is not what is expected.

Debugging tests

Recent versions of PHPUnit swallow output, and the various Devel functions to dump variables doe not work.

See alsoThere is a DSE question: How do I dump variables to the screen when running PHPUnit tests, that provides some ideas about how to get around this.

For example, if you're extending BrowserTestBase, you may add a message to the assertion:

$this->assertCount(1, $this->getMails(), 'One email is sent.', 'Bother!');

noteIt has also been suggested that you may use the --debug flag. In my experience has been that the --debug flag provides a bit more information about when tests start and stop, but not a variable dump. The message parameter produces no screen output, unless your test class is extending BrowserTestBase.

You may create your own logging function. Place this in the file containing the tests, before the test class:

function mylog($var, $label = NULL) {
  $tmp_file = '/tmp/my_debug.log';
  $output = '';
  if(!is_null($label)) {
    $output = $label . ': ';
  }
  $output .= print_r($var, 1) . PHP_EOL;
  file_put_contents($tmp_file, $output, FILE_APPEND);
}

I need to explore whether I can use this technique to look at the current page for debugging:

$pagecontent = $this->drupalGetContent();
debug($pagecontent);

Converting legacy tests to PHPUnit tests

Source: Drupal.org: Converting SimpleTests to PHPUnit tests.

Troubleshooting

Drupal\Core\Config\Schema\SchemaIncompleteException:
Schema errors for mothermayi.settings with the following errors:
  mothermayi.settings:_core missing schema,
  mothermayi.settings:langcode missing schema

This pair of errors indicate a missing or invalid type for the settings defined in the modules's .schema.yml file. For some background, see DSE: Why does simpletest report a missing schema for my custom module?

Final word

[TBA]

Last update: 2022-08-02 [gh].