Regression test suite

The Admidio regression test suite is intended to protect the behavior of Admidio Core against regressions during development.

Once set up, running regression tests after changes to the Admidio codebase is as simple as running composer test:all in the Admidion installation directory:

PS C:\Users\OpenTools\Development\admidio> composer test:all
PHPUnit 9.6.36 by Sebastian Bergmann and contributors.
 
Runtime:       PHP 8.4.3
Configuration: C:\Users\OpenTools\Development\admidio\phpunit.xml
 
......
  Setting up Admidio test database...
  Installing Admidio production setup...
  ✓ Dropped 49 existing tables
  ✓ Database initialized
  ✓ Schema created
  ✓ Default data installed
  ✓ Administrator user created
 
.........................................................  63 / 412 ( 15%)
............................................................... 126 / 412 ( 30%)
............................................................... 189 / 412 ( 45%)
............................................................... 252 / 412 ( 61%)
............................................................... 315 / 412 ( 76%)
............................................................... 378 / 412 ( 91%)
..................................                              412 / 412 (100%)
 
Time: 07:05.459, Memory: 50.00 MB
 
OK (412 tests, 6450 assertions)

It is useful both for developers contributing to Admidio itself and for third-party developers who build modules, plugins, integrations or other extensions against Admidio.

The suite does not only test isolated PHP classes. Depending on the test layer it also exercises the real Admidio database abstraction, Entities, Services, the production installer, command-line interface, filesystem handling and mail delivery.

This page describes how to set up, run and extend the regression test suite.

For the general setup of an Admidio development installation also see Set up a test environment.

Never run the database-backed regression tests against a production database or against a development database containing data you want to keep.

The regression environment is intentionally destructive.

Database-backed test runs recreate the Admidio schema using the production installer. Existing Admidio tables in the configured test database may therefore be removed.

The test harness contains additional safeguards. Among other checks, the configured database name must contain test as a separate token.

Use a dedicated database name such as:

admidio_test

Do not use a shared database and do not try to bypass the safety checks.

Filesystem tests have a similar safeguard. They are only allowed to perform destructive operations below:

tests/adm_my_files

and require the regression-test marker file contained in that directory.

The suite is divided into several layers. A test should be placed in the lowest layer that can reliably test the behavior in question.

The suite should not be interpreted as a browser UI test suite. Many tests deliberately stop at the Entity, Service or CLI acceptance boundary.

Historic version-to-version database upgrade coverage is a separate lifecycle concern. A green regression run should only be interpreted as covering upgrade paths when corresponding upgrade tests exist in the current branch.

The most important rule when adding regression tests is:

The action being tested must be performed by production Admidio code.

A test must not reproduce the expected Admidio behavior inside a fixture, helper or mock and then verify the behavior it implemented itself.

For example, a test for a Service operation should normally:

1. create only the prerequisites needed by the test;
2. call the real Admidio Service;
3. let that Service call the normal Admidio Entities and database abstraction;
4. verify the resulting state independently, for example through a new Entity instance or a direct prepared database query.

A test should not implement the same database writes itself and then claim that the Service was tested.

This distinction is especially important for operations that perform more than one action, for example:

1. creating reciprocal user relations;
2. creating or updating related records;
3. enforcing organization boundaries;
4. updating changelog information;
5. maintaining sequence values;
6. applying permissions;
7. sending messages or emails;
8. creating thumbnails or archive files.

Fixtures are there to create prerequisites. They are not substitutes for the production workflow being tested.

Before running the suite, install the normal development dependencies from the Admidio repository.

The exact PHP version and required PHP extensions are defined by the current composer.json.

You need:

1. a checkout of the Admidio source tree;
2. Composer;
3. the PHP extensions required by Admidio;
4. a dedicated MariaDB, PostgreSQL or MySQL test database;
5. the matching PDO database driver;
6. Mailpit when running the mail integration test;
7. GD and ZIP support for the photo/filesystem tests.

Install the Composer dependencies from the Admidio root directory:

composer install

Copy the supplied example configuration:

cp .env.test.example .env.test

On Windows, copy the file using Explorer or PowerShell instead.

The process environment takes precedence over values stored in .env.test, which is useful in CI environments.

A typical configuration looks like this:

TEST_DATABASE_ENGINE=mariadb
TEST_FILES_PATH=./tests/adm_my_files
 
TEST_DB_MARIADB_HOST=127.0.0.1
TEST_DB_MARIADB_PORT=3306
TEST_DB_MARIADB_USER=admidio
TEST_DB_MARIADB_PASS=admidio_test
TEST_DB_MARIADB_NAME=admidio_test
 
TEST_DB_POSTGRES_HOST=127.0.0.1
TEST_DB_POSTGRES_PORT=5432
TEST_DB_POSTGRES_USER=admidio
TEST_DB_POSTGRES_PASS=admidio_test
TEST_DB_POSTGRES_NAME=admidio_test
 
TEST_DB_MYSQL_HOST=127.0.0.1
TEST_DB_MYSQL_PORT=3306
TEST_DB_MYSQL_USER=admidio
TEST_DB_MYSQL_PASS=admidio_test
TEST_DB_MYSQL_NAME=admidio_test
 
TEST_MAIL_HOST=127.0.0.1
TEST_MAIL_PORT=1025
 
TEST_MAILPIT_API_HOST=127.0.0.1
TEST_MAILPIT_API_PORT=8025

The checked-in .env.test.example is the authoritative reference for the variables supported by the current branch.

Only configure credentials for disposable test databases.

The easiest way to provide the database servers and Mailpit is the Docker Compose test environment supplied with the repository.

From the directory containing the Compose configuration, start the test services:

docker compose up -d

The current regression environment provides database services for the normal test matrix and Mailpit for SMTP testing.

If you use your own database servers instead, simply adjust .env.test accordingly.

MySQL can also be tested against an external MySQL instance when one is not part of the local Compose configuration.

The default regression configuration expects:

The Mailpit regression test deliberately does not depend on Docker's health status.

Some Docker environments may show the Mailpit container as unhealthy even though Mailpit itself is working correctly. The test checks what actually matters:

1. whether Admidio can send the message through Mailpit's SMTP listener;
2. whether the delivered message appears through the Mailpit HTTP API.

Therefore a Docker unhealthy label alone is not a reason to disable the mail test.

Run the complete suite from the Admidio root directory:

composer test:all

This is the normal command before submitting a change that can affect several parts of Admidio.

For fast feedback while working on pure PHP logic:

composer test:unit

Unit tests deliberately do not initialize the regression database or filesystem environment.

During development it is often useful to run only the test currently being worked on:

vendor/bin/phpunit tests/Integration/Inventory/InventoryTest.php

or, for example:

vendor/bin/phpunit tests/Integration/Filesystem/DocumentsFilesystemServiceTest.php

PHPUnit filtering can be used for an individual regression:

vendor/bin/phpunit --filter testName

Replace testName with the actual method name.

After the focused test passes, run composer test:all before considering the change complete.

Database-backed PHPUnit runs use the current Admidio production installer to create the schema.

This is intentional: an old database dump must not allow an installation or schema regression to remain unnoticed.

Normal database integration tests run inside a transaction. The base test case rolls the transaction back after the test so that tests remain isolated from one another.

Tests should therefore not depend on execution order.

Do not assume that data created by another test still exists.

CLI subprocess tests are different from ordinary transaction-based integration tests.

They start the real Admidio executable as a separate process. That process has:

1. its own production bootstrap;
2. its own database connection;
3. its own exit status;
4. its own standard output and standard error.

This makes subprocess tests an important acceptance boundary.

A typical mutating CLI regression should follow this pattern:

1. process A creates or changes an object;
2. process B reads the object and proves that the change was committed;
3. another command removes the object again;
4. the test verifies that cleanup succeeded.

Do not verify a mutating CLI command only through data held in the test process. A second process or an independent database read should prove persistence.

Also remember that a CLI subprocess cannot see uncommitted data from the PHPUnit transaction of its parent process.

Therefore prerequisites for a mutating CLI scenario must either already exist in the committed baseline or be created through subprocess commands as part of the scenario.

Mutating subprocess tests must clean up after themselves, normally in a finally block, because their changes are committed and cannot be removed by PHPUnit transaction rollback.

The existing tests/Support/CliSubprocess.php and CLI process tests should be used as the pattern instead of implementing another subprocess launcher.

Tests that exercise documents, photos, imports, exports or other file operations must use the protected filesystem test base class.

The regression filesystem root is:

tests/adm_my_files

The FilesystemTestCase verifies that the configured Admidio data directory resolves to this location and that the regression marker exists.

If either check fails, destructive filesystem operations are refused.

When adding a filesystem test:

1. use FilesystemTestCase;
2. create files only below the test data root;
3. register created files and directories for cleanup;
4. call the actual Admidio Service or Entity that performs the filesystem operation;
5. verify both filesystem and database state where applicable;
6. verify cleanup explicitly.

Never point TEST_FILES_PATH at the adm_my_files directory of a real Admidio installation.

Mail regression tests should exercise the real Admidio email stack rather than mocking the mail sender.

The existing Mailpit test follows this path:

PreferencesService
    -> Admidio Email
    -> PHPMailer
    -> SMTP
    -> Mailpit
    -> Mailpit HTTP API

A good mail regression should use a unique recipient or other unique identifier so that it cannot accidentally match a message from an earlier test run.

The test should verify delivery through Mailpit, not merely whether a TCP port is reachable.

Do not make a pure Unit test extend DatabaseTestCase merely because the helper is convenient. Unit tests should remain fast and independent from external infrastructure.

When fixing a bug in a Service or adding a new Service feature, the preferred test structure is:

Arrange

Create the minimum required organizations, users, roles, categories or other prerequisite objects.

Prefer normal Admidio Entities and Services for fixtures.

Act

Call the real production method whose behavior is being tested.

Examples in the current suite include production paths through Services for inventory, profile fields, OIDC, categories, announcements, menu entries, roles, registrations, documents, photos, import/export and messages.

Assert independently

Read the result again independently.

Depending on the feature, use:

1. a new production Entity object;
2. $gDb→queryPrepared();
3. another Service read operation;
4. the physical file written by production code;
5. a second CLI subprocess;
6. the Mailpit HTTP API.

The assertion should not simply inspect an array or object populated by the test fixture.

Reusable fixtures are useful for common prerequisites such as:

1. organizations;
2. users;
3. roles;
4. memberships;
5. categories.

Whenever possible, fixtures should create these objects through the same Admidio Entities or Services used by production code.

A fixture may prepare state, but must not implement the behavior under test.

For example, if production code is expected to create two reciprocal relationship records, a test fixture must not create those same two records and then assert that both exist.

The production relationship operation must create them.

Direct SQL is useful for independent verification of persistence.

Use Admidio's database abstraction and prepared statements:

$row = $this->getDatabase()->queryPrepared(
    'SELECT ... FROM ' . TBL_EXAMPLE . ' WHERE ... = ?',
    array($value)
)->fetch();

Direct SQL is appropriate for asserting what production code wrote.

It should not be used to reproduce the business operation that the test claims to exercise.

Tests must also remain portable across the database engines supported by Admidio. Avoid database-specific SQL unless the test explicitly verifies database-specific abstraction behavior.

Permission and organization-isolation tests require special care.

A weak test can accidentally prove only that the test author knows how to write a secure SQL query.

For example, manually writing:

WHERE object_org_id = ?

inside the test does not prove that the production Admidio query applies that restriction.

Whenever the regression concerns visibility, permissions or organization isolation, invoke the production Entity, Service, rights object, presenter query or CLI operation that is responsible for enforcing the boundary.

Then verify that inaccessible data is really absent.

CLI tests cover two different areas.

Contract tests inspect command registration and validate characteristics such as:

1. command name;
2. description;
3. usage information;
4. arguments;
5. options;
6. callback availability.

When adding a new command, make sure it satisfies the generic CLI contract tests instead of adding exceptions for incomplete metadata.

Workflow tests exercise actual administration operations.

For mutating workflows, use the real executable and verify committed state through another process.

Use machine-readable output such as JSON where the command supports it, rather than parsing human-oriented console formatting.

The test must also verify exit codes and error output where appropriate.

Test file names should end in Test.php so PHPUnit can discover them through the configured test suites.

Test names and @testdox descriptions should state the actual behavior being verified.

Prefer a description such as:

PreferencesService sends a real email through Mailpit

over:

Email works

One regression test should have one clear reason to fail.

A test may perform several steps when those steps form one production workflow.

A regression test for a bug should ideally fail before the production fix and pass after it.

A useful workflow is:

1. reproduce the defect;
2. add the smallest test that demonstrates the incorrect production behavior;
3. run the test and confirm that it fails for the expected reason;
4. implement the production fix;
5. run the focused test again;
6. run related Integration or CLI tests;
7. finally run composer test:all.

Avoid writing the assertion only after changing the production code if doing so makes it impossible to prove that the test actually detects the regression.

Third-party modules and plugins benefit from following the same testing principles even when their tests are maintained outside the Admidio Core repository.

Use a checkout of the Admidio version against which the extension is developed and run tests against a dedicated test database.

For extension tests:

1. invoke real Admidio APIs rather than duplicating them;
2. use Admidio Entities and Services according to the same patterns used by Core;
3. never point tests at a production Admidio database;
4. keep filesystem fixtures separate from a real adm_my_files;
5. use Mailpit or another local SMTP sink for mail behavior;
6. verify organization and permission boundaries through production code;
7. test against all database engines your extension claims to support.

If a third-party change exposes a regression or missing contract in Admidio Core itself, consider contributing the corresponding regression test to the Core suite.

Do not add a test that:

1. stores expected data only in an in-memory array and reads it from the same array;
2. implements a fake Entity or fake Service instead of invoking Admidio;
3. manually repeats the database changes the production Service is supposed to make;
4. passes when no relevant database record exists;
5. catches an unexpected exception and then succeeds unconditionally;
6. uses assertions such as rowCount() >= 0 that cannot fail meaningfully;
7. depends on another test having executed first;
8. writes files outside the protected regression directory;
9. points to a non-test database;
10. assumes a subprocess can see an uncommitted PHPUnit transaction.

A regression test that cannot fail when the corresponding production feature is broken provides false confidence and should be corrected.

Before merging a new test, check the following questions:

1. Does the test invoke the actual production Entity, Service, CLI command or other production path it claims to test?
2. Does it independently verify the resulting state?
3. Does it verify a real database write when persistence is part of the feature?
4. Does it avoid duplicating the production business logic in the test?
5. Is the fixture limited to prerequisites?
6. Is the test isolated from other tests?
7. Does cleanup also run when an assertion fails?
8. Is filesystem access restricted to tests/adm_my_files?
9. Does a CLI write become visible to another independent process?
10. Does a mail test verify a message that actually reached Mailpit?
11. Are permissions and organization boundaries tested through production code?
12. Is the test portable across the relevant supported database engines?
13. Would the test fail if the production behavior it protects were removed?

If the answer to the last question is “no”, the test is probably testing its own setup rather than Admidio.

Use a dedicated database whose name contains test as a separate token, for example:

admidio_test

Check the selected TEST_DATABASE_ENGINE and the corresponding TEST_DB_* variables in .env.test.

Check that:

1. the selected database service is running;
2. hostname and port are correct;
3. the PDO driver is installed;
4. the test database exists;
5. the configured user has sufficient rights to create and remove the Admidio test tables.

Unit tests deliberately do not initialize the Admidio database.

Check .env.test and the database service first.

Verify:

TEST_FILES_PATH=./tests/adm_my_files

and make sure the checked-in file:

tests/adm_my_files/.admidio-regression-test

still exists.

Do not create a workaround that disables this protection.

Ignore the Docker health label initially and check the actual services.

The regression test uses:

SMTP:     127.0.0.1:1025
HTTP API: 127.0.0.1:8025

If those endpoints work, the Mailpit integration test can work even when Docker reports an incorrect health status.

A real CLI subprocess uses another database connection and cannot see uncommitted rows created inside the PHPUnit transaction.

Create the prerequisite through the CLI subprocess itself or use data that belongs to the committed regression baseline.

Normal DatabaseTestCase changes should disappear through transaction rollback.

Mutating subprocess changes are committed independently and must therefore be explicitly removed by the test.

Filesystem changes must be registered for cleanup through FilesystemTestCase.

Run the focused tests while developing, then run:

composer test:all

A successful regression run does not replace code review. Reviewers should still check whether the new tests exercise the correct Admidio production path and whether important error, permission and cross-organization cases are covered.

The goal of the regression suite is not to maximize the number of tests. The goal is to make real Admidio regressions visible as reliable, understandable test failures.