From The Linux Foundation

Test Development Guidelines for LSB testing

Test Assertions

Creation of a set of test assertions is a necessary step to bridge the gap between natural language specification and test suite code. The goal of the test suite code is to produce an executable representation of the requirements of the specification under test.

The LSB test project adopts an assertion based technique for test specifications. This follows the IEEE Standard for Test Methods (1003.3). An assertion is developed for each definitive statement in the specification under test. Each assertion is designed to determine whether the statement is true or false for the implementation under test.

Typically, a definitive statement within a specification contains or implies a shall, should or may. A shall statement is an unconditional requirement. A should or may statement is a conditional requirement, based on support of an option. Some judgement needs to be exercised by the assertion writer to ensure that the assertions generated are based upon the requirements of specification, and not in some cases based on non-normative advice to programmers or such like.

For example, a specification might state:

Required files for /bin If the C compiler exists, then the following commands shall exist in /bin. {A list of 34 commands}

This is written as 34 separate test assertions , for example

Reference 3.1-3 (C) If the implementation provides a C compiler: The XYZ command will exist in /bin.

etc.

The above example is a simple case, common examples might also include if-then-otherwise conditional tests etc. The key is to produce logical, testable statements that can be reasoned about, these are called test assertions. By producing test assertions, we can then produce tests (which maybe produced manually or if using Assertion Definition Language automatically). Typical test suites can contain thousands of test assertions.

Assertions also need to consider the level of testing. POSIX.3 defines three levels:

1. Exhaustive : Full testing of all aspects of a specification

requirement. For example, testing every possible value for an

argument to a function.

2. Thorough : Testing sufficientto validate that the implementation

under test matches a specification requirement without

exhaustive testing. For example, a subset of the possible values of an argument to a function, exercising boundary and representative conditions.

3. Identification : Testing that the implementation-under-test provides

the mechanism necessary to meet the specification

requirement. For example, that a function exists without validating its processing of any arguments.

Testing occurs at the exhaustive level where practical. However in many areas it may not be practical to do so due to the time and system resources required. Most test suites perform thorough testing which often serves as a fair compromise in the coverage/cost ratio. Identification testing is rarely used unless the specification requires it, by stating merely that a feature exists and defining no behaviours or values.

Criteria for Test Development

• The Test Specification

o The premise for all the tests is that they have to be based on a written, publically available test specification.

o The test specification upon which the test suite is based shall conform to the POSIX.3 methodology (which means use of test assertions).

• The Test Suite

o The test suite is aligned with the current draft/issue of the relevant specification. (Mandatory)

o The test suite must be able to establish conformance with all legitimate options and variables defined by the specification. (Mandatory).

o Each individually executable test must provide its own setup and clean up functionality (Mandatory)

o No individually executable test shall introduce side effects that will affect the results of any other tests(Mandatory)

o The unit of test execution should cover no more than a single interface function (Desirable)

o The test suite shall have a capability to check complete and correct configuration and installation. This may include a set of defined confidence checks (Mandatory)

o Configuration and installation checking facilities should be automated (Desirable)

o The test suite must include clear and comprehensive user and programmer documentation