Writing a test definition YAML file¶

metadata:
    format: Lava-Test Test Definition 1.0
    name: singlenode-advanced
    description: "Advanced (level 3): single node test commands for Linux Linaro ubuntu Images"

metadata:
    format: Lava-Test Test Definition 1.0
    name: singlenode-advanced
    description: "Advanced (level 3): single node test commands for Linux Linaro ubuntu Images"
    version: "1.0"

maintainer:
    - user.user@linaro.org
os:
    - ubuntu
scope:
    - functional
devices:
    - kvm
    - arndale
    - panda
    - beaglebone-black
    - beagle-xm

install:
    deps:
        - curl
        - realpath
        - ntpdate
        - lsb-release
        - usbutils

install:
    keys:
        - 7C751B3F
        - 6CCD4038
    sources:
        - https://security.debian.org
        - ppa:linaro-maintainers/tools
    deps:
        - curl
        - ntpdate
        - lavacli

run:
    steps:

Writing commands to run on the device¶

1. All commands need to be executables available on the device. This is why the metadata section includes an “os” flag, so that commands specific to that operating system can be accessed.

2. All tests will be run in a dedicated working directory. If a test repository is used, the local checkout of that repository will also be located within that same directory.

3. Avoid assumptions about the base system - if a test needs a particular interpreter, executable or environment, ensure that this is available. That can be done either by using the install step in the test definition, or by building or installing the components as a series of commands in the run steps. Many images will not contain any servers or compilers and many will only have a limited range of interpreters pre-installed. Some of those may also have reduced functionality compared to versions on other systems.

4. Keep the YAML files relatively small and clean to promote easier reuse in other tests or devices. It is often better to have many YAML files to be run in sequence than to have a large overly complex YAML file within which some tests will fail due to changed assumptions. e.g. a smoke test YAML file which checks for USB devices is not useful on devices where lsusb is not functional. It is much easier to scan through the test results if the baseline for the test is that all tests should be expected to pass on all supported platforms.

5. Check for the existence of one of the LAVA test helper scripts, like lava-test-case, in the directories specified by the PATH environment variable to determine how the script should report results. For example, the script may want to use echo or print() when not running inside LAVA and lava-test-case only when that script exists.

   See also

   Write portable test definitions

6. Avoid use of redirects and pipes inside the run steps. If the command needs to use redirection and/or pipes, use a custom script in your repository and execute that script instead. See Writing custom scripts to support tests

7. Take care with YAML syntax. These lines will fail with wrong syntax:

   - echo "test1: pass"
   - echo test2: fail
   

   While this syntax will pass:

   - echo "test1:" "pass"
   - echo "test2:" "fail"
   

Using inline test definitions¶

Rather than refer to a separate file or VCS repository, it is also possible to create a test definition directly inside the test action of a job submission. This is called an inline test definition:

- test:
    timeout:
      minutes: 4
    definitions:
    - repository:
        metadata:
          format: Lava-Test Test Definition 1.0
          name: apache-server
          description: "server installation"
          os:
          - debian
          scope:
          - functional
        run:
          steps:
          - apt -q update
          - apt -q -y install apache2
          - lava-test-case dpkg --shell dpkg -s apache2
      # remember to use -y to allow apt to proceed without interaction
      # -q simplifies the apt output for logging.
      from: inline
      name: apache-server
      path: inline/apache-server.yaml

An inline test definition must:

1. Use the from: inline method.
2. Provide a path to which the definition will be written
3. Specify the metadata, at least:
   1. format: Lava-Test Test Definition 1.0
   2. name
   3. description

Inline test definitions will be written out as single files, so if the test definition needs to call any scripts or programs, those need to be downloaded or installed before being called in the inline test definition.

Download or view the complete example: examples/test-jobs/inline-test-definition-example.yaml

Terminology reference¶

Writing custom scripts to support tests¶

When multiple actions are necessary to get usable output, write a custom script to go alongside the YAML and execute that script as a run step:

run:
    steps:
        - $(./my-script.sh arguments)

You can choose whatever scripting language you prefer, as long as you ensure that it is available in the test image.

Take care when using cd inside custom scripts - always store the initial return value or the value of pwd before the call and change back to that directory at the end of the script.

Example of a custom script wrapping the output:

https://git.linaro.org/lava-team/refactoring.git/tree/functional/dispatcher-unittests.sh

The script is simply called directly from the test shell definition:

https://git.linaro.org/lava-team/refactoring.git/tree/functional/server-unit-tests-stretch.yaml

Example V2 job using this support:

https://git.linaro.org/lava-team/refactoring.git/tree/functional/server-jessie-stretch-debian.yaml

Using commands as test cases¶

If all your test does is feed the textual output of commands to the log file, you will spend a lot of time reading log files. To make test results easier to parse, aggregate and compare, individual commands can be converted into test cases with a pass or fail result. The simplest way to do this is to use the exit value of the command. A non-zero exit value is a test case failure. This produces a simple list of passes and failures in the result bundle which can be easily tracked over time.

To use the exit value, simply precede the command with a call to lava-test-case with a test-case name (no spaces):

run:
    steps:
        - lava-test-case test-ls-command --shell ls /usr/bin/sort
        - lava-test-case test-ls-fail --shell ls /user/somewhere/else/

Use subshells instead of backticks to execute a command as an argument to another command:

- lava-test-case pointless-example --shell ls $(pwd)

For more details on the contents of the YAML file and how to construct YAML for your own tests, see the Writing Tests.

Recording test case results¶

lava-test-case can also be used with a parser with the extra support for checking the exit value of the call:

run:
   steps:
    - "lava-test-case fail-test --shell false"
    - "lava-test-case pass-test --shell true"

This syntax will result in extra test results:

fail-test -> fail
pass-test -> pass

Alternatively, the --result command can be used to output the result directly:

run:
   steps:
      - lava-test-case test5 --result pass
      - lava-test-case test6 --result fail

This syntax will result in the test results:

test5 -> pass
test6 -> fail

Recording test case measurements and units¶

Various tests require measurements and lava-test-case supports measurements and units per test at a precision of 10 digits.

--result must always be specified and only numbers can be recorded as measurements (to support charts based on measurement trends).

run:
   steps:
      - lava-test-case test5 --result pass --measurement 99 --units bottles
      - lava-test-case test6 --result fail --measurement 0 --units mugs

This syntax will result in the test results:

test5 -> pass -> 99.0000000000 bottles
test6 -> fail -> 0E-10 mugs

The simplest way to use this with real data is to use a custom script which runs lava-test-case with the relevant arguments.

Recording sets of test cases¶

Test Set is a way to allow test writers to subdivide individual results within a single Lava Test Shell Definition using an arbitrary label.

Some test definitions run the same test with different parameters. To distinguish between these similar tests, it can be useful to use a test set.

Recording test case references¶

Some test cases may relate to specific bug reports or have specific URLs associated with the result. Simple strings can be recorded separately but if you need to relate a test case result to a URL, consider using lava-test-reference:

lava-test-reference TEST_CASE_ID --result pass|fail|skip|unknown --reference URL

The TEST_CASE_ID can be the same as an existing test case or a new test case.

lava-test-reference has similar support as lava-test-case except that --measurement and --unit options are not supported.

run:
   steps:
      - lava-test-case checked --result pass
      - lava-test-reference checked --result pass --reference https://staging.validation.linaro.org/static/doc/v2/index.html

Test shell parameters¶

The test action in the job definition supports parameters which are passed to the test shell. These parameters can be used to allow different job definitions to use a single test shell definition in multiple ways. A common example of this is a hacking session.

The parameters themselves are inserted into the lava-test-runner and will be available to all Lava Test Shell Definitions used in that test job. The parameters are not exported. The test shell definition needs to support using the parameter and can then use that information to change how external programs behave. This may include using export, it may include changing the command line options.

Obtaining information about the device¶

Some elements of the static device configuration are exposed to the test shell, where it is safe to do so and where the admin has explicitly configured the information. The information is exposed using test shell helpers which currently include:

• lava-target-ip - Devices with a fixed IPv4 address will populate this field. Test writers are able to use this in an LXC to connect to the device, providing that the test shell has correctly raised a network connection and suitable services are configured and running on the device:

  ping -c4 $(lava-target-ip)
  

• lava-target-mac - An alternative to lava-target-ip, declaring the MAC address of the device. Depending on the use case, this may be useful to lookup the IP address of the device:

  echo `lava-target-mac`
  

• lava-target-storage - Where devices have alternative storage media fitted, the id of the block device can be exported. For example, this can help provide temporary storage on the device when the test shell is running a ramdisk or NFS. Some devices may provide a USB mass storage device which could also be exported in this way.

  Note

  This provision is designed to support temporary storage on devices which typically boot over NFS or ramdisk etc. It is intended to allow test writers to run operations which would typically fail without a local filesystem or would block network traffic such that NFS would time out.

  Only a single block device is supported per method. The method itself is simply a label specified by the admin. Often it will relate to the interface used by the block device, e.g. SATA or USB but it could be any string. In the example below, UMS is the label used by the device (as an abbreviation for USB Mass Storage).

  See also

  Extra device configuration and Persistence - test writers are responsible for handling persistence issues. The recommendation is that a new filesystem is created on the block device each time it is to be used.

  The output format contains one line per device, and each line contains the method and the ID for the storage using that method, separated by a TAB character:

  $ lava-target-storage
  UMS     /dev/disk/by-id/usb-Linux_UMS_disk_0_WaRP7-0xac2400d300000054-0:0
  SATA    /dev/disk/by-id/ata-ST500DM002-1BD142_W3T79GCW
  

  Usage: lava-target-storage method

  The output format contains one line per device assigned to the specified ID, with no whitespace. The matched method is not output.:

  $ lava-target-storage UMS
  /dev/disk/by-id/usb-Linux_UMS_disk_0_WaRP7-0xac2400d300000054-0:0
  

  If there is no matching method, exit non-zero and output nothing.

Recording test case data¶

lava-test-case ${VERSION} --result pass

Write portable test definitions¶

lava-test-shell is a useful helper but that can become a limitation. Avoid relying upon the helper for anything more than the automation by putting the logic and the parsing of your test into a more competent language. Remember: as test writer, you control which languages are available inside your test.

lava-test-shell has to try and get by with not much more than busybox ash as the lowest common denominator.

Please don’t expect lava-test-shell to do everything.

Let lava-test-shell provide you with a directory layout containing your scripts, some basic information about the job and a way of reporting test case results - that’s about all it should be doing outside of the MultiNode API.

Avoid using test definitions patterns

Test definitions which can use lava-test-case should not also use test definition patterns like:

"(?P<test_case_id>.*-*):\\s+(?P<result>(pass|fail))"

Test shell definition patterns are difficult to debug and almost impossible to make portable. If you have access to lava-test-case, there is no need to also use a pattern because you already have a shell on the DUT which is capable of much better pattern matching and parsing. Start by copying the relevant part of the test output and see how parsing can be improved:

• Is any kind of pattern needed at all? Can the process generating the output be called by a script which already understands the output?
• If you do need a pattern, put the pattern handling inside the test shell definition scripts and use copies of different sections of output to debug the pattern matching before submitting anything to LAVA.

Do not lock yourself out of your tests

1. Do not make your test code depend on the LAVA infrastructure any more than is necessary for automation. Make sure you can always run your tests by downloading the test code to a target device using a clean environment, installing its dependencies (the test code itself could do this), and running a single script. Emulation can be used in most cases where access to the device is difficult. Even if the values in the output change, the format of the output from the underlying test operation should remain the same, allowing a single script to parse the output in LAVA and in local testing.
2. Make the LAVA-specific part as small as possible, just enough to, for example, gather any inputs that you get via LAVA, call the main test program, and translate your regular output into ways to tell lava how the test went (if needed).
3. Standard test jobs are intended to showcase the design of the test job, not the test definitions. These test definitions tend to be very simplistic and are not intended to be examples of how to write test definitions, just how to prepare test jobs.

Rely less on install: steps¶

To make your test portable, the goal of the install block of any test definition should be to get the raw LAVA environment up to the point where a developer would be ready to start test-specific operations. For example, installing package dependencies. Putting the operations into the run: steps also means that the test writer can report results from these operations.

Whilst compatibility with V1 has been retained in most areas of the test shell, there can be differences in how the install steps behave between V1 and V2. Once V1 is removed, other changes are planned for the test shell to make it easier for test writers to create portable tests. It is possible that the install: behavior of the test shell could be restricted at this time.

Consider moving install: git-repos: into a run step or directly into a custom_script along with the other setup (for example, switching branches or compiling the source tree). Then, when debugging the test job, a test writer can setup a similar environment and simply call exactly the same script.

Use different test definitions for different test areas¶

Follow the standard UNIX model of Make each program do one thing well. Make a set of separate test definitions. Each definition should concentrate on one area of functionality and test that one area thoroughly.

Use different jobs for different test environments¶

While it is supported to reboot from one distribution and boot into a different one, the usefulness of this is limited. If the first environment fails, the subsequent tests might not run at all.

Use a limited number of test definitions per job¶

While LAVA tries to ensure that all tests are run, adding more and more test repositories to a single LAVA job increases the risk that one test will fail in a way that prevents the results from all tests being collected.

Overly long sets of test definitions also increase the complexity of the log files, which can make it hard to identify why a particular job failed.

Splitting a large job into smaller chunks also means that the device can run other jobs for other users in between the smaller jobs.

Retain at least some debug output in the final test definitions¶

Information about which commit or version of any third-party code is and will remain useful when debugging failures. When cloning such code, call a script in the code or use the version control tools to output information about the cloned copy. You may want to include the most recent commit message or the current commit hash or version control tag or branch name.

If an item of configuration is important to how the test operates, write a test case or a custom script which reports this information. Even if this only exists in the test job log output, it will still be useful when comparing the log files of other similar jobs.

Mock up the device output to test the scripts¶

Avoid waiting for a device to deploy and boot for each iteration in the development of test support scripts. Copy the output of a working device and use that as the input to the scripts which process the logs to identify results and cut out the noise.

Where possible, include such mock ups as tests which can be run in another CI process, triggered each time the scripts are modified.

Use functional tests to validate common functionality¶

Use the principles of Functional testing of LAVA source code to test common code used by the test jobs. For example, if a shell library is used, ensure that your smoke tests definitions are changed to use the shell library so that all health checks and functional tests provide test coverage for the shell library.

Check for specific support as a test case¶

If a particular package, service, script or utility must exist and / or function for the rest of your test definition to operate, test for this functionality.

Any command executed by lava-test-case <name> --shell will report a test case as pass if that command exits zero and fail if that command exited non-zero. If the command is complex or needs pipes or redirects, create a simple script which returns the exit code of the command.

Check custom scripts for side-effects¶

Subtle bugs can be introduced in custom scripts, so it is important to make the scripts portable so that bugs can be reproduced outside LAVA.

When interacting directly with LAVA, for example calling lava-test-case, it is possible to introduce control flow bugs. These can cause the output of lava-test-case to be received after the end of a test run and this can generate TestError exceptions. This section covers one example when using Python, there may be others.

This example checks for lava-test-case in $PATH to determine whether to use the LAVA helpers.

import os
import subprocess


def _which_check(path, match):
    """
    Simple replacement for the `which` command found on
    Debian based systems. Allows ordinary users to query
    the PATH used at runtime.
    """
    paths = os.environ['PATH'].split(':')
    if os.getuid() != 0:
        # avoid sudo - it may ask for a password on developer systems.
        paths.extend(['/usr/local/sbin', '/usr/sbin', '/sbin'])
    for dirname in paths:
        candidate = os.path.join(dirname, path)
        if match(candidate):
            return candidate
    return None


if _which_check(path='lava-test-case', match=os.path.isfile):
    subprocess.Popen([
         'lava-test-case', 'probe-results', '--result', 'pass',
         '--measurement', str(average), '--units', 'volts'])

The error is in this line:

subprocess.Popen([

Popen calls fork but returns immediately. Unless the script also calls wait, then the output of the subprocess can occur after the above function has returned. It is easy for this to happen at the end of a test definition, leading to intermittent bugs where some tests fail.

The solution is to use the existing subprocess functions which already use wait internally. For lava-test-case, this would be check_call which waits for the process to execute and checks the return value.

The fixed example looks like:

import os
import subprocess


def _which_check(path, match):
    """
    Simple replacement for the `which` command found on
    Debian based systems. Allows ordinary users to query
    the PATH used at runtime.
    """
    paths = os.environ['PATH'].split(':')
    if os.getuid() != 0:
        # avoid sudo - it may ask for a password on developer systems.
        paths.extend(['/usr/local/sbin', '/usr/sbin', '/sbin'])
    for dirname in paths:
        candidate = os.path.join(dirname, path)
        if match(candidate):
            return candidate
    return None


if _which_check(path='lava-test-case', match=os.path.isfile):
    subprocess.check_call([
         'lava-test-case', 'probe-results', '--result', 'pass',
         '--measurement', str(average), '--units', 'volts'])

Call lava-test-raise if setup fails¶

Most test jobs have setup routines which ensure that dependencies are available or that the directory layout is correct etc. In most cases, these routines are called early and a failure in the setup function would undermine all subsequent test operations.

The return code of some operations can be used to trigger an early failure.

run:
   steps:
       - apt-get update -q && lava-test-case "apt-update" --result pass || lava-test-raise "apt-update"

run:
  steps:
    - apt -y install wget
    - wget http://people.linaro.org/~neil.williams/setup-test.sh
    - sh -x setup-test.sh

# saved, committed and pushed as ./testdefs/lava-common

command(){
    if [ -n "$(which lava-test-case || true)" ]; then
        echo $2
        $2 && lava-test-case "$1" --result pass || lava-test-raise "$1"
    else
        echo $2
        $2
    fi
}

command(){
    # setup command - will abort the test job upon failure.
    # expects two quoted arguments
    # $1 - valid lava test case name (no spaces)
    # $2 - the full command line to execute
    # Note: avoid trying to set environment variables.
    # use an explicit export.
    CMD=""
    PREFIX=$1
    shift
    while [ "$1" != "" ]; do
      CMD="${CMD} $1" && shift;
    done;
    if [ -n "$(which lava-test-case || true)" ]; then
        echo "${CMD}"
        $CMD && lava-test-case "${PREFIX}" --result pass || lava-test-raise "${PREFIX}"
    else
        echo "${CMD}"
        $CMD
    fi
}

#!/bin/sh

# saved, committed and pushed as ./testdefs/local-run.sh

. ./lava-common

command 'setup-apt' "apt-get update -q"

run:
    steps:
      ./testdefs/local-run.sh

import os
import subprocess


def _which_check(path, match):
    """
    Simple replacement for the `which` command found on
    Debian based systems. Allows ordinary users to query
    the PATH used at runtime.
    """
    paths = os.environ['PATH'].split(':')
    if os.getuid() != 0:
        # avoid sudo - it may ask for a password on developer systems.
        paths.extend(['/usr/local/sbin', '/usr/sbin', '/sbin'])
    for dirname in paths:
        candidate = os.path.join(dirname, path)
        if match(candidate):
            return candidate
    return None


values = []
# other processing populates the values list
if not values:
    if _which_check(path='lava-test-raise', match=os.path.isfile):
        subprocess.check_call(['lava-test-raise', 'setup failed'])
    else:
        print("setup failed")
    return 1

Control the amount of output from scripts and tools¶

Many tools available in distributions have ways to control the amount of output during operation. A balance is needed and test writers are recommended to check for available support. Wherever possible, use the available options to opt for output intended for log file output rather than your typical terminal.

When writing your own scripts, consider using set -x or wrapping certain blocks with set -x, set +x when using shell scripts. With other languages, use print() and similar functions often, especially where the script uses a conditional that can be affected by parameters from within the test job.

Control the number of test cases reported¶

Creating a lava-test-case involves a database operation on the master. LAVA tries to optimize these calls to allow test jobs to report several tens of thousands of test cases per test job, including supporting streaming of test cases exported through the API. However, there will always be a practical limit to the total number of test cases per test job.

Groups of test cases should be separated into test sets and then into test suites (by using separate LAVA Test Shell Definition paths) to make it easier to find the relevant test case.

When writing the test shell definition, always try to report results on-the-fly instead of waiting until the test operation has written all the data to a file. This insulates you from early failures where the file is not written or cannot be parsed after being written. Wrapper scripts can be used to report LAVA test cases during the creation of the file.