Versioned test definitions¶

If your test definition is not part of a git repository then it is must include a version parameter in the metadata section like in the following example.

metadata:
  name: passfail
  format: "Lava-Test-Shell Test Definition 1.0"
  description: "A simple passfail test for demo."
  version: "1.0"

How a lava test shell is run¶

A lava-test-shell is run by:

• building the test definition into a shell script.

  Note

  This shell script will include set -e, so a failing step will abort the entire test run. If you need to specify a step that might fail, finish the command with || true to make that failure not abort the test run.

• copying an overlay onto the device. The overlay contains both the test script and the rest of the LAVA Test Helpers. and setup code to run the test script when the device boots

• booting the device and letting the test run

• retrieving the output from the device and turning it into a test result

• run subsequent test definitions, if any.

Writing a test for lava-test-shell¶

For the majority of cases, the above approach is the easiest thing to do: write shell code that outputs “test-case-id: result” for each test case you are interested in. See the Test Developer Guide:

• Writing Tests.
• Writing a Lava-Test Test Definition 1.0.

When a test runs, $PATH is arranged so that some LAVA-specific utilities are available:

• lava-test-case
• lava-background-process-start
• lava-background-process-stop

lava-test-case¶

lava-test-case records the results of a single test case. For example:

steps:
  - "lava-test-case simpletestcase --result pass"
  - "lava-test-case fail-test --shell false"

It has two forms. One takes arguments to describe the outcome of the test case. The other takes the shell command to run, and the exit code of this shell command is used to produce the test result.

Both forms take the name of the testcase as the first argument.

run:
  steps:
    - "lava-test-case simpletestcase --result pass"
    - "lava-test-case bottle-count --result pass --measurement 99 --units bottles"

#!/usr/bin/env python

from subprocess import call


def test_case():
    """
    Calculate something based on a test
    and return the data
    """
    return {"name": "test-rate", "result": "pass",
        "units": "Mb/s", "measurement": 4.23}


def main():
    data = test_case()
    call(
        ['lava-test-case',
         data['name'],
         '--result', data['result'],
         '--measurement', data['measurement'],
         '--units', data['units']])
    return 0

if __name__ == '__main__':
    main()

run:
  steps:
    - "lava-test-case fail-test --shell false"
    - "lava-test-case pass-test --shell true"

Using parameters in the job to update the definition¶

Parameters used in the test definition YAML can be controlled from the YAML job file. See the following YAML test definition for ean example of how it works.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

metadata:
    format: Lava-Test Test Definition 1.0
    name: params-test
    description: "test commands for Linux POSIX images with params"
    version: "1.0"
    maintainer:
        - neil.williams@linaro.org

params:
    VARIABLE_NAME_1: value_1
    VARIABLE_NAME_2: value_2

run:
    steps:
        - lava-test-case test3 --result pass
        - lava-test-case test4 --result fail
        - lava-test-case test5 --result pass --measurement 99 --units bottles
        - lava-test-case test6 --result fail --measurement 0 --units mugs
        - echo $VARIABLE_NAME_1
        - echo $VARIABLE_NAME_2
        - echo $SPACED_VAR
        - echo $PUB_KEY

Download or view params.yaml: examples/test-definitions/params.yaml

This Lava-Test Test Definition 1.0 can be used in a simple QEMU test job:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

- test:
    timeout:
      minutes: 5
    definitions:
    - repository: https://git.lavasoftware.org/lava/functional-tests.git
      from: git
      path: posix/parameters.yaml
      name: parse-params
      parameters:
        VARIABLE_NAME_1: "first variable value"
        VARIABLE_NAME_1: "first variable value"

Download or view the test job: examples/test-jobs/qemu-stretch-params.yaml

lava-background-process-start¶

This starts a process in the background, for example:

steps:
  - lava-background-process-start MEM --cmd "free -m | grep Mem | awk '{print $3}' >> /tmp/memusage"
  - lava-background-process-start CPU --cmd "grep 'cpu ' /proc/stat"
  - uname -a
  - lava-background-process-stop CPU
  - lava-background-process-stop MEM --attach /tmp/memusage text/plain --attach /proc/meminfo application/octet-stream

The arguments are:

1. The name that is used to identify the process later in lava-background-process-stop
2. The command line for the process to be run in the background

See Recording test case data.

lava-background-process-stop¶

This stops a process previously started in the background using lava-background-process-start. The user can attach files to the test run if there is a need.

For example:

steps:
  - lava-background-process-start MEM --cmd "free -m | grep Mem | awk '{print $3}' >> /tmp/memusage"
  - lava-background-process-start CPU --cmd "grep 'cpu ' /proc/stat"
  - uname -a
  - lava-background-process-stop CPU
  - lava-background-process-stop MEM --attach /tmp/memusage text/plain --attach /proc/meminfo application/octet-stream

The arguments are:

1. The name that was specified in lava-background-process-start
2. (optional) An indication that you want to attach file(s) to the test run with specified mime type. See Recording test case data.

Handling test attachments¶

Handling of attachments is in the control of the test writer. A separate publishing location can be configured or text based data is simply to output the contents into the log file.

Handling Dependencies (Debian)¶

If your test requires some packages to be installed before its run it can express that in the install section with:

install:
    deps:
        - linux-libc-dev
        - build-essential

Adding Git Repositories¶

If the test needs code from a shared repository, the action can clone this data on your behalf with:

install:
    git-repos:
        - git://git.linaro.org/people/davelong/lt_ti_lava.git

run:
    steps:
        - cd lt_ti_lava
        - echo "now in the git cloned directory"

install:
    git-repos:
        - url: https://git.lavasoftware.org/lava/lava.git
          skip_by_default: False
        - url: https://git.lavasoftware.org/lava/lava.git
          destination:  lava-d-r
          branch:       release
        - url: https://git.lavasoftware.org/lava/lava.git
          destination:  lava-d-s
          branch:       staging

Install Steps¶

Before the test shell code is executed, it will optionally do some install work if needed. For example if you needed to build some code from a git repo you could do:

install:
    git-repos:
        - git://git.linaro.org/people/davelong/lt_ti_lava.git

    steps:
        - cd lt_ti_lava
        - make

Parse patterns¶

You may need to incorporate an existing test that doesn’t output results in in the required pass/fail/skip/unknown format required by LAVA. The parse section has a fixup mechanism that can help:

parse:
    pattern: "(?P<test_case_id>.*-*)\\s+:\\s+(?P<result>(PASS|FAIL))"
    fixupdict:
        PASS: pass
        FAIL: fail

Single quote example:

parse:
    pattern: '(?P<test_case_id>.*-*)\s+:\s+(?P<result>(PASS|FAIL))'
    fixupdict:
        PASS: pass
        FAIL: fail