Replacing VMGroups using Secondary Connections

One of the primary use cases for secondary connections is the ability to watch a job from multiple perspectives. This includes being able to start a task in one job and interact with that task from another job. For example, if one job runs QEMU to start a VM on the host machine, another connection can test that the VM responds on the relevant ports, depending on what is installed inside the image downloaded onto the host machine. More complex jobs could start a daemon in a debugger and connect to the output of the daemon itself in one job and to the output of the debugger in another job. This does not necessarily require the watching job to do anything, the job could just record the output.

The role of a Virtual Machine Group is to arrange a test job such that a host device, which must have virtualization support, boots into a base image and installs a daemon to allow other test jobs to connect - typically this will be Secure Shell connections (ssh).

See also

Considerations with a secondary connection

Once the secondary connections are logged in, any program accessible in the base image can be used by the secondary connection. As highlighted in the link above, tasks involving package installation or other system-wide operations are best done in the test job managing the host device and preferably before the secondary connection jobs start. So the install: deps: of all jobs in the group should all be collated into the test definition(s) of the host role.

The secondary connection support performs all the synchronization steps that are required prior to the test shell starting in each of the secondary test jobs. The MultiNode API remains available for any synchronization requirements during the test shell operation. Importantly the test shell running on the host device should wait for a lava-sync before completing the test shell definition. Otherwise the secondary connection jobs may find that the underlying system suddenly goes away unexpectedly.

With all of that in place, the actual operation of a test involving multiple virtual machines is little different to a test involving a daemon or long running task. Something in the secondary job test shell needs to start the process and something needs to end the process so that the final sync can occur. If the task is not capable of terminating itself, it will be necessary for one test shell (often the host role on the host device) to be able to terminate it or the test shell will not complete. The choice of how to do this is left to the test writer.

The command line to use to start the VM(s) on the host device is controlled entirely by the test specification. This is different to the old deprecated vmgroups support. Image files and other components can be present in the image deployed to the host device or downloaded by the host device or after the secondary connection is logged in. The delayed start requirements are already covered by the secondary connections, so the test shell can start the VM immediately, if desired, or do any preliminary tests first.

Structure of an example job for a mustang

This job uses an nfsrootfs for the host device on a mustang (using the U-Boot pipeline support for the mustang) as the host role. The NFS is a base Debian Jessie arm64, so the initial lava test shell operation on the host is to install openssh-server and then use the lava-echo-ipv4 helper to declare the IPv4 address of the host machine.

The secondary connection support picks up the IP address in the pipeline actions, so the id of the message needs to be declared to the relevant action. The test shell sends:

          - ifconfig

Then it tells the secondary connections to start:

          - lava-send ipv4 ipaddr=$(lava-echo-ipv4 eth0)

This tells the dispatcher for the guest role to start the deployment. The test definition files for secondary connection jobs are copied onto the host device immediately before the connection is made, then the contents are unpacked to be able to run the test shell. The tarball will exist on the host device as /<job_id>-overlay-<level>.tar.gz. For job ID 74585 and level 1.1.5, this would result in /74585-overlay-1.1.5.tar.gz.

In this example job, the host test shell does nothing except wait for the clients to complete their own tests before proceeding to run a final test shell of smoke tests.

The receiving action is declared as:

    protocols:
      lava-multinode:
      - action: prepare-scp-overlay
        request: lava-wait
        # messageID matches hostID
        messageID: ipv4
        message:
          # the key of the message matches value of the host_key
          # the value of the message gets substituted
          ipaddr: $ipaddr

Note

the messageID specified to lava-send (ipv4), is also the messageID specified to the prepare-scp-overlay action within the pipeline. In addition, the content of the sent message is declared. lava-send uses the syntax key=value, the YAML uses the equivalent syntax of key: value. As the value will be substituted with the real IP address, the value in the YAML is marked as replaceable using the $ prefix.

The message parameters are passed to the boot action of the guest role so that the details can be retrieved:

Finally, the jobs with the guest role are booted - this establishes the connection between the dispatcher and the host device using ssh. Once logged in, each job completes the boot stage and starts the test shell for that job.

- boot:
    role:
    - guest
    method: ssh
    connection: ssh
    prompts:
    - 'root@linaro-developer:'
    parameters:
      hostID: ipv4  # messageID

Notes

  • Starting the VM(s) is for the test writer to implement, depending on the support required and the objectives of the test. In the example below, the host device simply runs the smoke tests definition in the position where images could be downloaded and QEMU started.

  • Use inlines - this example keeps all of the MultiNode API calls to the inline definitions. This is a recommended practice and future developments will make it easier to match up the synchronization calls from inline definitions. So, to adapt this job to do other tasks while the secondary connections jobs are running those test shells, move the final lava-sync clients to another inline definition and do the other calls in between.

    See also

    Running operations inside the guest VM

  • Completion - It is useful for the host device test shell to do something after completing the final lava-sync or the host device may complete the test shell before the secondary connections can logout correctly, resulting in the secondary connection jobs being incomplete. A final test definition of smoke tests or other quick checks could be useful.

View or Download mustang-ssh-guest.yaml

  1job_name: ssh guest with standard images
  2
  3timeouts:
  4  job:
  5    minutes: 60
  6  action:
  7    minutes: 20
  8  connection:
  9    minutes: 2
 10
 11priority: medium
 12visibility: public
 13
 14# PROTOCOLS_BLOCK
 15protocols:
 16  lava-multinode:
 17    roles:
 18      guest:
 19        connection: ssh
 20        count: 3
 21        expect_role: host
 22        host_role: host
 23        request: lava-start
 24        timeout:
 25          minutes: 15
 26      host:
 27        count: 1
 28        device_type: mustang
 29        timeout:
 30          minutes: 10
 31
 32metadata:
 33  docs-source: actions-deploy-to-ssh
 34  docs-filename: examples/test-jobs/mustang-ssh-guest.yaml
 35  build-readme: http://images.validation.linaro.org/snapshots.linaro.org/components/lava/standard/debian/stretch/arm64/3/debian-arm64-readme.html
 36  build-console: https://ci.linaro.org/view/lava-ci/job/lava-debian-stretch-arm64/3/console
 37  build-script: http://images.validation.linaro.org/snapshots.linaro.org/components/lava/standard/debian/stretch/arm64/3/build-foreign-nfs.sh
 38
 39actions:
 40- deploy:
 41    role:
 42      - host
 43    authorize: ssh
 44    kernel:
 45      url: http://images.validation.linaro.org/snapshots.linaro.org/components/lava/standard/debian/stretch/arm64/3/vmlinuz-4.9.0-2-arm64
 46      type: zimage
 47    modules:
 48      url: http://images.validation.linaro.org/snapshots.linaro.org/components/lava/standard/debian/stretch/arm64/3/modules.tar.gz
 49      compression: gz
 50    nfsrootfs:
 51      url: http://images.validation.linaro.org/snapshots.linaro.org/components/lava/standard/debian/stretch/arm64/3/stretch-arm64-nfs.tar.gz
 52      compression: gz
 53    ramdisk:
 54      url: http://images.validation.linaro.org/snapshots.linaro.org/components/lava/standard/debian/stretch/arm64/3/initrd.img-4.9.0-2-arm64
 55      compression: gz
 56    timeout:
 57      minutes: 12
 58    to: tftp
 59
 60# DEPLOY_SSH_BLOCK
 61- deploy:
 62    role:
 63    - guest
 64    connection: ssh
 65    protocols:
 66      lava-multinode:
 67      - action: prepare-scp-overlay
 68        request: lava-wait
 69        # messageID matches hostID
 70        messageID: ipv4
 71        message:
 72          # the key of the message matches value of the host_key
 73          # the value of the message gets substituted
 74          ipaddr: $ipaddr
 75        timeout:  # delay_start timeout
 76          minutes: 21
 77    timeout:
 78      minutes: 22
 79    to: ssh
 80
 81# BOOT_HOST_BLOCK
 82- boot:
 83    role:
 84    - host
 85    auto_login:
 86      login_prompt: 'login:'
 87      username: root
 88    commands: nfs
 89    method: grub-efi
 90    prompts:
 91    - 'root@stretch:'
 92    timeout:
 93      minutes: 5
 94
 95- boot:
 96    role:
 97    - guest
 98    method: ssh
 99    connection: ssh
100    prompts:
101    - 'root@linaro-developer:'
102    parameters:
103      hostID: ipv4  # messageID
104      host_key: ipaddr  # message key
105    timeout:
106      minutes: 23
107
108- test:
109    role:
110    - host
111    definitions:
112    - from: inline
113      name: ssh-inline
114      path: inline/ssh-install.yaml
115      repository:
116        install:
117          deps:
118          - openssh-server
119        metadata:
120          description: install step
121          format: Lava-Test Test Definition 1.0
122          name: install-ssh
123          os:
124          - debian
125          scope:
126          - functional
127        install:
128          deps:
129          - openssh-server
130          - net-tools
131          - iproute2
132        run:
133          steps:
134          - ls -al /root/.ssh/
135          - ifconfig
136          - lava-send ipv4 ipaddr=$(lava-echo-ipv4 eth0)
137          - lava-send lava_start
138          - lava-sync clients
139    - repository: http://git.linaro.org/lava-team/lava-functional-tests.git
140      from: git
141      path: lava-test-shell/smoke-tests-basic.yaml
142      name: smoke-tests
143    - from: git
144      repository: http://git.linaro.org/lava-team/lava-functional-tests.git
145      path: lava-test-shell/params/nfs.yaml
146      name: nfs-dns
147    - repository: http://git.linaro.org/lava-team/lava-functional-tests.git
148      from: git
149      path: lava-test-shell/single-node/singlenode03.yaml
150      name: singlenode-advanced
151    timeout:
152      minutes: 30
153
154- test:
155    role:
156    - guest
157    definitions:
158    - repository: http://git.linaro.org/lava-team/lava-functional-tests.git
159      from: git
160      path: lava-test-shell/smoke-tests-basic.yaml
161      name: pre-smoke-tests
162    - from: inline
163      name: ssh-client
164      path: inline/ssh-client.yaml
165      repository:
166        metadata:
167          description: client starts
168          format: Lava-Test Test Definition 1.0
169          name: client-ssh
170          os:
171          - debian
172          scope:
173          - functional
174        run:
175          steps:
176          - lava-sync clients
177          - tar -tzf /$(pwd|awk -F'-' '{print $2}'|awk -F'/' '{print $1}')-overlay*.tar.gz
178    - repository: http://git.linaro.org/lava-team/lava-functional-tests.git
179      from: git
180      path: lava-test-shell/smoke-tests-basic.yaml
181      name: post-smoke-tests
182    timeout:
183      minutes: 5

Running operations inside the guest VM

A guest VM started by running QEMU on the command line is not a LAVA environment (unless the test writer deliberately copies files into it from another job), so it will not run a lava test shell by default. Tasks can be executed within the VM from any of the other jobs running on the host device, dependent on support provided by the test writer.

Remember, although LAVA tries to stay out of the way of how the test runs once the secondary connection has logged in, there are some things test writers need to consider to be able to automate tests like these.

  1. If you start QEMU with the -nographics option rather than as a daemon, the secondary connection gets connected to the console of that VM at the point within the test shell where the call to QEMU is made.

  2. Make sure you know if the image being used has a serial console configured.

  3. If the image being launched stops at a login: prompt, the test definition will need to handle that prompt or log in to the VM in some other way. e.g. by having one of the other secondary connections set up a configuration to use ssh to log in to the VM - the keys needed for this login will need to be handled by the test writer.

  4. The test shell will pause, waiting for QEMU to return, unless QEMU is configured to do otherwise or a wrapper like pexpect is used. (The LAVA QEMU devices run a QEMU command using pexpect.spawn but this is not necessarily suitable for test jobs.)

  5. If the VM is started as a daemon, then the test shell will need to have a way of monitoring when the VM is ready and then connect to the VM, as appropriate.

Note

The lava-start API call only acts once - i.e. the host role starts, then the other jobs wait until lava-start is sent - at which point these jobs will download any test shell definitions and try to connect to the IP address declared. It is better to have a synchronization which the test writer controls, after all the jobs have connected to the host device.