Administering LAVA using Docker¶

Prerequisites¶

For buster or later, use docker.io from Debian https://tracker.debian.org/pkg/docker.io
For stretch, use docker-ce from Docker. https://docs.docker.com/install/linux/docker-ce/debian/
add yourself to the docker group to avoid the need for sudo.
Ensure that your docker installation has created a suitable network bridge (often called docker0).

See also

Docker Networking
Not all docker images are available for all architectures. LAVA has support for amd64 and arm64 docker images.

Using LAVA with Docker is also a regular topic of discussion on the lava-devel mailing list.

https://docs.docker.com/engine/reference/commandline/run/

Docker Networking¶

https://docs.docker.com/engine/reference/commandline/network_create/

LAVA typically uses network connections not just for the communication between master and worker but also to control the DUT. It is important to have reliable networking for the docker components.

If you have not done so already, create a new network for docker to use, based on the docker0 bridge which should already have been created by docker. The subnet you choose is up to you.

$ docker network create --subnet=172.18.0.0/16 dockernet
f9ca89ac46ab97e5d905a0858a03f8983941e8f4583b71e1996bbaa0d57d3138

When you start containers, you can specify that network name and then provide a static IPv4 address for the container. This is particularly important if you want to start a lava-server container and then connect a worker. The worker can be another docker container or it can use lava-dispatcher installed directly on this or another system. You will need to make sure that the docker network is accessible to any worker you want to attach to the new container. Depending on local configuration, you may also need to configure the ZMQ ports on your lava-server container. See the docker documentation for how to forward ports to different numbers outside the container.

$ docker run --net dockernet --ip 172.18.0.5 -it hub.lavasoftware.org/lava/lava/lava-dispatcher:2018.10

This IP address (or a hostname if you configure local DNS appropriately) can then be used in commands to docker run to start a lava-dispatcher container by updating the MASTER_URL and LOGGER_URL variables.

...
-e "LOGGER_URL=tcp://172.18.0.5:5555" \
-e "MASTER_URL=tcp://172.18.0.5:5556" \
...

Official LAVA Software Docker images¶

Official LAVA Software Docker images are available via hub.lavasoftware.org and hub.docker.com. In each case, two images are built at the same version: lava-dispatcher and lava-server, each image is built for two architectures, amd64 and aarch64.

Daily builds¶

hub.lavasoftware.org hosts CI images regularly built from the master branch of lava. Images are listed in GitLab:

https://git.lavasoftware.org/lava/lava/container_registry

Official LAVA Releases using Docker¶

The lavasoftware organization on hub.docker.com hosts releases of LAVA (https://hub.docker.com/u/lavasoftware/) by retagging images from hub.lavasoftware.org and pushing to hub.docker.com . Users are free to use either hub.

Note

Due to naming conventions on hub.docker.com, the architecture is included in the image name amd64-lava-server when tagged for hub.docker.com.

lava-dispatcher¶

https://hub.docker.com/r/lavasoftware/lava-dispatcher

docker pull lavasoftware/lava-dispatcher:2019.01

docker pull hub.lavasoftware.org/lava/lava/lava-dispatcher:2019.01

lava-server¶

https://hub.docker.com/r/lavasoftware/lava-server/

docker pull lavasoftware/lava-server:2019.01

docker pull hub.lavasoftware.org/lava/lava/lava-server:2019.01

Command lines¶

The use of docker with LAVA is an active area of development, including how to configure containers for a variety of situations and how to manage a LAVA lab where docker is in use. If you are doing work in this area, please subscribe to the lava-devel mailing list and ask for advice on how to use LAVA and docker for your use case.

POSIX shell¶

Command lines get long, so use wrapper scripts, e.g.:

#!/bin/sh
set -e
set -x

docker run \
-e "DISPATCHER_HOSTNAME=--hostname=calvin-2018.7-88" \
-e "LOGGER_URL=tcp://calvin:5555" \
-e "MASTER_URL=tcp://calvin:5556" \
--name calvin-docker-88-3 \
hub.lavasoftware.org/lava/lava/lava-dispatcher/master:2018.7-88-ga7b7939dd

Python¶

If you are using docker for more than a few test containers, you will probably find the Python docker SDK library very useful.

Debian (stretch-backports and later) apt install python3-docker https://packages.debian.org/unstable/python3-docker - If you install the full lava set on Debian Buster or newer, python3-docker and docker.io will be installed by the lava metapackage.
GitHub https://github.com/docker/docker-py
Documentation: https://docker-py.readthedocs.io/en/stable/

The Python Docker SDK lets you do anything the docker command does, but from within Python apps – run containers, manage containers, manage Swarms, etc.

import docker
client = docker.from_env()
container_id = client.containers.run("debian", detach=True)

Supporting encryption¶

Always use encryption to any master outside your local network. Create a docker volume to act as a fileshare, mounting the specified directory from the host machine inside the docker container at the specified location to exchange files from the host to the container and vice versa:

-v $PWD/my-certificates.d:/etc/lava-dispatcher/certificates.d/

Then use these certificates in the commands:

-e ENCRYPT="--encrypt" \
-e MASTER_CERT='/etc/lava-dispatcher/certificates.d/master.key' \
-e SLAVE_CERT='/etc/lava-dispatcher/certificates.d/docker-slave-1.key_secret'

See also

lava-dispatcher docker images - part 2 - note that the options changed since this content was written.

lava-dispatcher in docker¶

The official LAVA Software docker images for lava-dispatcher do not include details like ser2net configuration or pdudaemon or other remote power control scripts. These will need to be added according to your local lab configuration. Depending on the size of your lab, you may choose to use a docker volume or docker build to create one or more customized docker images based on the official images.

Mocking up LXC inside docker¶

LXC cannot be installed/used inside a Docker container and the Docker container can replace the need for the LXC. This has the useful advantages that specialized tools which need to be isolated inside an LXC can be pre-installed in a docker container instead of needing to be installed or compiled within the LXC.

However, there are also disadvantages:

The Docker is persistent - currently, lava-slave and lava-run need to be inside the container, so the next test job for that slave picks up the changes to the docker from this test job.
The test job would need modification to not call LXC.

Work is underway to solve the persistence problem. In the meantime, it is possible to run test jobs using Docker if the persistence is handled correctly but this is usually only practical for single-user developer instances.

lava-lxc-mocker exists to solve the second problem. By mocking up the calls to lxc-* utilities, lava-lxc-mocker allows the same test job to be run on a device managed by a lava-slave in Docker as on a device managed by a lava-slave running on bare metal.

lava-lxc-mocker is pre-installed in all Official LAVA Software Docker images.

lava-server in docker¶

The official LAVA Software docker images for lava-server currently include PostgreSQL. Work is planned to use an external PostgreSQL.

See also

Superusers

LAVA Coordinator¶

lava-coordinator is neither installed nor configured in any official LAVA Software docker image. Therefore, a worker running from one of these images will not have the configuration file /etc/lava-coordinator/lava-coordinator.conf to use lava-coordinator, so cannot run MultiNode LAVA test jobs. The configuration file would need to be provided (configured for an external coordinator installed using packages), either using a docker volume used as a fileshare or by a modification to the docker image for lava-dispatcher.

Work is planned to refactor lava-coordinator to not require external configuration or packaging.

CI images¶

lava/lava/lava-dispatcher/master on hub.lavasoftware.org contains images like 2018.7-101-g5987db8b5

See also

Official LAVA Releases using Docker

lava-dispatcher¶

This example runs a new worker for an existing master which can be:

on the same machine but installed from packages, not docker
on a different machine and accessible through DNS

In either case, the machine running lava-server is accessible on the network as calvin. (Replace this hostname with your local machine hostname.)

To run both master and worker on a single machine, both using docker, see Running lava-server & lava-dispatcher together.

DISPATCHER_HOSTNAME=--hostname=calvin-2018.7-88

/usr/bin/lava-slave --level $LOGLEVEL --log-file $LOGFILE --master $MASTER_URL --socket-addr $LOGGER_URL $IPV6 $ENCRYPT $MASTER_CERT $SLAVE_CERT $DISPATCHER_HOSTNAME

$ docker run -e "DISPATCHER_HOSTNAME=--hostname=calvin-2018.7-88" -e "LOGGER_URL=tcp://calvin:5555" -e "MASTER_URL=tcp://calvin:5556"  --name calvin-docker-88-4  hub.lavasoftware.org/lava/lava/lava-dispatcher/master:2018.7-88-ga7b7939dd
2018-10-03 15:08:32,852    INFO [INIT] LAVA slave has started.
2018-10-03 15:08:32,852    INFO [INIT] Using protocol version 3
2018-10-03 15:08:32,853   DEBUG [INIT] Connection is not encrypted
2018-10-03 15:08:32,965    INFO [BTSP] Connecting to master [tcp://calvin:5556] as <calvin-2018.7-88>
2018-10-03 15:08:32,965    INFO [BTSP] Greeting the master [tcp://calvin:5556] => 'HELLO'
2018-10-03 15:08:32,966    INFO [BTSP] Connection with master [tcp://calvin:5556] established
2018-10-03 15:08:32,966    INFO Master is ONLINE
2018-10-03 15:08:37,971   DEBUG PING => master (last message 5s ago)
2018-10-03 15:08:37,973   DEBUG master => PONG(20)

If you make mistakes, set the worker to Retired in the Django admin interface and use docker rm <name> to allow you to re-use the same container with different arguments next time.

lava-server¶

$ docker run --net dockernet --ip 172.18.0.5 -it hub.lavasoftware.org/lava/lava/lava-server/master:2018.7-88-ga7b7939dd

Note

the dockernet docker network needs to already exist and is just an example name - choose your own name according to your own preferences. See https://docs.docker.com/network/bridge/#differences-between-user-defined-bridges-and-the-default-bridge

See also

Docker Networking

Superusers¶

There is no superuser in the lava-server docker container, admins need to login to the container and create an initial superuser:

$ docker exec -it a936cc14b913 lava-server manage users add --staff --superuser --email <EMAIL> --passwd <PASSWORD> <USERNAME>

Then this user can login through the normal UI and create Authentication Tokens.

Running lava-server & lava-dispatcher together¶

The worker must be on the same docker network as the master because docker only exposes the master ports to that network.

Containers connected to the same user-defined bridge network automatically expose all ports to each other, and no ports to the outside world. This allows containerized applications to communicate with each other easily, without accidentally opening access to the outside world.

So to run a worker in docker to work with a master in docker on the same machine, the worker must be given the --net dockernet option.

Depending on the tasks, you should also assign an IP address to the worker, on the same docker network.

$ docker run --net dockernet --ip 172.18.0.6 ....

(This is why docker start up scripts are going to be so useful.)

See also

Docker Networking

Separate services in separate Docker containers¶

Work is beginning to extend the Docker support to have different parts of LAVA in different containers. Some parts of this are easier to implement than others, so the support will arrive in stages.

See also

Configuring the LAVA UI