Setting up remote worker¶
Test execution in LAVA is performed by 'lava-worker'. It can run on the same physical hardware as 'lava-server' but also can run separately on different physical host. The latter case is called 'remote worker'. Remote workers can connect to server on local network or using Internet. Connection is established over http/https protocol.
lava-dispatcher settings¶
In order to point lava-worker to the correct server, it needs the following settings:
URL="http://<lava-server-dns>/"
Location of the settings depends on the way lava-dispatcher is started.
When using standalone installation either from sources or from debian package,
settings can be found in /etc/lava-dispatcher/lava-worker
file. In case
docker-compose is used
settings should be updated in .env file:
DC_LAVA_SERVER_HOSTNAME="<lava-server-dns>"
connection encryption¶
It is advised to always encrypt the connection between server and workers. We advice to use https instead of http for worker connection.
http_proxy settings¶
Worker specific http_proxy settings should be defined on the server in
/etc/lava-server/dispatcher.d/<name>/env.yaml
scheduler.workers.set_env
XML-RPC
* POST on /api/v0.2/workers/
REST API
* PUT on /api/v0.2/workers/<hostname>/env
REST API
local devices connected to remote worker¶
All device dictionaries are stored on the server node and passed to workers before starting test jobs. In case both workers and server are administered by the same admin there should be no issues.
However in case remote worker admin is different than server admin, there needs to be coordination of device dictionaries. It is advised that device dictionary files are maintained in version control system. This way bad changes can be reverted quickly and full instance recovery is possible without major problems.
Current APIs allow to upload device dictionaries directly to the server. These
are scheduler.devices.set_dictionary
XML-RPC method and POST request on
/api/v0.2/devices/<hostname>/dictionary
REST API.
Alternatively server admins might decide to use a configuration tool (salt, ansible, etc.) to copy files from version control to the lava-server host.
database entries for device types and devices¶
Although all device specific settings are stored in device dictionary, there is
still a need to create a database entry. This can be done using django admin UI,
XML-RPC API scheduler.devices.add
or issuing POST request on
/api/v0.2/devices/
.
local device types (not yet available in LAVA)¶
Devices on which test jobs are executed should be supported by LAVA. This means that "device type" should be present in the server node. Device type is used to render full device dictionary. There may be cases that remote workers include devices that are not yet supported by LAVA. In such case the following actions should be performed:
- new device type should be added to LAVA code base
- temporarily server node admins might add new device type on their host. This operation is depending on how server configuration is maintained. As with device dictionaries it is advised to use version control.
dispatcher version¶
Currently lava-dispatcher version should be in sync with lava-server version. In case of version mismatch, lava-worker connection will be rejected by lava-server.
upgrades¶
Currently LAVA doesn't support any mechanism for remote dispatcher updates. It is responsibility of admins to perform the updates.
remote dispatcher - host installation¶
The instructions assume installation from debian packages but should also be valid for installation from sources. These steps should result in remote worker registering with LAVA server.
install required packages¶
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys A791358F2E49B100
apt-get update
apt-get install lava-dispatcher lava-dispatcher-host
installing from source¶
lava-dispatcher-host package installs udev rules file. In order to perform this step when installing from source one needs to run:
lava-dispatcher-host rules install
edit /etc/lava-dispatcher/lava-worker and update settings¶
URL="http://<lava-server-dns>/"
start lava-dispatcher¶
systemctl start lava-worker
remote dispatcher - docker installation¶
docker-compose file to setup an instance lava-dispatcher from scratch. In this setup, every service will be running in a separate container.
Usage¶
Requirements¶
In order to use this docker-compose file, you need:
- docker.io
- docker-compose
You can install the dependencies using:
apt install docker.io docker-compose
Installing¶
You just need to fetch the sources:
git clone https://git.lavasoftware.org/lava/pkg/docker-compose
cd docker-compose
Using it¶
Configuration (simple, for QEMU purposes)¶
All configuration is stored in .env
file. Some of the steps are required
whilst others are optional.
- Change DC_LAVA_SERVER_HOSTNAME to
which points to the running LAVA server instance. - Execute
make lava-dispatcher
; at this point multiple containers should be up and running and the worker should connect to the LAVA server instance of your choosing. - Add a new device and set its' device template (alternatively you can update
existing device to use this new worker)
Example QEMU device template:
You can do this via XMLRPC, lavacli or REST API (if using version 2020.01 and higher).
{% extends 'qemu.jinja2' %} {% set mac_addr = 'DF:AD:BE:EF:33:02' %} {% set memory = 1024 %}
- (optional) If the lab where this container runs is behind a proxy or you require any specific worker environment settings, you will need to update the proxy settings by setting the worker environment You can do this via this XMLRPC API call.
Firewall
If the server instance is behind a firewall, you will need to create a port forwarding so that ports 80 and maybe 443 are open to the public.
Configuration (advanced, for physical DUT purposes)¶
If you're setting up a standalone dispatcher container, make sure you go through the above configuration first, it is mandatory for this step. In order to run test jobs on physical devices we will need a couple of additional setup steps:
- PDU control:
- The dispatcher docker container will already download pdu scripts from
lava-lab repo which you can use
in device configuration but if you use custom PDU scripts you need to
provide them and copy them into
dispatcher/power-control
directory; they will be copied into/root/power-control
path in the container. - If you need SSH keys for PDU control, copy the private key to the
dispatcher/ssh
directory and the public key on to the PDU - SSH config - if there's a need for a specific SSH configuration (like
tunnel passthrough, proxy, strict host checking, kexalgorithm etc), create
the config file with relevant settings and copy it into
dispatcher/ssh
dir; it will be copied to/root/.ssh
directory on the dispatcher container. - ser2net config - update
ser2net/ser2net.config
with the corresponding serial port and device settings - Update/add device dictionary with power commands and connection command
- Add dispatcher_ip setting to the dispatcher configuration. Alternatively you can use REST API if you are using version 2020.01 or higher:
dispatcher_ip: <docker host ip address>
- Disable/stop rpcbind service on host machine if it's running - docker service nfs will need port 111 available on the host.
Running¶
In order to start the containers, run:
docker-compose build lava-dispatcher
docker-compose up lava-dispatcher
or, alternatively:
make lava-dispatcher