Adding new device types¶

Device Requirements¶

The LAVA software and lab teams have built up a set of guidelines relating to the integration of new device-types. The further a device deviates from one or more of these guidelines, the harder it will become to automate such a device. Always remember that the way that the device is supported must scale to large labs which already contain a range of other devices, each with their own issues. It is not acceptable to add a new device-type which is incompatible with devices which are already supported or which imposes restrictions on how many devices of any type can be used in any one lab.

The guidelines only consider a limited number of possible problems with device integration. The guidelines are written using our experiences of a variety of poorly behaving devices over years of development of automation software like LAVA. Depending on local admins, some labs can cope with hardware which does not comply with all guidelines, particularly if the devices are not being used at scale. However, the more devices are deployed in any lab, the more it will be necessary for every device to fully comply or such labs will quickly deteriorate, generating unreliable results.

These guidelines describe device behavior as a whole. This is a combination of the device hardware and the firmware. Some devices support replacing the firmware. Sometimes this can aid automation, sometimes is can cause more problems and complexity.

Device integration issues are often invisible when testing with a single device attached to a single developer machine, so the single device implementation must be proven to be reliable and reproducible before starting to add more devices. For best results, only ever change one thing at a time.

It is not possible to automate every piece of hardware, there are a number of critical limitations.

Integration process¶

To add support for a new device type, a certain amount of development and testing will be required.

For some new device types, only a new device type jinja2 template will be required. Every new template requires testing and a certain amount of debugging. Device type templates need to be considered as code, not only configuration. Some familiarity with how to debug a LAVA instance will be necessary.

For other device types, new dispatcher Action classes and new or modified strategy classes will be needed. This typically involves a lot of development time - make sure that you Contributing Upstream so that your local changes do not break when you next upgrade your LAVA instance(s).

In addition, every new device type will need to be tested on a local LAVA instance, so an amount of LAVA administration work will be necessary.

It is strongly recommended that everyone who starts work to integrate a new device type into LAVA is already familiar with administering their own LAVA instance and has submitted dozens of LAVA test jobs on at least two different device types already known to work in LAVA V2. In most cases, a development instance will be needed as well, so some familiarity with installing and upgrading a LAVA instance is also recommended.

This means that developers adding new device types should already be familiar with:

• Pre-requisites to start with development
• Device type templates
• Developing using device-type templates
• Testing the design
• Administrator triage
• Where to find debug information
• Creating a device dictionary for the device
• Writing Tests
• Installing on a Debian system
• Setting up a LAVA instance
• Gold standard test jobs
• Debugging LAVA test failures
• Administrator debugging
• Unit-tests
• Hidden assumptions in the manual operations

In addition, some device types will require the developer to also be familiar with:

• Adding new classes
• Using strategy classes
• Contributing Upstream - maintaining new dispatcher classes without upstream support is not recommended. LAVA development moves relatively quickly.
• LAVA schema - if your new device type needs changes to the test job submission schema.
• Deploying test images using LXC
• Using the LXC protocol to support Android
• Debugging MultiNode tests

Find a similar existing device type¶

There are a number of places to check for similar types of device which are already supported in LAVA V2.

1. https://master.lavasoftware.org/scheduler/
2. https://staging.validation.linaro.org/scheduler/
3. https://validation.linaro.org/scheduler/
4. https://lng.validation.linaro.org/scheduler/
5. https://git.lavasoftware.org/lava/lava/tree/master/lava_scheduler_app/tests/device-types

Check for:

• similar bootloader
• similar deployment type
• similar deployment or boot process
• similar sequence of boot steps

If you do not find something similar, we strongly recommend that you stop here and talk to us before doing anything else. Be clear about exactly what kind of device you are trying to integrate. Include details of exactly how the device currently boots and exactly how new files are deployed to the device. Do not resort to simplistic testing.

Extend from an existing device type template¶

All new device type templates need to extend 'base.jinja2' but there are also other base templates which simplify the process for certain bootloaders. For example, all new U-Boot device type templates should extend 'base-uboot.jinja2. Many new fastboot device type templates can extend 'base-fastboot.jinja2. Avoid directly extending any of the templates which do not have the base prefix - instead copy the existing template for your new device type. When this template is contributed upstream, a new base template can be considered as part of the review process.

Extend the template unit tests¶

All device type template files in lava_scheduler_app/tests/device-types will be checked for simple YAML validity by the test_all_templates unit test. However, a dedicated unit test is recommended for all but the simplest of new device type templates. At the very least, having a unit test for your new device type template will assist in debugging why the test job does not run to completion. The full device configuration can be output as part of running the unit test by changing the debug value to True at the top of the TestTemplates class in test_templates.py.

Add your new device-type template to lava_scheduler_app/tests/device-types. Edit lava_scheduler_app/tests/test_templates.py and add a new unit test for your device-type based on one of the existing test functions. Create a dummy device dictionary as a data string and ensure that the combination of the template and the dictionary creates a valid device. This can be as simple as:

   def test_pixel_template(self):
       self.assertTrue(self.validate_data('staging-pixel-01', """{% extends 'pixel.jinja2' %}
{% set adb_serial_number = 'FDAC1231DAD' %}
{% set fastboot_serial_number = 'FDAC1231DAD' %}
{% set device_info = [{'board_id': 'FDAC1231DAD'}] %}
"""))

In many cases, some of the default values in the base template will need to be altered for your new device-type. For example:

{% set boot_character_delay = 150 %}

If the value may also need to be extended for some devices of this device type, you should provide the new value as a default in the template so that a device dictionary can set an override:

{% set baud_rate = baud_rate | default(115200) %}

{% extends 'base.jinja2' %}
{% set boot_character_delay = 150 %}
{% set console_device = console_device | default('ttyAMA0') %}
{% set baud_rate = baud_rate | default(115200) %}

{% set base_nfsroot_args = nfsroot_args | default(base_nfsroot_args) -%}
{% set kernel_args = kernel_args | default('acpi=force') %}

{% block body %}

Every time you make a change to the new template in lava_scheduler_app/tests/device-types, re-run the specific unit test for your new device type. For example, a new unit test function defined as test_foobar_template can be run without running the rest of the unit tests:

$ python3 -m unittest -vcf lava_scheduler_app.tests.test_templates.TestTemplates.test_foobar_template

Remember that device type templates are not just configuration files - the templates are processed as source code at runtime and can use various types of logic to substitute the correct variables and omit other variables. Always make your changes in lava_scheduler_app/tests/device-types and always run the unit test to ensure that changes to the template continue to produce a valid device configuration after each change.

Only when the unit test passes should the new device type template be copied to /etc/lava-server/dispatcher-config/device-types/. If the scheduler tries to assign a test job to a device using this template, a check will be made to ensure that the output of the template and the device dictionary is valid. If that check fails, the test job will not start and the failure will be logged:

[WARNING] [lava-master] [9] Refusing to reserve for broken V2 device intel-smecher

This message indicates that test job ID 9 will never start to run until the device dictionary and the device type template for the device intel-smecher are fixed so that the output is valid. It is common for the rendering of new device type templates to cause subtle YAML syntax errors. It is also common for the output to be valid YAML but not valid device configuration. The unit test must check for a valid device configuration, not simply valid YAML. In addition, whenever it is imperative that a certain value is overridden in the device type template compared to the base template, the unit test must check that this value has been correctly set in the generated pipeline. Check the other unit tests in the test_*_templates.py files to see how this is done. e.g. for QEMU from test_qemu_templates.py

   def test_qemu_installer(self):
       data = """{% extends 'qemu.jinja2' %}
{% set mac_addr = 'DE:AD:BE:EF:28:01' %}
{% set memory = 512 %}"""
       job_ctx = {'arch': 'amd64'}
       template_dict = prepare_jinja_template('staging-qemu-01', data, job_ctx=job_ctx, raw=False)
       self.assertEqual(
           'c',
           template_dict['actions']['boot']['methods']['qemu']['parameters']['boot_options']['boot_order']
       )