GEISA Conformance - a GEISA validation framework

GEISA conformance is designed to help developers to validate the conformance with the GEISA Specification.

The GEISA Specification is an effort by the Grid Edge Interoperability and Security Alliance to define a consistent, secure, and interoperable computing environment for embedded devices at the very edge of the electric utility grid, like electric meters and distribution automation devices, for the benefit of utilities, platform vendors, and software vendors. If you would like to get involved, please head over to our Wiki page for details on participation (https://lfenergy.org/projects/geisa/). Follow the onboarding link for details about participating in our community process.

Usage

Launch tests automatically

Requirements

The automatic test launcher requires the following requirements:

• On the target:
  • With ssh:
    • Board with a connexion to the network
    • SSH access to the board
  • With serial:
    • Serial connection to the board
    • lrzsz package
• On the host:
  • With ssh: OpenSSH client uses the default SSH key first. sshpass is required when no usable key is found and password authentication is needed (On Ubuntu, install with sudo apt install sshpass)
  • With serial:
    • python3 (On ubuntu, install with sudo apt install python3)
    • pyserial (On ubuntu, install with sudo apt install python3-serial)
    • pexpect (On ubuntu, install with sudo apt install python3-pexpect)
    • lrzsz (On ubuntu, install with sudo apt install lrzsz)
  • For report generation:
    • python3 (On ubuntu, install with sudo apt install python3)
    • python3-junitparser (On ubuntu, install with sudo apt install python3-junitparser)
    • asciidoctor-pdf (On ubuntu, install with sudo apt install ruby-asciidoctor-pdf)
  • For api tests:
    • podman (On ubuntu, install with sudo apt install podman)
    • qemu-user-static (On ubuntu, install with sudo apt install qemu-user-static)
    • mksquashfs (On ubuntu, install with sudo apt install squashfs-tools)
    • launch_gapi_test_app.sh script see API Launching script section
  • For adm tests:
    • bsdmainutils (On ubuntu, install with sudo apt install bsdmainutils)
    • curl (On ubuntu, install with sudo apt install curl)
    • default-jre-headless (On ubuntu, install with sudo apt install default-jre-headless)
    • wget (On ubuntu, install with sudo apt install wget)

A docker support is also available to launch the tests with a container, it requires:

• cqfd (See requirements and installation steps on github)
• qemu-user-static to build the container for cross-compilation (On ubuntu, install with sudo apt install qemu-user-static)

Launch tests

A script is provided to launch all tests automatically. This script will execute the tests and create a report.

./launch_conformance_tests.sh [options]

Required options:

• --ip <board_ip>: IP address of the board to test or
• --serial <serial_port>: Serial port of the board to test

:Warning: When using serial, do not use another tool to access the serial port while running the tests, as it may interfere with the tests and cause unexpected results.

Optional options:

• --user <username>: The username for the target device (default: root)
• --password <password>: Password fallback used only when SSH key authentication fails
• --key-path <path>: Optional SSH private key path. Without it, standard OpenSSH key locations and agents are used
• --no-reports : Do not generate test reports (only run tests and display results)
• --no-reboot: Do not reset the board, skipping the tests that require a reboot
• --baudrate <baudrate>: The baudrate for the serial port of the board (default: 115200)
• --no-glee-tests: Do not run GEISA Linux Execution Environment Conformance tests
• --no-gadm-tests: Do not run GEISA Application & Device Management Conformance tests
• --no-gapi-tests: Do not run GEISA Application Programming Interface Conformance tests
• --clean-up: Clean artifacts for the selected test suites and exit. Use the --no-<glee|gadm|gapi>-tests options to exclude corresponding suites
• --help: display help message

The LEE tests are configured in src/GEISA-LEE-tests/user_configuration.conf, which is read by both the host launcher and the target. Edit it before launching a run to select the applications under test and describe the container layout of the implementation under test. APPLICATION_MANIFEST_PATH must be an absolute target-side Deployment Manifest path. It initially points to the reference application's manifest transferred by the SSH launcher. Change it when testing another application layout. It is independent of LXC_PATH. Each container resource setting is a separate absolute target-side path. The defaults locate the reference application's resources under LXC_PATH. Override the settings individually when testing another layout. The geisa-simple reference application is pinned in the src/GEISA-LEE-tests/src/community submodule.

The SSH launcher transfers only the Cukinia runner and the LEE test runtime files. Reference application sources remain on the host. When reference provisioning is selected, only the built application image and its complete manifest are transferred to the board.

GADM test options (optional):

• --host-ip <host_ip>: IP address of the host running the EMS server
• --api-endpoint <endpoint>: EMS API endpoint (default: /api)
• --client-name <name>: ADM client endpoint name (default: geisa_adm_client)
• --client-path <path>: Path to ADM client binary on the board (default: /usr/bin/adm_client)
• --client-psk-identity <id>: PSK identity for the ADM client (default: )
• --client-psk-value <value>: PSK secret for the ADM client (default: auto-generated hex string of length 16)
• --client-params <params>: Parameters to start the ADM client (use case: )
• --package-path <path>: Path to the ADM package to test (a squashfs file containing the client binary for the current implementation)
• --server-url <url>: EMS server URL (default: http://localhost:8080)

Environment variables can also be used to configure the script:

• CONFORMANCE_SCP_ARGS: Additional arguments for the scp command
• CONFORMANCE_SSH_ARGS: Additional arguments for the ssh command
• GLEE_TESTS: Specify the list of GEISA LEE tests name to run (default: all tests)

SSH and SCP first use normal OpenSSH key authentication, including configured identities and agents. If key authentication fails and --password was given, the launcher retries with sshpass.

The tests names correspond to a part of the filename. Example: GLEE_TESTS="operating_system app_isolation" will run only the operating_system and app_isolation tests.

Some conformance tests only make sense once the device has been reset. Each pillar groups them in its own reboot.conf suite (if there is any), which the launcher keeps off the board until it has actually reset it. The other suites therefore run and report first.

A xml and pdf report will be generated in the reports directory.

For ADM tests, if your software package is outside the project directory, you may want to mount its directory in the container by setting the CQFD_EXTRA_RUN_ARGS environment variable before running tests. For example:

export CQFD_EXTRA_RUN_ARGS="-v path/to/package:path/to/package"

To use the docker support run with the following commands:

$ cqfd init
$ cqfd run ./launch_conformance_tests.sh --ip <board_ip> [options]
or
$ cqfd run ./launch_conformance_tests.sh --serial <serial_port> [options]

Launch tests manually

Requirements

The manual launch is requiring some dependencies to generate the report on the host:

• python3 (On ubuntu, install with sudo apt install python3)
• python3-junitparser (On ubuntu, install with sudo apt install python3-junitparser)
• asciidoctor-pdf (On ubuntu, install with sudo apt install ruby-asciidoctor-pdf)

A docker support is also available to generate the report on the host, it requires:

• cqfd (See requirements and installation steps on github)

For the tests to run, you need to have the following requirements on the target:

Launch tests

If you want to launch the tests manually, you can transfer the tests (src/GEISA-LEE-tests) and the orchestrator (src/cukinia) folders to /tmp/conformance_tests folder on the target.

Then on the target, you can run the tests with the following command:

$ /tmp/conformance_tests/cukinia/cukinia -f junitxml -o geisa-lee-conformance-report.xml /tmp/conformance_tests/GEISA-LEE-tests/cukinia.conf

This will generate a geisa-lee-conformance-report.xml file in the current directory. This file will be used to generated the PDF report. If you only want to run the tests without generating the report, you can run the following command:

$ /tmp/conformance_tests/cukinia/cukinia /tmp/conformance_tests/GEISA-LEE-tests/cukinia.conf

Generate report

To generate the PDF report, transfer the XML report to the test-report-pdf folder (/path/to/conformance/src/test-report-pdf) on your host, then generate it with the following commands:

cd /path/to/conformance/src/test-report-pdf
./compile.py -i . -p 'GEISA conformance tests' -d ../pdf_themes -c ../GEISA-LEE-tests/GEISA-LEE-matrix.csv --allow_absent

This will generate a PDF report in the current directory named test-report.pdf.

or you can use the docker support to generate the report on the host with the following commands:

$ cd /path/to/conformance/
$ cqfd init
$ cqfd run "cd src/test-report-pdf && ./compile.py -i . -p 'GEISA conformance tests' -d ../pdf_themes -c ../GEISA-LEE-tests/GEISA-LEE-matrix.csv --allow_absent"

This will generate a PDF report in the test-report-pdf directory named test-report.pdf.

Configuration file

A user configuration file is provided for the test environment and reference application provisioner. It is located in src/GEISA-LEE-tests/user_configuration.conf. All supported parameters and their defaults are declared in that file.

Installation

Download the repository and run the following command to download the dependencies:

$ git submodule update --init

Testing

Static test is provided to validate the code of conformance tests.

Run the test via CQFD

Requirements

Install cqfd, see requirements and installation on github

Run the test

Run the following command to execute the static test:

$ cqfd init
$ cqfd run

Run the test manually

Requirements

The following requirements are needed to run the static test manually:

• shellcheck (On ubuntu, install with sudo apt install shellcheck)
• pylint (On ubuntu, install with sudo apt install pylint)
• black (On ubuntu, install with sudo apt install black)
• clang-format (On ubuntu, install with sudo apt install clang-format)
• clang-tidy (On ubuntu, install with sudo apt install clang-tidy)

Run the test

Run the following command to execute the static test:

$ shellcheck -xo all launch_conformance_tests.sh src/*.sh src/cukinia-tests/tests.d/*.sh
$ pylint src/launch_glee_conformance_tests_serial.py
$ black --check --diff src/launch_glee_conformance_tests_serial.py
$ clang-format --Werror --dry-run src/GEISA-API-tests/src/*.c src/GEISA-API-tests/src/*.h
$ cd src/GEISA-API-tests/src/ && mkdir -p build && protoc --nanopb_out=build/schemas schemas/*.proto -I schemas --nanopb_opt='-Inanopb_options' && cd -
$ clang-tidy -warnings-as-errors=*  -checks=readability-*,clang-analyzer-* src/GEISA-API-tests/src/*.c src/GEISA-API-tests/src/*.h
$ rm -rf src/GEISA-API-tests/src/build

CI

The CI is configured to run static tests (shellcheck, pylint, black, clang-format, clang-tidy) and on target tests on each push.

If you want to add a new target in the CI, add your runner in github settings with a label corresponding to the target and modify .github/workflows/ci-conformance-on-target.yml:

• To add ssh tests add the following code snippet in the jobs section:

on-target-tests-ssh-<target_name>:
    uses: ./.github/workflows/ci-conformance-on-target-ssh.yml
    with:
        runner: <target_name>
        user: <target_user>
    secrets:
        target_ip: ${{ secrets.<target_ip_secret> }}
        target_password: ${{ secrets.<target_ip_password> }}
    needs:
        - on-target-tests-approval

with :

• <target_name> being the name of your target (corresponding to the label you configured).

• <target_user> being the user to connect to the target (optional, default: root).

• <target_ip_secret> being the name of the secret containing the IP address of your target.

• <target_ip_password> being the name of the secret containing the password of your target. (optional, if not provided, no password will be used for the SSH connection)

• To add serial tests add the following code snippet in the jobs section:

on-target-tests-serial-<target_name>:
    uses: ./.github/workflows/ci-conformance-on-target-serial.yml
    with:
        runner: <target_name>
        user: <target_user>
        target_tty: <target_tty>
        target_baudrate: <target_baudrate>
    secrets:
        target_password: ${{ secrets.<target_ip_password> }}
    needs:
        - on-target-tests-approval
        - on-target-tests-ssh-<target_name>

with :

• <target_name> being the name of your target (corresponding to the label you configured).
• <target_user> being the user to connect to the target (optional, default: root).
• <target_tty> being the serial port of your target (e.g. /dev/ttyUSB0).
• <target_baudrate> being the baudrate of your target (optional, default: 115200)
• <target_ip_password> being the name of the secret containing the password of your target. (optional, if not provided, no password will be used for the serial connection)

On-target tests approval

For security reasons, the on-target tests are not automatically executed on each push. A manual approval by a maintainer is required to run the tests. Exception is made for main branch, where the tests are automatically executed on each push.

API Launching script

When executing the API tests using ssh, the launch_gapi_test_app.sh script is required to launch the test application on the target. This script is responsible for setting up the environment and executing the test application. Its goal is to provide a generic way to launch the test application on different targets and with different configurations. A template is available in launch_gapi_test_app.sh.template