Proper quality assurance can save lives and money. (Anyone remember the Ariane 5 explosion in 1996?) That’s why the openSUSE and Fedora communities heavily rely on openQA, a fully automated and open source test framework. OpenQA is primarily used to test different Linux distributions, but applications can also be installed on top of it to test desktop environments like GNOME and KDE.

How does it work?

OpenQA started in 2009 inside the openSUSE community and is now an integral part of the openSUSE ecosystem. It tests software the same way a human being does: Input is given by keyboard and mouse and results are recorded by comparing screenshots of the process to a set of predefined images. Just like a human tester, openQA detects failures and error messages by comparing what it sees with what it expects.

The first step is the package submission to a new operating system (OS) build. OpenQA runs through a basic pre-build package set to detect basic issues very early. After building a new version of the OS in the Open Build Service, this ISO will be automatically recognized by openQA and validated. The next step is extended “post-validation” testing.

The pictures openQA references during testing are narrowed down to a specific area called needles, and only that small area is compared to the screenshot of the system under test. Needles can be defined by graphical interfaces or plain text.

How can I use it?

There’s a wide variety of scenarios where you can start with an image, install applications on top, and use openQA for graphical or script-based quality assurance. Supported scenarios include:

• VM and bare metal (via IPMI and IP KVM) installs
• A variety of arches (Intel, ppc64le, s390x, and aarch64)
• A variety of Hypervisors (KVM, XEN, KVM for zSystems, HyperV, and VMWare)
• Multi-machine testing
• Disk image creation and testing without OS installation (i.e., testing from image)

In addition to comparing screenshots, openQA supports console scripts written in the openQA DSL as well as other script languages like Bash, Perl, and Python.

Once testing is complete, openQA presents the results in a web interface. From there, users can add bugs or comments, compare test results of previous builds, watch the test process as a video, analyze logs, and much more.

Marita will give a talk about how openQA can help you avoid disasters of biblical proportions at LinuxCon Europe 2016.