GUI testadapter with computer vision

About the Project

My graduation project at Croonwolter&dros was my first real assignment in C#, and at the same time a project that immediately combined responsibility with technical depth. Within the organization, large scale infrastructure and utility projects are developed, including SCADA systems for bridges and locks.

These systems contain complex graphical user interfaces with dozens to hundreds of animated objects, status indicators, and notifications. Each project is extensively tested to ensure the system behaves correctly under various conditions. Part of the existing test system verified the GUI by comparing specific pixel colors with predefined configuration values.

This approach worked, but it was fragile. Small changes in position, size, or color of an object required configuration updates. In addition, the test system could not simulate real user interaction. Buttons were triggered internally through tags rather than through mouse or keyboard interaction.

The assignment was to investigate how image recognition could be used to make this test system more robust, flexible, and realistic, and to actually implement and integrate this solution within the existing testing landscape.

The Challenge

The goal was to develop a GUI test adapter that could:

• Recognize visual objects and animations using image recognition

• Detect and validate texts and notifications

• Be less sensitive to small changes in shape, position, or color

• Execute control commands in the same way a real user would, using mouse and keyboard interaction

• Integrate with the existing test system

• Be applicable on both Windows and Linux

The solution had to use an open source image recognition library and remain extensible, maintainable, and reliable.

This required in depth research into:

• Image processing and template matching

• Optical character recognition

• Detection of animation states and GUI elements

• Determining similarity levels using confidence scoring

• Cross platform applicability

• Integration within an existing, business critical test system

Approach and Implementation

The project was executed in three phases: analysis, proof of concept, and realization.

In the first phase, I analyzed the existing test system, interviewed colleagues, and fully mapped the current testing methodology. In the second phase, I demonstrated through a proof of concept that image recognition could reliably detect GUI objects and texts, including position and similarity scoring.

In the final phase, I further developed the solution into a fully functional product that was taken into production. The test adapter was developed in C# and extended with a configuration interface that made it easier and faster to add and manage test objects.

The final solution was capable of:

• Recognizing animation objects and determining their state

• Recognizing and validating texts and notifications

• Calculating the match with the expected configuration

• Executing control commands through simulated user interaction

The solution was applied within the Major Maintenance Bridges and Locks project, where the GUI of Urkersluis served as one of the primary test environments.

Result

The new GUI test adapter significantly improved the robustness of the original pixel based testing approach. Small modifications in GUI elements no longer immediately caused failing tests, while both animations and texts could be reliably recognized and validated. User interaction could also be simulated in a more realistic manner.

The solution was successfully deployed to production and made available for both existing and future projects. This resulted in a structural improvement of the SCADA testing process.

Reflection

This project marked the beginning of my work with C#. It was technically challenging and required independent research, experimentation, and translating theoretical image processing concepts into a practical, production ready system.

The graduation thesis received a grade of 9, which confirmed both the technical depth and practical applicability of the solution. After completing my studies, I continued working at Croonwolter&dros for several years, further developing myself as a software engineer.

What I learned in this project, analyzing existing systems, critically questioning assumptions, and developing a proof of concept into a production solution, still forms the foundation of how I approach projects today.