Alpha version
Organization
B2B, on-premise software (at the time of the project), 100-130 FTE
The challenge
The company was working on a complete overhaul of their flagship on-premise software. An important improvement was to provide expert users with a lot more configuration capabilities through the user interface, reducing the reliance on their custom-made scripts. Which software bugs would these users encounter during initial set-up and configuration of the new software?
Roles
- My role: Software tester
- Team members: 3 developers
- Consulted: 3 operations engineers
Constraints
- No access to end-users
- No information about users (e.g. knowledge level and experience, user needs, user journeys, etc.)
My approach
Phase 1: Preparation
Course in software testing
As the first software tester in the company with no background in testing, I did a course in software testing, to learn how to systematically look for situations where reality doesn’t align with expectations.
There are many types of software testing. However, given my skill set, my testing would be limited to documentation testing (testing the requirements for the project and its features) and system testing (which covers e.g. functionality testing, performance testing, UI testing).
Exploratory research – Interviews with developers
To understand what the product is used for, how it is used, and what was expected from me, I held multiple interviews with the developers, especially the lead developer who also was the product owner. I used the interviews to create a high-level test plan for testing the system.
Exploratory research – Desk research, interviews with Operations
To understand the journey of end-users, I read user guides from the preceding software and its add-on components. I also interviewed various internal end-users at Operations, reviewed the settings of several production systems, installed the software multiple times, and tried to find my way in the software.
To be able to test software requirements, I asked around for documentation about the current project and the features to be built. However, there was no documentation. Therefore, documentation testing was no option.
Phase 2: Test scenarios, expert review and automated tests
Based on the learnings from my research preparation, I learned that my focus needed to be on system testing, especially functionality testing, performance testing, and UI testing.
I created test scenarios that covered the set-up of a new system. First, I used these test scenarios to do an expert review of the installation and the set-up process. Second, I created test scripts that contained various expected and unexpected values for input fields, and set up an automated testing tool to test the scenarios after every test release.
Result
- Produced high-level test plan, test scenarios, automated test scripts
- Produced an expert review
- Produced test findings as bug tickets in Jira
- Produced suggestions for improvement
Impact
- Developers used my feedback about functionality issues performance issues to fix bugs and increase the processing performance of the software. Ultimately, this resulted in the launch of new software which the company has been selling for many years after launch.
- From interviews with Operations I learned that this team often collaborated asynchronously on configuring customer systems, but had no easy way to inform each other about the specifics of certain settings. Based on my design, the developers then build functionality into the software that enabled admin users to document their decisions in the software itself, using configuration notes in the configuration screens. Years later I noticed that other teams reused this idea and also implemented something similar.
Reflection
- From this project, I learned that the company and particularly this development team held a strong belief that the best way to help our customers was by delivering more technical features. The absence of interactions with end-users, vocal internal subject matter veterans, and time pressure makes it a challenging environment to do user research or to make suggestions for improving the intuitiveness and learnability of the software. To understand whether there is room for user-centered initiatives, I learned it’s important to first gauge a team’s level of curiosity or eagerness to learn.
- To mitigate the complexity of the software, the company decided that end-users needed software documentation. Therefore, after software testing, I became the technical writer for this software, and for other complex software. In this role, I took the opportunity to not limit myself to explaining fields and buttons. During my own process of getting to know the software, I found that usage of this software requires a specific mental model. Therefore, I decided to use part of the user guide to explain the mental model behind the features and the software itself in order to support end-users.