Regular testing with users is essential to any product development. Many companies however fail to incorporate ongoing user research into their development processes because it's expensive and time consuming to outsource or to establish in-house. Here's what I learned from building a user research culture within a company.
I've spent the last year working with Polymorph, a medium sized software company, to create in-house user testing capability. They design and develop mobile apps using lean product development and agile principles. While they had a rigorous development pipeline, they recognised that they needed to incorporate user research and testing into their processes.
Testing with users ensures that a company's target market will find its products usable, valuable and satisfying. In an agile working environment, user testing should be done regularly to check designs and help to prioritise the product backlog.
If you do not test regularly, there is a danger of falling into one of the pitfalls of waterfall software development, even though you follow agile methods: finding out that what you've developed is not what your users want.
If you outsource testing, it rapidly becomes expensive to do regularly. If you are setting up in-house user testing capability, the initial costs in time, money and attention are also large because you need to:
- Organise and train people to do the testing, analysis and feedback of results,
- Work out how to find target users,
- Create physical space to do the testing and record results,
- Persuade clients to pay for testing to make it feasible,
- Make time for tests in project timelines and sprint cycles and
- Change processes and tools to incorporate testing.
Even with organisational goodwill, the change in mind-sets and structures can be daunting.
The value of user testing is obvious once the testing results are made visible and accessible to the whole company. The trick is to get a foot in the door.
The plan to develop user research capability
Polymorph's creative director and I had several discussions about how he could increase user testing capability within his organisation. His long-term aim was to create a user research unit within the company that had dedicated staff and space so they could eventually provide testing services to other companies. In the medium-term, he wanted to enable regular user testing on every project without external assistance.
There were several barriers that made this a non-trivial task:
- Limited organisational support: Although the creative director was pushing for user testing, the CEO was unconvinced about its cost effectiveness. Initially, the creative director could help me get a foot in the door but only provide limited resources and support.
- Lack of expertise: None of the employees were skilled or practiced in user research. They did not have the confidence to conduct even simple user tests. This meant I would have to take a strong leadership role until employees had enough experience to develop the confidence to plan and conduct tests themselves.
- Lack of process: When time is tight, which it usually is, people fall back on established processes to get the job done. Testing was not part of these processes. All employees had a primary job which took precedence over testing.
We agreed that I would need to work around these constraints and find ways to remove barriers. This would be done using a lean and agile approach so as not to disrupt the organisation too much. I spent the next year implementing minimal user testing, building on successes and learning from failures to grow the user research culture within this organisation and its employees. Here are some of the lessons I learned.
Lesson 1: Regular, embedded expert guidance is needed to create a user research culture
My first attempt to help Polymorph to create a user research culture occurred a year before I joined them on long-term retainer. It was a standalone workshop on how, why and when to do user research.
The workshop consisted of brief theoretical lectures, followed by discussions and activities to practice the ideas. It gave an overview of the different types of user research, looked at usability testing in detail and provided practical checklists for planning and conducting tests.
This workshop was very well received. Participants voiced their excitement about implementing testing plans. I gave them a take-home pack that contained all the practical information they would need to begin user testing.
However, one year later, nothing concrete had happened. Some attempts had been made immediately after the workshop, but these fell by the wayside as people became busy in their primary jobs.
One workshop was not enough to give people the confidence to continue without expert support.
I needed to be more involved in the practical work of promoting, planning, conducting and reporting on user testing. That's why we agreed that I would become more embedded within the company for an extended period of time. As a part time contractor I would be able to develop a better understanding of its processes and people.
Lesson 2: Start low-cost user testing to get results and gain leverage
Even small companies deal with bureaucracy and take time to pivot. There was a period during my employment when I was only employed for 8 hours a week. There was no budget for conducting tests on any projects at this point.
This suited my initial strategy: I decided to start by helping selected employees become comfortable, practised and skilled with user testing in a casual, informal and low cost environment. As employees who had been exposed to UX principles the most, I started working with Polymorph's designers. They already saw the value of testing their designs with users and now just had to learn the principles of corridor testing.
Corridor testing is informal testing with people in the same building or company as the development team. That obviously means that we are not testing the ideal target users, unless we are very lucky, but corridor tests require very little time and resources to conduct. This made them a perfect starting point for us: The designers could just walk down the corridor and ask for 20 minutes of a colleague's time.
In this way, before we had any budget, designers could:
- Begin getting feedback on their designs from people outside their team,
- Practice their facilitation and notetaking skills in a less intimidating and costly situation than a more formal test, and
- Show the value of user testing to clients by reporting on how the feedback had helped to refine their designs.
Lesson 3: User testing needs to be understood as an important part of the design work
Properly done user testing takes time. Consider how long it takes to plan, conduct, and understand results from an interview:
- 2 hours to organise the interview, create a script, and prepare materials such as recording devices and note-taking sheets
- 1 hour for the interview and travel
- 1 hour to review and analyse the results - As a rule of thumb, review and analysis take at least the same amount of time as the interview, and often double if you need to refer back to recordings and transcribe results.
- 1 hour to share results and design implications, and extract user stories
While the designers were open to user testing their work, they did not have very much time to do so. This meant that I would often do the preparation and analysis for them and then show them how I did it. However, for them to become comfortable and capable, they needed to do the work themselves.
Time is usually a scarce resource in business. If something is important to do, it must be prioritised.
The creative director and I began to define user testing as an integral part of design work, so that designers could prioritise it along with other work. We did this informally, through meetings with project managers and the designers. This allowed the designers to push back on design or meeting workload so that they had time to properly prepare for and conduct user tests, and feed the results back into design.
Lesson 4: Make user testing part of existing processes and tools to include the whole team
Once we started using prototypes built by the developers instead of designs, the issue of finding time to do the work had to be dealt with more formally. Tensions arose because the testers needed stable versions to test with in every sprint. This interfered with the developers' working processes. The developers also did not see the positive results of testing, because these were discussed in meetings they didn't attend. While the testers did share a spreadsheet-link to the results, this was easy to ignore in the mess of day-to-day communications.
Testing is not useful if it is not fed back into design and development. It is easy for test feedback to be forgotten if feedback-delivery is not an integral part of each sprint.
We needed to make user testing part of existing processes, so that it fitted into the agile sprint cadence that the team followed. It also needed to be related to the project management board and user stories in project management software. In this way, testing would become an integral part of the product development cycle, visible to all team members.
The company used Slack for communications, and Taiga or Jira for project management. I engaged with project managers to plan how to incorporate testing into sprints, processes and tools:
- We redesigned the sprint schedules to include testing. In this way, the whole team was aware before each sprint when the testing would happen and so when test materials needed to be ready.
- Test highlights were presented in Slack very soon after the test, with a link to more detailed highlights. This meant that the team would at least be aware of the most important test findings.
- The regular design review meeting was redefined to include test feedback and the lead developer. This meant there was a meeting every sprint where test results could be discussed by the whole team.
- At the design review meeting, the team created tickets in Jira or Taiga based on test results. This ensured that the results would be fed back into design and included in the next backlog refinement.
- A portion of the sprint review meeting was provided for feedback of test results to make them visible to clients. It was important for clients to see the value of testing and the usefulness of the test results. The sprint review meeting was the only one that they definitely attended.
Lesson 5: Put user testing on the project roadmap to ensure that it happens effectively
Once we had incorporated user testing in project sprint cycles, another problem emerged. Polymorph used the Goal Oriented Project Roadmap devised by Roman Pichler to loosely plan ahead, but user research was often not explicitly included. If testing was at all included in the roadmap, the time-frame didn't accommodate for necessary changes based on the testing, as opposed to adding new features. This made the whole timeline less reliable and increased the stress of everyone on the project team, as they tried to fit everything in.
When a product is tested, there is a good chance that the design and developed application will have to change. These changes need to be designed as well as implemented, as usability tests show where a design is not working, not how to fix it.
As a company sets clients' expectations about agile processes and iterations at the beginning of a project, they must also be set about user testing time and iteration.
I worked with project managers to redesign the product roadmap. We made sure that there was space for design and development changes based on test results in every month of the roadmap. This helped to set client expectations about the balance of creating new features against improving or redesigning existing features. Making this an explicit item had the added benefit of ensuring that there was time to step back from detailed design and review the product as a whole from a user perspective.
Lesson 6: Use non-specialised and part-time roles to get a foot in the door
There are two examples of my experience with Polymorph that show how compromising on full-time user testing capability can still lead to success:
The creative director initially persuaded clients to pay for my extra time by emphasising my UX design skills over the UX research role for which I had been contracted. Clients could see the value that I brought to design meetings with my UX knowledge, so they were happy to pay for that. I then motivated for and spearheaded testing as part of my role. Increasingly, I did more UX research work and less UX design work as user testing gained traction in the organisation.
When the company first hired someone to manage and conduct user testing instead of the designers, this was not a full-time or specialised role. She was hired as a part-time contractor, who assisted the company in various ways: social media and business development, project manager assistance as well as testing. This made her employment more acceptable to the organisation which was still not convinced about the value or necessity of hiring a full time user researcher.
This part-time role helped the company to commit to user research even more. Over time, the employee spent more time testing and less on the other work. Through her, the company's testing and research capacity was increased, which in turn meant that more projects could do user testing.
Lesson 7: Leverage successes to increase capability
Three examples highlight how project success stories worked to increase user testing capability within the organisation:
Once testing began happening regularly and was included in processes and tools, developers began to see the results. They also began seeing user testing as part of their work. In some cases they attended user tests and became converts to its usefulness for their development work. We asked these developers to speak about their experiences with testing at a team meeting. This opened other developers to the possibilities. Company-wide discussions about testing started happening.
Organisational buzz was that user testing had saved a project. This project was at the bottom of every team member's work list and had a remote manager who lived in a different city. The need for testing brought the team together every sprint. They had to make progress so there was something new to test; there was an explicit date when the prototype had to be ready to put in front of users. The results of tests also gave team members concrete feedback on what was most important to work on. The clients for this project were among the happiest of all the company's clients and the project was on track. A project that should have failed was one of the most successful. This worked to increase organisational buy in, as investing in user testing was shown to provide clear benefits.
I recruited the CEO as a user for testing in one project. Although he did fit the target group, this was also an internal marketing exercise to show him intimately what a user experiences during a test. His perception about the value of testing changed after this experience. He began to refer to user testing as "the QA of design". Given that the company saw the value of QA testing, this was a great success for enhancing the user testing culture within the organisation.
Lesson 8: Know when to be embedded and when to step back
Joining selected projects at the beginning of this job was very useful, as I was immersed in Polymorph's processes and team dynamics. This gave me a sense of their culture and organisational dynamics which is important for creating an effective plan for change.
However, this method also created a challenge: Immediate UX and testing needs within the projects made it difficult to take a step back and reflect on the overall testing capability role. This meant overall progress was slow, especially where actions were required that were not immediately useful on either project. I did not have time to engage with other projects.
Another problem with being embedded in projects was that, with me spearheading testing, no one within the organisation needed to step up and do the work. In order to progress an internal user testing capability, I needed to step back from specific projects and become less embedded within the organisation. Some of the ways in which I began doing this were:
- Moving from managing testing to mentoring others. The benefits of user-testing had been shown in the example projects. The company was prepared to spend more on testing and included more projects. They hired a part-time user test manager to take over much of my embedded work. I moved from doing the work to mentoring her in doing it.
- Moving away from specific projects to oversight across multiple projects. I became less embedded in individual projects and took a more oversight and mentoring role across all company projects, and a more strategic role in working out how to include user testing in processes, contracts, and tools.
- Working more systematically on creating reference resources. I began working more systematically on creating a database of tools, checklists and documents that would be a reference resource. This enabled user research knowledge to remain with the company rather than an individual.
When I stepped back and allowed others to fulfil the day-to-day user research role, several things happened.
- Testing did stall on a couple of projects when I was not there to guide it. However, this was noticed and corrected after a sprint or two.
- Testing was planned and conducted on two new projects without my assistance.
- The part time user tester began planning and conducting additional research more independently of my input. She wanted my advice, but was owning the user research role within the company. This was an interesting challenge for me, as I had to train myself to let go of control and give her the freedom to conduct testing and research, and report back on it, in her own words and using her own methods.
- The company began writing testing into client contracts so it was automatically scheduled into every new project.
Conclusion
One year after I began working with Polymorph systematically, they now have a thriving user testing culture that enriches every project. This happened without any great disruptions or hard decisions by management. I starting doing what we could with limited resources and made it work until we could motivate for more resources. By incorporating user research into processes and tools we ensured that it is sustainable.
There are some challenges to sustainability, such as the fact that the user researcher role is still contracted and part-time. In an agile business world, time will always be tight and processes are not perfect; it is still possible to break the loop and not deliver test feedback into design and onto the backlog. However, challenges will always exist. The user testing culture here is alive, growing and showing value.
Useful resources
- The GO product roadmap by Roman Pichler
- Rocket surgery made easy: the do-it-yourself guide to finding and fixing usability problems by Steve Krug
Cara Winterbottom is a UX consultant and author who has trained extensively in both Psychology and Computing and has a passion for both. The skills and knowledge that she has learned in these areas are powerful tools that she brings to bear on her user experience and software design work.
