API Usability Testing in 8 Simple Steps

If you are building an API product, developer experience should be at the top of your mind. Testing your APIs for usability is crucial to offer your customers the best experience possible. While there are many people who offer usability tests for user-facing products, API usability testing is a bit more tricky, since your customers are developers.

In this blog post, I will provide you with a guide on how you can get started with API usability testing without burning through your valuable startup budget. We will look at when to test, who to test and eight simple steps to organize your testing.

The Timing

The best time to start usability testing is right at the beginning of your project. If you have not done that, the second best time is now.

It is true. Do not fall into the trap of testing your APIs for usability shortly before a release. Make it part of your development process early on and gather feedback as early as possible, so you can iterate on that feedback and change your product while it is easy to do so.

If you are working in a somehow agile development process, which I will assume at this point, your development cycle will probably look something like this.

The best time to do your usability testing is after you have designed your product or in our case, the API. With modern tools like Swagger and the OpenAPI standard you can easily generate Mock APIs which can then be used for usability testing with real users or your team.

If you do not have mock APIs and prefer to develop first, usability tests can also be done after the Development stage.

The Users

Developers come in different forms and shapes. They vary in their roles and skill sets, but also in the way they perceive content. If you can, try to determine which roles and skills your customers have and derive user personas from it. This will help you to view your APIs from your customers perspective and find suitable candidates for a usability test. We will not go into detail on how to define user personas, but I recommend watching this short video by Pluralsight to get an idea.

In reality, we often do not know the developers who will integrate our APIs when we start developing a new API. Therefore, it is helpful to assume a broad variety of different skill sets and find users of different backgrounds. As we do not want to hire an agency and spend as little money as possible on usability testing, there are mainly two easy options:

• Freelance developer platforms
  Upwork, fiverr, etc.. Look out for developers of various backgrounds and hire them for a 30-60 minute usability test. We will go into detail on how to do that in the next section.
• Onboarding of new developers
  When a new developer joins your team, he can be tasked to make a usability test of the API. This has the nice side effect that they get to know the product/API and a unique view of how users perceive it.

API Usability Testing in 8 Steps

To learn as much as possible from our tests, we need a bit of preparation. These are eight simple steps, to get everything out of your usability tests:

1. Define an explicit task on what your tester is supposed to test.
2. Give as little information as possible, but as much as needed to get started with the assigned task.
3. Find a way to observe the user when using your API. Nowadays, where in-person meetings are less common, you can make use of screen sharing/recording tools.
4. Make sure, the user speaks out loud while testing your product, explaining what he or she is thinking at each step.
5. Provide a possibility for questions, if the user gets stuck at a certain point of the process and does not find any other option than asking for help.
6. Create a detailed report on the observations during the test.
7. Derive learnings and tasks from your observations.
8. Share your learnings and observations with your team.

One other way to get started with usability testing is to do a Hackathon based on your API. This has the positive side effect that you do not only learn about the usability but also potential use cases of your API.

Conclusion

API usability testing does not have to be an expensive and tedious task. Getting started with it will immediately improve your understanding of your own product and how users perceive it. You will be astonished by how much your personal conceptions may differ from a user's fresh view on your API and your customers will thank you for providing the best developer experience. Since, API products are still quite new, there is not much content around usability testing for APIs. If you want to share your learnings and discuss, feel free to reach out to me on Twitter.