Qualitative evaluation of onboarding new developers: A UX benchmarking study (Part 2)

In this two-part series, we explore the ins and outs of the developer experience. In this final chapter, we discuss the qualitative results from our single blind UX benchmarking study measuring the onboarding experience for users of Sendbird vs. Stream, a competitor. You can learn more about the study in Part 1 of this series.

To borrow poorly from Socrates, “An unexamined product is not worth developing.” In our journey to deliver the best possible in-app chat development experience, we knew we needed to understand the experience that app developers have when onboarding with our product.

That’s why we conducted a single-blind study (i.e., participants didn’t know that Sendbird was conducting the study. This ensured that participants could provide candid responses without feeling pressured to provide positive ratings for a certain product.) to measure the onboarding experience between Sendbird and Stream, a competitor. The goal was to conduct UX benchmarking and examine the virtues and follies of our developer onboarding experience. In Part 1, we discussed the quantitative results of our UX benchmarking study. In this 2nd and last part of our blog series about the developer experience, we’ll talk about the qualitative results of the UX benchmarking study. Let’s dive in!

Understanding what makes a good developer onboarding experience

The 20 participants of our study included front-end and full-stack developers from enterprises and SMBs. Each participant had a set amount of time to complete the onboarding tasks for two platforms they weren’t familiar with: Sendbird and Stream.

Participants focused on three main factors that make a quality onboarding experience: minimal cognitive effort during onboarding, an experience that promoted developers’ learning, and consistent guidance throughout the experience. Let’s take a closer look at each of these three factors.

Figure

1. According to study participants, these are the three tenets of a

quality developer onboarding experience: Minimal cognitive effort,

educational, consistent

Minimize cognitive effort

The first experience with your product should not be a difficult one. Developers don’t want extra work distracting them from focusing on the onboarding process and ultimately adopting the product.

One participant said, “If it doesn’t build right out of the box, I don’t want to fiddle around with it … it should be a no-brainer.”

What kind of extra work detracts from an onboarding experience? Some examples include complicated directions that take too long to decipher, the use of custom coding right out the gate, and a long time between starting the onboarding and getting the app running.

One area in Sendbird that participants highlighted for improvement: making copy-and-paste code available in the documentation and ensuring it’s correct from the outset. For that participant, having to tweak the sample code as it went into the product took away from the speed and simplicity of the experience. We have since resolved this issue in our documentation.

By minimizing cognitive load during onboarding, they can focus better on what matters: creating the core chat experience.

Promote learning and continued education

It’s necessary to strike a balance between minimizing cognitive load and promoting learning.

Participants felt that onboarding success without understanding what they did to succeed was a hollow experience. For example, one participant commented that “Stream’s onboarding document seemed like a cakewalk.” It was easy to just copy paste and all, the participant said, but it didn’t specify what the information was. Ironically, the pre-filled values (e.g., API key) that were meant to speed up the onboarding process hindered the developers’ understanding of what those values meant or where to get them. We also observed multiple participants being confused whether the API key and app ID they had already entered during Stream’s onboarding document were a sample key or their own.

While developers are guided through a (distraction-free) onboarding experience, they should still be learning fundamental concepts and actions like:

1. The basic structure of the API
2. The components within the scope of creating a core chat experience
3. How to use the API on their own to implement more advanced features

To promote comprehension in the early stages of using a product, developer onboarding should strive to flatten the learning curve. One of the most effective ways to do this, according to one participant, is to make “very consumable pieces” that are small enough so it’s straightforward to pinpoint and solve problems that arise during or after onboarding.

Consistent instructions across different resources

Writing onboarding and supporting documentation for developers’ use takes a village to produce and keep up to date. But disparate contributions from a diverse group of people often leads to inconsistencies between resources over time.

Participants in our study expected the same instructions, in the same order, with the same code snippets across different resources.

Even something as simple as a difference in steps across documentation can make users work harder to comprehend instructions and question whether both versions are up to date. (See Figure 2 below for a comparison example.) Documentation that doesn’t match the product onboarding experience can quickly lead to confusion—and it increases the odds that developers won’t learn what they’re meant to.

VS

Figure 2. Presenting different code for the same step leads to confusion

The two examples above show different codes for the same step, confusing users about which set of code they should copy and use. While the top image guides users differently according to the Gradle version, the bottom image does not differentiate the approach. We’ve corrected these discrepancies in our updated documentation based on this feedback.

To meet developers’ onboarding expectations, when one resource is updated with a code structure, required 3rd party tool versions, new information, etc., all related documentation and resources should be updated at the same time.

UX benchmarking: What developers expect from resources

Figure 3. Resource ratings (scores out of 7) are higher for Sendbird in the ease of task completion, time taken, & support information satisfaction

A good onboarding experience is built with quality resources that support developers’ learning and make everything simple.

In our study, participants relied on three key resources: a document about how to get started, GitHub repositories, and sample apps. Each was crucial in getting developers acclimated to and excited about the platforms.

Let’s look at what participants expected and experienced with each resource as part of the onboarding experience.

Getting Started docs create a clear path to success

Clear instructions are incredibly important during onboarding to help developers understand the core process of creating in-app chat. Getting Started documentation was considered the most useful resource for learning how to implement the API. These docs are basically the starter template, providing step-by-step instructions.

Participants working with Sendbird almost exclusively began with Getting Started documentation because it was an easy-to-follow way to dive in.

The best Getting Started documentation takes no step for granted. No step is too obvious to be included. But once again it’s about balance—extraneous information detracts from the Getting Started doc. One participant noted that explaining the meaning of properties is not the same as walking developers through the process of getting started. Thus, Getting Started documents should contain the steps required to get started for a rich but streamlined onboarding process.

Figure 4. Resources each participant used to complete the POC, in the order of use

GitHub repository is a gateway to further information

While Getting Started docs help developers learn to crawl, the GitHub repository is where developers learn to fly. Though the GitHub repositories were not central to the onboarding experience for either brand in this research study, they did reassure participants who, as one said, “will be referring to this again and again to check it but not in the initial time.”

So, why is the GitHub repository still crucial for the developer onboarding experience? With its documentation, demos, and samples, GitHub will be an essential resource for users moving forward.

To help developers go beyond “Getting Started,” the participants looked for a repository that clearly provided:

• • A readme that explains what’s in the repository and how to find it
  • Instructions in the readme on how to get started with GitHub
  • Ways to resolve specific questions that go beyond the scope of the onboarding

Sample apps aren’t built for onboarding

Starting with a working sample application might seem like an ideal way to begin onboarding, but participants found this method an inadequate substitute for the rest of the onboarding experience.

Even though both companies’ sample apps provided an in-depth look at a functioning in-app chat instance, in one participant’s words, there was a ton of stuff going on in general, making sample apps “kind of confusing” as an onboarding resource.

Like the GitHub repository, the sample app is a great resource for users who have advanced beyond onboarding to help them build a more advanced application, or to pull styles and features from the sample for their own development purposes.

During onboarding, though, the sample app had too much information in disparate places and not enough instructions to put it all together. Especially in our timed scenario, these qualities diminished the value of the sample app for our participants.

How to build a better developer onboarding process: Recommendations from the UX benchmarking study

To sum up our findings from the experiences of the participants in our UX benchmarking study, we’ve come up with the following recommendations to improve the developer onboarding experience:

Figure 5. Infographic showing recommendations for building a robust developer onboarding process

We’re taking these lessons to heart so we can help you build a modern in-app chat experience quickly and painlessly. If all this talk of developer onboarding has you excited to get started yourself, learn more about our free Developer chat plan or try a full-featured 30-day trial. With all the information available in our docs, tutorials, and samples on GitHub (check out the samples for iOS, Android, and React!), you’ll get in-app chat up and running in no time! 🚀