Yi Nie - Loan Application form A/B testing

Background

Credible is a marketplace for student loans, student loan refinancing, and personal loan. We help borrowers compare personalized and accurate rates from multiple lenders so that borrowers can choose the one that best fits their need.

The important part of the borrowing process

Borrowers land on Credible, finish the pre-qualification form and they will be able to see their rates on dashboards. Then borrowers pick one offer, finish up the rest of the information lenders need. Finally, they will be able to submit their loan request to lenders.

Pre-qualification form is an important part of the whole process and this project is mostly focusing on pre-qualification form.

Goal and Metrics

Increase the conversation of pre-qualification form

Before I started, it’s important to understand the problem and make sure all the stakeholders are aligned on the same goal. So I talked to the product manager, loan operation and client success team and listed out the following design success metrics.

Increase the conversation rate of pre-qualification form - increase the percentage of user who finished the form compared to the number of users who start the form.
Improve conversion rates through the full funnel: A success design shouldn’t hurt the conversion of later steps like choosing a loan, submitting the request and closing the loan. On the other hand, if the design has a positive impact on these aspects, we could consider it as successful.

UX goal:

Improve the overall experience of the pre-qualification form. Make the form more efficient and friendly to fill out.
Make sure we can learn from the a/b testing and use the insights for future UX improvements.

1st round - A broader exploration

Research

Where are the opportunities?

Conversation data from Google Analytics reveals that there is a large drop off on the first step and third step. To better understand users’ behavior and pain points, I conducted user testing on the old prequalification form.

Current form pain points:

Users didn’t want to create accounts
Users had difficulties answering some questions (e.g. income), they need to click tooltips and use the help content. But some users didn’t find tooltips.
Overall, users feel the form is straightforward and standard as other loan application site, they expect they will be asked personal information.

Hypothesis and Concepts

Explore different concepts

The important part of A/B testing is to define the hypothesis so that we can control the baseline and test variation to make sure we can confidently learn from these experiments. Based on our user research insights, I came out with concepts and hypothesis to test:

Direction 1: Different flows of creating accounts

I learned from user research that creating accounts is the biggest friction of the whole process. But having an account is necessary because users will need this to access their rates and finalize the loan with lenders. My first hypothesis is that different flow of creating accounts will impact user’s perception of the process and impact conversion. Here are several options I’ve explored.

Current

Create account at end of form

Option 1

Create account at beginning

Option 2

Create account after result summary

Option 3

Separate email and password

Direction 2: Surface helper content to streamline form filling

Users had difficulties answering questions like income, housing payment etc. This may add extra friction to finish the process. Though users in the user testing click the tooltip to get more information, in real life users may not discover the tooltip and abandon. Our second hypothesis is that surfacing helper content will streamline the process and reduce drop off.

Current

Click tooltip to open help content

Option 1

Focus on field to reveal help content

Option 2

Surface tooltip on right column

Direction 3: Humanized form to bring positive feelings to users

The form’s look and feel makes users feel standard and straightforward. But users had the mindset of being asked personal information, which is kind of a negative subconscious. So what if we can make the form look less like a form and bring more humanized into it? Below are the two concept prototypes I’ve made.

Option 1

Nature language form

Option 2

Conversational form

Design and A/B testing

Test creating account flow

One of the challenges we have is that, in order to track different designs impact around the full funnel, the traffic on Credible is not large enough to run multiple ab testing at the same time. So we have to select one direction and run only 2 to 3 testing at the same time to confidently test the hypothesis. We ranked these three directions based on impact and effort. We choose to test the create account flow because we believe it will have the highest impact with relative low implementation efforts.

Baseline: create accounts at the end of the form

In the current design, we try to push the difficult part to the end of form so that user will start easily and make through the end with momentum. But we are still seeing a lot of drop-off at the create account step.

Create account first

Creating accounts first will definitely cause a lot of drop-off at the very beginning. But on the other hand, by revealing the most sensitive information at the front, this will filter out low intent users and improve the conversion later in the process.

Create account after user get results

Give user tease of the result will motivate users to create accounts. For users who receive rates, give users a tease of how many options they have got and overlay the modal on their dashboards. But for users who didn’t receive any rates, we don’t want to mislead them so the main message is “Your result is ready” and it won’t be an overlay of the dashboard.

Ask email at front and password at the end

The hypothesis here is that separating email and password will reduce the feeling of creating accounts so that user feel this is less friction.

Result

Creating accounts at beginning won

The experiment has been running for about 1 months to make sure we have enough data and draw the conclusion without bias.

Variation 1 - creating accounts first, has 1.3% lift on form conversion but 15.7% lift on from starting the form to close a loan.
Variation 3 has negative performance compared to baseline. It has a 6.4% decrease on form conversion and 1.9% decrease on close loan.

Learning

Creating accounts is not the real problem

Even though variation 1 and 2 ’s form conversion are pretty similar, but creating accounts first have a more positive impact later in the process with indicates that our hypothesis is correct. Users who are willing to create accounts first have higher intend and will convert better later in the process.
Separating the email address and password field variation performs worst. This indicates that “creating account” itself is not the real problem. There is some reason behind “I don’t want to create account”

2nd round - Dig deep into the problem

Research

Why don’t users want to create accounts?

Quantitative

The 1st round experiment has set up a good foundation for us. The data reveals that on creating account page, there is a large amount of drop off, but after creating accounts, there are very few users abandon the form. So it’s clear that the biggest opportunity is how can get more low intend users filling the form and checking their rates.

Qualitative:

The testing result from the first iteration also reveals that what really bothers users is not creating accounts, we need to dip deeper to learn the reason behind that. So I did more depth interview with users. The real insight comes after I asked several why.

The user research reveals the real friction of creating accounts is that we are asking user their personal/private information without offering them any value. Users don’t have any idea of what’s going happen next and what they can get.

Hypothesis and Concepts

Transparency at beginning of the form

Based on user research the primary problem we are trying to address is to build trust with users through bringing more transparency.

Choose the testing candidates

Same reason as mentioned above, now we need to choose directions we want to test. Since the improvements are all focusing on create account page, efforts to implement the design are about the same, this time I did internal concept testing. With the testing, I asked participants to rank different concepts based on the helpfulness. Concept 5 and concept 3 get the highest ranking. Below is the summary of this concept evaluation:

Option 1 - Value prompt:

Participants like it's straightforward but value prompt is too vague and doesn’t resonate with them

Option 2 - Social proof

Participants think social proof is important but they prefer reviews from the third party. Participant won’t trust testimonials.

Option 3 -Process Video

Participants find it’s very helpful to understand the process, the video makes it easier to understand

Option 4 - Sample dashboard

Participants like that they can know what lenders are on our platform but the information is too generic

Option 5 - Estimator

Participants all feel this is really helpful to get an estimated range before jumping into creating accounts they indicated that if the rates are good they are more likely to sign up.

Design Iterations

Iterations on rate estimator

After the internal testing, I decided to move forward with the video walkthrough concept and the rate estimator concept. It’s time to push the details. I did several iterations to make sure users can understand how to use it.

Iteration 1

(Option 5 from previous step)

Iteration 2

Design with real content and data

Iteration 3

Rate chart add too much cognitive load

Iteration 4

1. Credit score section took too much space. 2. Too much number to process

Iteration 5

Use red on poor credit score bad may give user negative feeling

Iteration 6

Users indicated they don’t know the slider is interactive

Final Design

Variation 1: Video to illustrate the process

This variation features a quick walkthrough video of our process up to a dashboard. The hypothesis was that showing users our simple process would help users know what to expect, and therefore make the process feel known and simple.

Variation 2: Rate estimator

This variation aims to see if showing potential rates would, in fact, increase sign-ups and overall conversion. This variation features an interactive slider where users can estimate their credit score (by 10 point increments), and we will display the lowest possible rate across lenders.

Result

Video variation won the test

Video variation has a slightly 1.8% decrease on account creating compared to baseline, but it has a 2.1% lift on form conversion and 6.6% lift on close loan.
Rate estimator variation has 12.3% decrease on account creating compared to baseline and 3.6% decrease on close loan.
We decided to call V1 Video the winner and allocate 100% traffic to the video variation and shut off the traffic to the baseline page.

Learning

Transparency helps to build trust

From the winning variation, we learned that providing users context about the process and what to expect appears to help build trust with users and contribute to higher overall conversion after a user has been prequalified.
Even though rates estimator concept performs worse than the video we still want to understand why users are leaving. So I put up an exit survey asking users why they are leaving this page. It turns out that the primary reason is that the rates are too high for them. Even though this variation didn’t perform very well, we still think this is an interesting concept that we could evolve and use in other scenarios in the future.

3rd round - Opportunities on Mobile

Research

Mobile traffic ramped up

Quantitative

The loan application is a complex thing that not a lot of people will expect to finish on your phone. However, in 2017 Credible’s mobile traffic start to ramp up and exceed desktop traffic but the form conversion rate on mobile is pretty low. So it’s time to put more investment on mobile and improve the experience on mobile.

Qualitative:

In order to understand the pain points and frictions on mobile, I did a usability test on the old mobile form. Here are the key insights:

Limited information in viewport causes a lot of scrolling back and forth.
Users’ initial impression is the form is really long and would take them a while to finish
Users had difficulties answering some questions (e.g. income), they need to click tooltips and use the help content. But some users didn’t find tooltips.
Overall, users feel the form is straightforward and standard as other loan application site, they expect they will be asked personal information.

Hypothesis and Concepts

More efficient way to fill the form on mobile

Based on the explorations I did on first round. , here are the approaches we can make form filling more efficient:

Reduce scroll back and forth
Make the form shorter or make the form feel shorter
Surface the help content to streamline the process
Add more humanize factor to the form

I came up with 3 options: the chat UI, one question per step and linking to other social network to prefill information. In order to pick the which option we are going to implement, we listed out how each concept would solve the problem and how difficult it is to implement and decide to move forward with one question per page option first.

Option 1

How does this solve the problem?

Reduce scroll back and forth
Make the form feel shorter
Surface the help content

Effort to implement

Easy

Option 2

How does this solve the problem?

Reduce scroll back and forth
Make the form feel shorter
Add more humanize factor to the form

Effort to implement

Medium

Option 3

How does this solve the problem?

Make the form shorter
Surface the help content

Effort to implement

Hard

Design Iterations

Iterations on one question per page

Iteration 1: Humanized form

One of the issues I didn’t address is to humanize the form to give users a more positive feeling. So I changed the tooltip to a Client Success Specialist explaining how to feel the form.

Iteration 2: Turn dropdown to multiple selections

I tested the prototype above, it successfully reduced the back and forth scrolling. But I still found that all the drop-down fields still requires multiple taps. In order to simplify this part, I changed the drop-down fields to multiple selection fields.

The challenge here is that if the selection has more than five options, what should we do? The product manager helped me analysis the option selection data. We found out that the first four selected options cover 93.17% of our users. So the solution is to surface the top four selected options and hide other options in “Other”.

Iteration 3: Add delight to the product

The last iteration is to add more delight to the design and make the look and feel more friendly for our users. So I added icons on every page and animations on some key pages.

Final Design

Test one question per page

We tested the final one question per page design with old design (baseline). Below is the two design’s overall flow.

Result

One question per page won the test

Variation 1 has an 18.5% lift on form conversation compared to baseline and an 8.8% lift on closing loan.
We decided to call one question per page as the winner and allocate 100% traffic.

Learning

Overall the analysis validated our hypothesis that asking one question at one time streamline the process.
Another interesting insight is that by surfacing the help content and why we are asking these questions, users have less difficulty answering questions. We see a decreased drop off rate on some “hard” questions.
In the future, we are planning to test the new design also on the desktop.