The Art of “Frankenstein-ing”: when and how to combine designs through rapid A/B testing
Our design team was faced with a new question: how can we reconcile our partners’ preexisting design work with our own user research and testing?
My work on this project was sponsored by Dartmouth’s Digital Applied Learning Innovation (DALI) Lab, at which I’ve worked for the last ~four years. Partners approach us with ideas and problems and together we work to create new solutions.
Phase 1: Scoping
The Problem
This past spring, my team was tasked with designing a life management platform, a product created by our extremely entrepreneurial partners.
Our partners came to the table with a vision for the end product:
- They conducted their own brainstorming, developed their own feature spec, and hired a freelance designer to create a hi-fi prototype.
- The prototype was detailed but lacked the interactions needed to develop the platform.
- We — the designers — believed we could solve this issue by diving further into user research.
Overall, we knew there were a lot of moving parts that would complicate our ability to conduct thorough research, testing and development.
One thing we tried to keep in mind: our team might think of solutions that differ from the partners’ existing prototype.
Brainstorming
Our team of designers analyzed the existing prototype to identify the key functions of the platform.
We spent a long time brainstorming how to approach conducting user research without negating all of our partner’s previous work.
We briefly considered taking their existing prototype and refining it, but knew that more user research was still needed to flesh out many of the interactions.
The Solution
After a lot of brainstorming, our team of designers and developers honed in on the most feasible goals for our ten-week term. We decided to use the partners’ prototype as a reference point, start from scratch, and build from the ground up:
- design three fleshed-out main pages in hi-fidelity
- establish the most important interactions for every page and how each one connects to the others
- conduct preliminary user testing
- package a new finalized prototype at time of handoff, to be used for the partners’ beta testing
Phase 2: The UX Process
Product Research:
We conducted a lot of online competitive research, trying out two dozen market competitors in categories like content management, time management, productivity, habit-tracking, et. cetera.
User Interviews:
We also conducted preliminary interviews with potential users of the platform.
While these interviews did not completely contradict our partners’ preexisting research, they did reveal new insights our partners did not previously consider.
These interviews laid the groundwork for getting into the user’s mindset and focusing on how we could best approach a solution that would fit their needs best.
User Personas, User Flows & Feature Brainstorming:
After we conducted our interviews, we took our main insights and started brainstorming:
- We made empathy maps, empathy journeys, user journeys, etc
- We used Mural in lieu of physical whiteboards, which helped us map out all of our thoughts into more synthesize-able information
- Our main takeaways continued to differ from the partners’ perception of their user base
- In consultation with our team’s developers, we developed a list of about ~30 features that we felt were integral to the platform
Here’s where things got tricky
Since our partners had already approached us with a full-blown prototype of their ideas, they were hesitant to scrap most of their initial plans in favor of the features we had fleshed out during Phase 1.
Our team and our partners had the same goal, but sought very different approaches.
- As designers, we knew we couldn’t move forward with the assumptions outlined by our partners, but also wanted to fully maintain the integrity of their product
- Our partners wanted to make the most successful product possible, but had already spent a lot of time and effort developing their initial prototype
So what’s the compromise?
Our lab’s design manager recommended this:
- We take the feature ideas we brainstormed from our UX research and make them into grayscales
- Simultaneously we lower the fidelity of the partners’ prototype
- Then, using these two grayscale prototypes, we conduct rapid A/B testing and identify the best parts of each
Hence the name “Frankenstein-ing”.
Phase 3: A/B Testing
Sketches and Grayscales:
We worked pretty quickly to develop our feature brainstorm into sketches and then into grayscales.
- We divided and conquered, with each designer taking full ownership of a single page
- With the amount of detail we were working in, being an expert on one particular page’s interactions was integral to efficient prototyping
- We took the original prototype and brought it down to an equal fidelity
A/B Testing:
We developed two different sets of questions tailored to each prototype and showed our testers both prototypes in random order.
We tried to avoid questions that pitted one prototype against the other, and instead tried to ask open-ended questions whenever possible.
Gathering Insights:
We took all of our user interview transcripts and overlaid everything visually in Figma:
- We left summarized notes next to each prototype screen, using color to denote need for change or works well
- We took the most important quotes and used Figma comments to every inch our prototype
- We also summarized insights for each screen as well as each anonymized user
- Our user testing was incredibly helpful in narrowing down exactly what users wanted and didn’t want in a life management product
- Since we had each already established ownership of a single page, we could transform the two prototypes into a new hybrid version
User Testing Round 2:
Our partners provided us with several rounds of feedback on the new hybrid version, and we continued to tweak the platform until we had finished enough hi-fidelity screens to do a second round of user testing.
Incorporating all this testing
Through the combination of our user testing insights and our partners’ input throughout the process, we were able to collect and incorporate an incredible amount of feedback.
- Since we each took ownership of one page, we were able to focus on the insights for our specific features, develop them in-depth, and iterate repeatedly
We found that users were much more enthusiastic about the hybrid than anyone had been in our A/B testing round.
- We were able to validate the features that were working and tweak the ones that weren’t
- We also scrapped some future features and added some new ones
Next Steps:
By the end of the term, we were able to produce a complete hi-fidelity web prototype ready for another round of user testing.
Key Takeaways
💡Team cohesion was key.
- Much of our design team’s productivity was due to the level of expertise that we each held in different areas of UI/UX
- Speed was needed complete the grayscale prototype, A/B test, test again, AND complete a web hi-fi prototype ready for another round of testing, all within 10 weeks
💡Combing through insights could be overwhelming.
- At some point, we had more feedback than we knew what to do with.
- Looking at the insights from every level — details, big picture, etc. — ensured we could incorporate as much as possible
💡Finding a balance with a partner or client can be challenging but rewarding.
- There were times when I questioned my own decisions, but my fellow designers helped me find the confidence I needed to become a better designer and a better researcher
- Constantly communicating with our partners about the platform’s functionality forced us to thoroughly define every part of the product
tl;dr:
While rapid A/B testing can seem daunting, with the right approach it can be extremely valuable for defining a product’s features. I’m so proud of my team and of myself for what we were able to accomplish in ten weeks.
For more on the DALI Lab’s design and development process, check out our website and Medium page.
