Test-driven product development: design steering the ship with product
A Mental Shift.
There was a mental shift that happened with our developer coworkers that needs to happen with the rest of the balanced product team. This mental shift is perfectly displayed in developers practice of Test Driven Development (TDD).
Being that I am taking a main concept from TDD to come up with the idea of the main theme of this article, Test Driven Product Development (TDPD), it is best to explain what TDD is.
From Wikipedia, “Test-driven development is a software development process that relies on the repetition of a very short development cycle: requirements are turned into very specific test cases, then the software is written to pass the new tests, only.”
There is a step by step process for a developer working in TDD. It is as follows.
- Write a test
- Run the test to see if it fails
- Write your code to the test
- Run the test
- Refactor code
We don’t need to dive deeply into this process and its benefits for developers. The main thing I want you, dear reader, to take from it is that people who partake in this process have found value in determining what the test is before they work on what they are going to be testing.
This is a shift in a mindset from building something and handing it off for someone else to try and prove it’s working as expected, to first thinking about how we want it to be working and then building it to meet those demands. I would say it’s a shift to more enlightened thinking where one is skeptical of what is happening and needs proof of what was delivered works as expected. It’s a shift towards scientific thinking.
I propose this mental shift needs to be made in the minds of Product and Design team members. It produces too many benefits for us not to. I also have decided to call this shift in mindset and subsequent design/product actions Test Driven Product Development (TDPD). Reasoning for this will be readily apparent as you continue to read.
What does this mean for design?
The three main stages of the flow of value in a product team when delivering on a feature that Design is most responsible for are:
- The clarification or creation of the story that needs design
- The discovery of the stories problem space
- The framing of stories possible solutions
This isn’t to say that Design isn’t involved in the other aspects of the development process, however these are just their main areas of responsibility.
Instead of taking a story out of the design backlog and working on designs for it, designers need to shift their mental model into being skeptical of the value they are planning to bring. They should come up with a test. They should be thinking of the test first.
An easy way to nudge yourself into test first thinking would be to require a user hypothesis statement as well as a business hypothesis statement within the story. I recommend to do this collaboratively with product and design since it has to do with both the user and the business. The solution that the designer will come up with has to solve for both hypotheses and their tests. These statements will provide value, but in different ways.
User Value Hypothesis
The user value hypothesis is something that the designer could test before the work gets to the developers. Thus, it should be written before you start work on the initial story. Below is an example and where it fits into the team’s process.
- User Value Hypothesis: We believe this [user outcome] will be achieved if [these users] successfully [attain this user benefit] with [this solution/feature].
Filling this out will help product and design to realize what in specific they need to test to see if the users will be able to get to the value. This forces design to think through how the user will use the design. Let’s use Googles’ smart answers as an example.
- We believe [knowing the height of Jada Smith] will be achieved if [fans of celebrities] successfully [find Jada’s height with our search box, and subsequent smart answer].
With this design can then go through its normal process of discovery and framing. Design can create mocks or a prototype that they could put in front of users to see if the users actions end up triggering the solution (Jada’s height). We know what to test before giving it to development. If the majority of test participants are able to get Jada’s height, we know we can move forward because we have a good idea that what we have designed is usable. That we have designed something that will bring a product value fit to market. Having this test before development maintains a high level of design quality. It passes the test!
If your design doesn’t pass the initial user value test, then it’s time to meet with product again and discuss what to do. Do you move forward with a different solution? Do you make some edits to your initial designs and test again? Do you pivot and provide the value in a completely different way? A great thing to keep in mind during this discussion is the cost of delay, and the value of what you’re working on compares to other things you could be working on (make sure it’s still the highest priority). This could be a moment where your design doesn’t move on to development. Be happy about this! You are saving the company time, money, and the developers won’t be building something that nobody is going to use. In other words, you are reducing the waste that the team was going to create and you are reducing the chance of having to redesign it later! Less design debt is always a beautiful thing.
Business Value Hypothesis
The business value hypothesis is something that the full team can use to measure (or test) your business results.
- Business Value Hypothesis: We believe this [business outcome] will be achieved if [these users] show this [measurable action] due to [this solution/feature].
Sitting down with product and filling this hypothesis statement out will force the designer to think in terms of what the business is going to (hopefully) get in return for providing the user value. It ensures a deeper understanding of the business goals before design gets started. It also forces the team to think though what needs to happen to prove or show the business value.
I’ve found that discussion of the business value hypothesis happens twice. Once to create and align design and product, and then again before development picks it up. The second happens (if working in XP) in the iteration planning meeting.
Let’s go back to the Google example.
- We believe that [ad share revenue increase] will be achieved if [fans of celebrities] show [an increase in return visits due to their ability to find Jada’s height].
The designer and product manager can now determine how this should be tested. The business value is an increase in return visits of people who use the height smart answer, and then the subsequent increase in ad revenue. They can determine multiple ways to test this to make sure that the business is getting value from it. In other words, that there is a business market fit.
One way to test this is to make sure that in the story the appropriate measurements are in place so that when the story is delivered we will be able to validate this hypothesis. In this case you could run an AB test where half the people do not get the smart answer of Jada’s height, and half do. Then track to see what percentage come back to use Google. This will give you the data to go back to your stakeholders and say “Hey, we released this new smart answer feature of people’s height, and it shows a 5% increase of likelihood of people returning.”
The other measurement that you need to use to prove the business value hypothesis we created is the difference in amount of revenue between your control group and the test group in your AB test. You would have to track the control and test group for a while to see if their actions change at all into a higher amount of $$$ the company receives. The hypothesis would then be proven incorrect or correct. If there wasn’t a revenue increase, the team could have a discussion about just the user value (because you see people returning) and if they should release the feature or not. If there was an increase in revenue, then there would most likely be no question that the team should release.
Steering the Ship
This Test Driven Product Development process gives design and product something measurable to show stakeholders. It gives the team data. Business discussions can now be taken away from the HIPPO (Highest paid persons opinion), or the loudest person, and towards a data driven decision. As the design decisions and product decisions can be easily defended by the data from the tests. This also helps smooth over relations between everyone because data has a tendency to it bring rational people into close alignment.
Each round of hypotheses and tests drives the product forward in a healthy manner. This is because it is constantly checking BOTH the business value and the user value. It keeps the design work small and iterative, meaning that the course of the product or the “ship” is being corrected each iteration. A ship with small changes in course correction based on informed data is more likely to succeed in getting to its destination as quickly as possible. If a team were to work and produce work in a way where they aren’t receiving quantitative and qualitative data regularly, they aren’t steering the ship at all. These are the teams that a year down the line, the business realizes don’t deliver the value they wanted or expected and end up getting disbanded. Don’t be that team.
But what happens if you can’t fill out the hypothesis statements?
If you can’t figure out the user value, or the business value, you shouldn’t be working on this story at all. The risk of designing, building, and delivering something of no value to anyone is too high when you can’t even imagine the benefit. This is the perfect time to go back to whomever believes there is a problem to be solved (possibly your stakeholders) and seek to understand their thoughts in order to be able to fill out these hypothesis statements. This is a perfect example of these statements making sure you are going to build the right thing and that thing is going to provide value to both the user and the business… steering in the correct direction.
The Negatives
I would be remiss if I didn’t mention some of the negatives continually working in TDPD.
When working in small iterations, your team may find themselves delivering a story, but not thinking through edge cases. A perfect example of this is all the various error states that may happen with each new feature. Often prioritization of the team as a whole won’t favor these items. The best thing to do is to keep track of these cases by writing stories in the product backlog. Forcing prioritization to happen regularly.
Also, working in this fashion also doesn’t favor what I would consider more “traditional” UX work. The designer is so swept up in the iterative development that the overall system may not come into perfect alignment. Much like building a house a room at a time and getting a funky layout. The more traditional roles of UX would have planned it out from the beginning. Luckily our medium is digital, and evidence grows for the case of taking time to restructure the product once it has proven its user value and market fit.
Neither of these negatives constitute enough of an argument that working in this way should not be your default.
To Conclude
Go try it! Try coming up with both the user value and business value hypothesis with your next story! Try making sure you test both and try to share your findings with your stakeholders! Hopefully you will see all the benefits that I have presented as well. Good luck and Happy Test Driven Product Development to you all. As always, would love to hear your thoughts and experiences in the comments below.
