Formative vs. summative research
Quick and dirty versus slow and rigorous usability research.
This article aims to explore the definition, difference and purpose of formative and summative usability research, how and when they fit into the product design cycle, and how to conduct them.
First, formative and summative evaluations are both a form of evaluative usability research. They are each conducted to understand how a product performs compared to benchmarks in the products history. Unlike summative, formative is also considered to be an exploratory research method, whereby it is used to gain insight into the subject the product deals with and helps in problem identification and solving. Regardless if the research is formative or summative, the goal is to identify and resolve usability issues in order to improve a product’s design.
Note: Processes in this document come from Human Factors Usability Engineering in the medical field, as well as Balance Innovation and Design’s human-centered approach to product design.
Formative Research
In general, formative usability evaluation is done to find and fix problems with an existing product in order to make it more usable. It is done early with simulations and early working prototypes of designs and explores whether usability objectives are obtainable or not. Formative evaluations tell you what is not usable about an existing product, or, what is broken. The purpose is to discover insights and shape the designs direction. Unlike summative, formative testing can and should be done multiple times in a product design cycle. The goal is to help steer design in the right path by discovering usability issues and fixing them in the next iteration. (not release)
One important aspect of formative evaluation is that the audience for the observations and recommendations is the project team itself, used to immediately improve the design of the product or service and refine the development specifications. Results can be less formal than in summative evaluation, as suits the needs of designers, developers, project managers, and other project participants.
Benefits of formative evaluation vs. summative evaluation
Formative is the way to go when a product is being iterated on without being released. It requires much less rigor than a summative evaluation and is intended to be done rapidly and in succession in an iterative process. The only audience of a formative study should be the design team (or other stakeholders). It is simply less-expensive and less time-consuming than a summative evaluation.
When should formative research be done? Early and often in the product development cycle
How many participants are necessary? 4–5 per user group
Example formative methods: Heuristic evaluation, usability testing, and interviews, contextual analysis
Examples of formative usability testing methods (Handbook of Human Factors in Medical Device Design)
- Exploratory testing- Tests users performing high-level tasks or walking through the tasks using low-fidelity simulations (e.g., paper or foam models). Concepts are being tested at this stage of development (e.g., a usability test of a computer simulation of a touch-screen user interface to a heart monitor or paper sketches of the navigational buttons and menus for such a device)
- Assessment testing-Tests that give the users realistic tasks to perform on working prototypes or more fully developed simulations, usually without any major risks involved (e.g. any patients attached to the device). For example, the usability testing of the feel and control of a working prototype of a working prototype of a handheld glucose meter.
- Comparison (contrast) testing (A/B)- Tests comparing two or more design alternatives (e.g., a test to measure the effectiveness and alerting properties of two sets of competing auditory alarms for an infusion pump).
- Comparison (competitive) testing- Tests that gather usability performance of competitor’s products. These tests could be part of the design exploration to understand the best features of existing devices or could be used to support marketing claims for a device (e.g., usability test comparing task success rate and time to run a blood chemistry test using a variety of on-market handheld point-of-care blood analyzers.)
Steps of a Formative Research Process Document
- Statement of evaluation purpose
- Description of interface to be evaluated
- Use case scenarios and tasks involved in the testing process
- Evaluation of the end users — how many groups are there?
- Data collection and data analysis methods to be used
- Explanation of how results will be used to trigger design modifications that will mitigate hazards
Summative Research
Summative usability evaluation is done to understand what is usable about a design after it is complete, or, what is working. It is done when a design is finished (i.e. ready to be released to the public/safe for the public) in order to test that it performs better than a previously released product (or competitive product). Summative evaluations are done for verification and validation that it is safe to release to the public. It is recommended best practice to have formal acceptance criteria (e.g., usability objectives for human performance and user satisfaction rates). The purpose is to answer: how usable is my product? Or, in the case of most medical devices, have usability objectives been met?
Overall, a summative study will help validate or invalidate the usability of a system. They are intended to serve as reference points in the iterative design process to benchmark if designs are being improved over time. Summative tests are meant to be done with market-ready designs as opposed to prototypes or simulations.
Benefits of formative evaluation vs. summative evaluation
Summative evaulation is the way to go when a product is about to be released and the design team needs proof with statistical validity that the new design performs better than the previous design. It should use a sufficient sample size to allow performance of statistical tests on the reliability of the results. It requires more rigor than a formative evaluation and is intended only to be done right before (or right after) a design is released to it’s customers. The audience for a summative evaluation is ultimately the product’s users (which could potentially be millions of customers). It is usually more expensive and time consuming compared to a formative evaluation.
When should summative research be done? Right before or right after a re-design.
How many participants are necessary? 15–20 per user group
Example formative methods: Time on task, completion rates, success rates
Examples of summative usability testing methods (Handbook of Human Factors in Medical Device Design)
- Validation — Used to verify and validate interface design; usability objectives have acceptance criteria. Should include training, documentation and labeling. Tests real users and real tasks with a market ready device. Sample size requires a minimum of 15–20 participants per user group.
Steps of a Summative Research process (Handbook of Human Factors in Medical Device Design)
- Develop a usability task list (tasks that the product must satisfy)
- Determine methods, equipment, materials, procedures and measurements necessary for the evaluation
- Definition of user groups and recruitment for each (15–20 users/group)
- Definition of acceptance criteria. For example, eighty percent of all participants will give the device a rating of 5 or higher (on a 7-point scale) with respect to their overall satisfaction with the device.
- Conduct the validation test using a real (not prototype or simulation) market-ready product
- Analyze data qualitatively by aggregating objective and subjective data to identify potential use errors and determine root causes
- Explanation of how results will be used to trigger design modifications that will mitigate hazards
- Address use errors through risk management strategies
- Conduct human factors (re)validation testing (until no new errors are introduced)
- Document Human Factors Engineering/Usability Engineering process
Conclusion
Formative and summative evaluations both serve discreet purposes in product design. We can understand the concepts further if we step back and look at how the terms are used in the field of education. Formative evaluations in education are meant to help ‘form’ students’ knowledge about a particular subject. Summative evaluations (e.g. final exams) are meant to test that knowledge (in sum) using pass/fail criteria.
As designers, our products must be usable enough to pass final examinations as well. It is up as designers to determine what tasks our designs support, how well it helps users perform them, what potential hazards the design might introduced, and how to overcome those hazards in the next iteration. This is especially true in medical device design. Otherwise, a product might reach our users and perform worse, or even more dangerously, than a previous version.
For more information on formative and summative evaluations and how they are used in the medical field, be sure to check out IEC 62366 and how human factors are used in medical device design. (source)
