DIY digital diary studies

Using digital tools to collect data-on-the-go from participants over time (part 2 of 2)

• This post walks through the use of Google Sheets as a digital diary tool.
• Tips are provided for creating your own collection tool, including set-up, moderation, and analysis.

There are a variety of research methods available when you want to better understand the behavior of your customers or users. Behaviors are always most meaningful in their natural context, but sometimes it’s not practical to directly observe people when and where those behaviors are occurring. Diary studies are a traditional solution to this problem. There are now a variety of reliable digital diary tools that allow you to collect this behavioral data remotely. With a little extra prep work, you can even set up your own custom diary study using freely available tools and resources.

For a quick primer on diary studies and their benefits, see my previous post: Diary Study Basics

As a quick review:

Diary studies are way to collect data from your users or customers over time. In a typical diary study participants self-report their behaviors, frustrations, opinions, desires, and aspirations at defined intervals or in response to carefully designed prompts or tasks. Diary studies are a good choice in any of the following circumstances:

• the behaviors or actions of interest happen sporadically or in unplanned moments
• these behaviors or actions are part of a series or flow which takes place over time
• the research aims to understand change over time
• there is reason to worry that observers would influence behavior or opinion to an unreasonable extent (with private activities, for example)
• sensitive populations that can not tolerate direct observation

The greatest benefits of Diary Studies can be summed up as a trio of time, access, and serendipity.

Procedure

The traditional diary study method uses paper notebooks sent out to participants with instructions and prompts already included. The research team then waits anxiously while diary respondents (hopefully) complete all their diary tasks over the prescribed period of time and then sends them back. At this point the research team is flooded with data and begins the process of sorting, organizing, analyzing, and synthesizing — often finding along the way that some of their tasks were misinterpreted and that a few of their participants were either a bad match for the study or were not able, or willing, to provide the level of response necessary to be useful. If this scenario has happened with several tasks or for numerous participants the yield is a less robust data set than anticipated, which can mean the need to relaunch the study with revised tasks and stricter recruiting criteria. Careful preparation can mitigate the risk, but people are endlessly creative in the ways they interpret instructions and it is generally advised to run a pilot test on all tasks and then over-recruit to ensure an adequate final sample size.

People are endlessly creative in the ways they interpret instructions.

Enter the digital age and the ability to remotely moderate a diary study in-flight. Several great tools exist at a variety of price points. By moderating during a diary study you are able to identify both low-effort participants (who can be uninvited and replaced) and well-meaning participants who have not understood the tasks (who can be redirected).

Case Study

Our needs for a particular study were very specific, and none of the tools we evaluated would accommodate the particular structure we needed for data collection. You may run across a similar need for data outside the capabilities of existing diary study tools, or you simply may not have the budget to pay for one of those tools. There is another way.

Creating Your Own Digital Diary Tool

When the team sat down and outlined what we needed to collect, we found that the desired data easily fit into a typical spreadsheet format. In our case we were collecting personal financial data — which really is perfect for spreadsheets — but we wanted a good deal of qualitative open-response data also, which we concluded fit well into columns alongside the numerical data. Here’s a sample of what we wanted to collect:

• date and time of financial transaction
• amount of transaction and whether is was outgoing or incoming
• method of payment
• description of transaction (what was is for)
• planned or unplanned (was this an expected transaction, or more spontaneous)
• mood at the time (this yielded some very interesting data)
• additional note (also very interesting)
• moderator’s comment or question

To keep participants data private we created one sheet for each participant, and then within each of those sheets we had a tab for each week (three weeks total). This meant that as researchers we had 25 google sheets to keep track of. It was a lot of manual work on our end, certainly a full-time job while the study was running, but it was worth it to keep participant’s financial data separate. We surely got more honest reporting from people who weren’t reporting in a single, shared sheet.

While your study may not involve personal or business finances, if it involves data that fits into a form spreadsheets do that job well, even if they aren’t the most attractive solution.

Using Google Forms is also an option for DIY Digital Diaries, although a single form doesn’t remain accessible if you’re asking for tasks that participants need to add to at their convenience or over time — you may need to create several forms, one for each entry, each with their own link. This would work for daily entries or for tasks that can be completed and reported on in a single attempt.

Other research teams have reported using Google Docs to have open, on-going conversations with diary participants. I prefer the structure that Sheets offers though, because our team set up multiple tabs (one for each week plus a pre- and post- survey) and then hid and unhid them as the study progressed — effectively “locking in” participants’ responses each week. Sheets also lets you lock specific cells with each tab, so we could lock the cells which contained our instructions as prevention against accidental overwrite by participants. Again, lots of manual work but very worth it.

Preparation and Launch

By creating our own survey tool we knew we were giving up the support of a dedicated vendor and the confidence of an established tool. On the other hand we were gaining the ability for complete customization. To make sure we ended up with a usable tool we piloted the Google sheet ourselves. Over the course of a weekend everyone on the team tracked our financial ins and outs. To really put the tool to the test we used the mobile version of sheets, which we had reason to believe would be our participant’s preference. Looking at, and entering data into, a spreadsheet on a phone doesn’t have a reputation for being an awesome experience. We learned pretty quickly that we needed to pin our column headers to the top. We made several other adjustments the following week and then we were ready to launch our study.

Make sure you test your diary tool before expecting your participants to stick with it over the course of the study.

As noted, we created a unique sheet for each participant. For us, this meant setting up 25 sheets each with five identical tabs. Everyone on our team had full access to every sheet. Participants could only access their own sheet — and they were locked out of editing any of the cells with instructional content or data headers. We set up the following tabs:

• Pre-survey: to be filled out within the first few days. This included a few questions to clarify the traits we had screened for in recruiting, plus an opt-in and consent question. You can link a docu-sign consent form here if needed. Participants had to opt-in before we unlocked any of the other tabs.
• Weeks 1, 2, and 3: each on their own tab, locked and unlocked as the weeks progressed.
• Post-survey: a few questions asking participants to reflect on their finances over the past 3 weeks, and one question about their experience participating in the study.

Each tab included (very brief) instructions clearly called out at the top, plus a section at the bottom which noted their moderator’s name and email in case they needed support using the tool.

Moderation (in all things…)

Moderating diary studies in flight is *work.* There is a lot going on across your participant pool, especially during the first few days of on-boarding when participants are seeing the tool for the first time. Think of it this way though: All those participants needing help during those first few days would likely be making incorrect choices on their own if the study wasn’t being monitored. By paying attention and troubleshooting with them now you are ensuring better quality data throughout the study. We also checked on our participant’s progress each day, providing notes of encouragement and thanks, and asking for additional details where needed. This makes so much difference.

I also suggest giving all participants your email address and in cases with high-touch participants (a group you know to be technologically less savvy, for example) consider allowing them to text or call you. You should be prepared to accept some communication during evenings and weekends — but do feel free to set boundaries. If someone can’t get into the diary tool and is panicking about missing a deadline, return that email right away. If someone has a less urgent issue (a question about the method of compensation for participating, perhaps) let that wait until the next working morning. Consider arranging a little flex-time at the office if you find you’re working with a group that requires a lot of off-working-hours support.

Managers: Give your team that flex time. They are working hard.

Bonus Tips for keeping participants on track (Thanks john hicks for asking this question!)

When running a diary study there are three ways to combat non-participating, er, participants.

1. Use a digital diary that you can moderate in-flight. Offer up positive feedback and little nudges as you go.
2. Leverage the full power of incentivization with a schedule of payments. Pay-out over time as sets of tasks are completed. For example, for a 3-week long study (a long time to keep participants engaged!) you might offer $50 per week, paid out the following Monday if all tasks for the previous week are completed by midnight Sunday. Anyone who just barely qualified, or technically qualified but you felt like didn’t put in much effort, can be compensated for that week but notified they are no longer needed for the remainder of the study. Then you can either replace them or just accept a reduced sample size (but one with higher quality overall).
3. Offer a *Bonus* incentive at the end. We call it a High-Quality Bonus and we update participants on their eligibility for the bonus each week. Our intent is to give this to everyone because we really want everyone to provide high-quality responses. However we have it as leverage to use for those who are not putting in much effort. We might let them know, “John, thank you for completing your tasks this week. They are a little slim on detail though, so your eligibility for the high-quality bonus is at risk. For next week, could you please include the [X thing that we were hoping they’d include]. Doing this will put you back in place for the bonus.” This way, if you end up cutting John after that week (sorry John!) you know you gave him a fighting chance.

Note that for tips 2 and 3 above we are very communicative at the outset and throughout the study, so that no one is confused about why they got $50 and then were uninvited, instead of $150 plus a bonus like they signed up for.

After Data Collection

Once the data collection period ended, we exported all the Sheets into Excel so that we could combine them into a single spreadsheet — basically creating a searchable, sortable database for every transaction logged during the 3-week period. It was just over 1,000 transactions.

Tip: don’t forget to add a column for participant number when you combine the sheets, so that your database lets you see trends between participants.

Enhancing Your Understanding with a Diary Study

Diary studies offer some additional benefits when paired with follow-up interviews, surveys, or visits. These benefits maximize the value gained from those in-person visits.

First, diaries allow the research team to “get smarter” about the population, and about the specific individuals in it, before going out to the field. You then meet each participant in person, or over the phone, knowing about specific instances which you can probe into in more depth.

Second, getting to know participants a bit before meeting face-to-face helps break down some initial barriers. It takes time to establish a comfort zone when first engaging with a research participant. With diary studies, you establish a rapport and can “pick up where we left off” when you arrive.

Ethical Considerations

Diary studies are a useful alternative to collecting in-context data when field work isn’t a good option. As with all projects that collect data on human beings, do not forget to keep ethical considerations in mind. Most importantly:

• Ensure that you have what we call informed consent — participants know you are collecting data and they know why. Do not collect any data without permission.
• Use images, quotations, and voice recordings respectfully, and with permission.
• Do not sell participants’ contact information.
• Store data responsibly, or delete it at the end of the project.

*Note that while I work for Google now, I did not at the time of the diary study described above. The decision to use Google tools was made based on their free availability for both the team and the participants.

Sources

Useful References on Diary Studies

Palen, Leysia and Marilyn Salzman. “Voice-Mail Diary Studies for Naturalistic Data Capture under Mobile Conditions.” Department of Computer Science, University of Colorado, Boulder. https://cmci.colorado.edu/~palen/palen_papers/palen-diarystudy.pdf

Rieman, J. (1993). “The Diary Study: A Workplace Oriented Research Tool to Guide Laboratory Efforts Collecting User-Information for System Design.” Proceedings of the ACM INTERCHI’93 Conference on Human Factors in Computing Systems, p.321–326.

***

Kelly Moran utilizes an innate curiosity and unceasing desire to ask “why” to understand how people use products and services to accomplish their goals — whether those goals be work or play. Kelly was formerly a Principal Experience Researcher at the Dallas-based consultancy projekt202, and now works in UX research with Google. Find her other writing at: https://medium.com/@Kel_Moran or follow her on twitter:

Kelly Moran (@Kel_Moran) | Twitter