Avoiding pitfalls in reporting user behavior changes

Interpreting design-induced changes in user behavior: tips to avoid common mistakes and achieve accurate analysis.

Photo by Adam Śmigielski on Unsplash

• Introduction
• Key Factors That Can Affect User Behavior Changes Over Time
  1. Cognitive Fluctuations
  2. Psycho-Social Factors
  3. Sociological Factors
  4. Economical Factors
  5. Political Factors
  6. Technological Factors
  7. Environmental Factors
  8. Demographic Factors
• Effective Approaches for Measuring the Impact of a New Design: Statistical Analysis and Time Series Modeling
• Conclusion

Introduction

When a product undergoes a redesign, it’s common practice to report changes in user behavior patterns as a percentage. However, such reporting assumes that users behave like robots and that any change in design will have a linear effect on their behavior. In reality, user behavior is nonlinear and dynamic, and any design change can have a complex impact on their behavioral patterns. Failing to consider the dynamic nature of user behavior due to a lack of expertise can lead to several problems:

1. Errors in change detection: Reporting changes in user behavior based on percentages can be misleading because it assumes a linear relationship between the design change and user behavior. In reality, users may not react to a design change in a linear way, leading to errors in detecting changes in user behavior.
2. Not measuring dynamic nonlinear changes: Users’ behavior may change in complex and nonlinear ways following a design change. Changes may include shifts in the direction of the trend, changes in the slope of the trend, the creation of new patterns and rhythms, and the emergence of new self-repeating patterns. Failing to measure such changes can result in a limited understanding of the impact of the design change.
3. Missing out on late changes: Changes in user behavior may occur weeks or even months after a redesign, leading to a misinterpretation of the design’s impact. By failing to monitor user behavior over time, late changes may be missed, resulting in an incomplete picture of the design’s impact.

To avoid these problems, it’s crucial to take a more nuanced approach to measuring the impact of a design change. This can be done in two ways:

1. Assessing significance using statistical tests: Statistical tests, such as t-tests and chi-tests, can be used to determine whether the change in user behavior is significant. However, these tests assume a linear relationship between the design change and user behavior and may not capture the complex and dynamic nature of user behavior.
2. Analyzing time-series data: Analyzing time-series data involves monitoring user behavior over time before and after a design change to identify dynamic nonlinear changes in user behavior. This approach allows for a more nuanced understanding of the impact of the design change, including changes in the direction of trends, changes in the slope of trends, and the creation of new patterns and rhythms. However, this approach requires expertise in time-series analysis and may be more challenging to implement.

To effectively measure the impact of a design change, UX metrics and UX research are crucial components. UX metrics, such as task completion rate, time on task, and error rate, can provide valuable insights into user behavior.

In the following sections, I will describe several factors that can influence changes in user behavior over time, beyond the impact of design changes. By understanding these factors, product designers and UX researchers can gain a more comprehensive understanding of user behavior and make more informed decisions about design changes.

Key Factors That Can Affect User Behavior Changes Over Time

1. Cognitive Fluctuations

Cognitive fluctuations refer to the natural changes in cognitive function that occur over time. These fluctuations can affect attention, memory, mood, and decision-making abilities. Understanding the factors that influence cognitive fluctuations can help product designers optimize their designs for maximum usage rates.

• Attention and Vigilance: Attention and vigilance are crucial for many products, especially those that require sustained attention. Factors such as time of day and ultradian rhythms can affect attention and vigilance. Research suggests that attention tends to be highest in the late morning, while ultradian rhythms can cause fluctuations in attention throughout the day. Designers should take into account these factors when creating products that require sustained attention, as usage rates may be higher during times of day when attention is at its peak.
• Memory and Learning: Memory and learning abilities can also fluctuate over time. Factors such as time of day and seasonal changes can affect memory and learning. For example, memory tends to be strongest in the afternoon, while research has suggested that memory and learning abilities are strongest in the summer. Designers should take these factors into account when creating products that are designed to help users learn or remember information, as usage rates may be higher during times of day or seasons when these cognitive functions are at their peak.
• Mood and Affect: Mood and affect can also fluctuate over time, and research suggests that they may vary depending on the time of day and season. For example, people tend to be happier in the summer and more depressed in the winter. Designers should consider these factors when creating products that are designed to improve users’ mood or affect, as usage rates may be higher during times of day or seasons when these cognitive functions are at their lowest.
• Decision Making: Decision-making abilities can also fluctuate over time, and factors such as time of day and ultradian rhythms can affect them. Research suggests that decision-making abilities tend to be strongest in the morning. Designers should take into account these factors when creating products that require users to make decisions, as usage rates may be higher during times of day when decision-making abilities are at their peak.

In conclusion, understanding the factors that influence cognitive fluctuations and their effects on user behavior change is crucial for product designers. By taking into account the factors that affect cognitive function, designers can optimize their products for the best user behavior.

2. Psycho-Social Factors

• Social Norms: Refers to the unwritten rules of behavior that are considered acceptable in a particular social group or culture. Users’ adherence to social norms can affect their engagement with a product, depending on the time of day and the specific social norm in question. For example, if a social norm dictates that users should engage with a particular type of product during work hours, usage rates for that product may be higher during the day than at night.
• Peer Influence: Refers to the influence that peers have on an individual’s behavior, attitudes, and beliefs. Peer influence can affect users’ engagement with a product, depending on the time of day and the specific type of influence in question. For example, if a user’s peers are more likely to engage with a particular product during their free time, that user may be more likely to engage with the product during those same hours.
• Emotional States: Refers to the different moods and emotions that an individual experiences throughout the day. Users’ emotional states can affect their engagement with a product, depending on the time of day and the specific emotion in question. For example, if a user is feeling stressed or anxious, they may be less likely to engage with a product during that time period.
• Motivation: Refers to the drive or desire to achieve a certain goal or outcome. Users’ motivation can affect their engagement with a product, depending on the time of day and the level of motivation in question. For example, if a user is highly motivated to achieve a goal, they may be more likely to engage with a product that helps them achieve that goal, regardless of the time of day.
• Cognitive Load: Refers to the amount of mental effort or resources required to complete a task. Users’ cognitive load can affect their engagement with a product, depending on the time of day and the specific task in question. For example, if a user is already experiencing high cognitive load due to work or other tasks, they may be less likely to engage with a product that requires additional mental effort.

3. Sociological Factors

• Work schedules: The timing and structure of work hours and shifts, which can affect when users are available to engage with a product.
• Commuting patterns: The timing and duration of regular travel to and from work or other locations, which can affect when and where users are available to engage with a product.
• Social activities: The timing and nature of social events, such as meetings, parties, and gatherings, which can affect when and how much users engage with a product.
• Seasonal patterns: The natural rhythms of the changing seasons, which can affect users’ routines, schedules, and preferences for engaging with a product.
• Cultural events: The timing and nature of significant cultural or religious events, such as holidays, festivals, and celebrations, which can affect users’ availability and engagement with a product.

4. Economical Factors

• Disposable income: the amount of money that individuals have available to spend on goods and services, which can impact their willingness and ability to purchase and use certain products.
• Unemployment rate: the percentage of the population that is unemployed, which can impact the availability of funds for discretionary spending.
• Inflation rate: the rate at which the general level of prices for goods and services is increasing, which can impact the affordability of products.
• Interest rates: the cost of borrowing money, which can impact the availability of funds for discretionary spending and investments.
• Consumer confidence: the degree of optimism or pessimism that individuals have regarding the state of the economy and their financial future, which can impact their willingness to make purchases and investments.

These economic factors can affect user behavior in a number of ways, such as changes in consumer demand due to changes in disposable income or consumer confidence, or changes in investment patterns due to fluctuations in interest rates. For example, during periods of high unemployment and low consumer confidence, individuals may be less likely to spend money on discretionary products, resulting in lower usage rates for certain products or services. Similarly, during periods of high inflation, individuals may be less willing to pay higher prices for products, which can lead to lower usage rates.

5. Political Factors

• Government Policies: Government policies such as internet regulations, censorship, and privacy laws can influence the way people use digital products and services. For example, if a government bans certain types of content, users may not be able to access that content, leading to lower usage rates.
• Elections and Campaigns: During election seasons, users may spend more time on social media and news websites to stay informed about candidates and issues. This could lead to higher usage rates for these types of platforms.
• International Relations: Political tensions and conflicts between countries can affect global markets, which in turn can affect the purchasing power of users and their ability to use digital products and services. For example, if there is an economic sanction against a particular country, users from that country may not have access to certain services, leading to lower usage rates.
• Regulatory Changes: Changes in regulations such as taxes and trade policies can affect businesses that provide digital products and services. For example, if there is a new tax on digital advertising, businesses may have to adjust their pricing strategies, which could affect user behavior and usage rates.
• Political Events and Movements: Political events such as protests and movements can have a significant impact on social media and other online platforms. For example, during the Arab Spring protests, social media played a critical role in organizing demonstrations and spreading information, leading to higher usage rates for these platforms.

6. Technological Factors

• Device availability: The availability and access to devices, such as smartphones, tablets, and laptops, could influence user behavior.
• Network connectivity: The quality and reliability of internet connectivity and cellular networks could impact user behavior.
• Software updates: The release of software updates and new features could impact user behavior as users adapt to changes in the user interface or new functionalities.
• Technological advancements: Advances in technology, such as the development of augmented reality or virtual reality, could create new opportunities for usage and engagement.
• Device compatibility: Compatibility issues between different devices or software versions could limit usage rates.
• User interface design: The design and usability of the user interface could affect user behavior, as a more intuitive and user-friendly interface may encourage more frequent and sustained usage.
• Security and privacy concerns: Security breaches or privacy concerns could reduce users’ trust and confidence in a product, leading to decreased usage rates.
• Technical support: The availability and quality of technical support, such as online guides or customer service, could impact user behavior as users may be more likely to continue using a product if they receive prompt and effective technical assistance.

7. Environmental Factors

• Weather Conditions: Weather can significantly affect user behavior, with people more likely to engage in certain activities or use certain products depending on the weather conditions. For example, people may be more likely to use indoor entertainment products like streaming services or online games during rainy or snowy weather.
• Natural Disasters: Natural disasters like hurricanes, floods, and wildfires can disrupt normal routines and cause people to use products or services in different ways. For example, people may need to rely more heavily on online shopping or communication tools during and after a disaster.
• Geographic Location: Geographic location can also impact user behavior, with people in different regions or areas having different needs and preferences for products and services. For example, people in urban areas may be more likely to use ride-sharing services or food delivery apps, while those in rural areas may be more likely to use online shopping or delivery services.
• Environmental Concerns: Environmental concerns like climate change, pollution, and waste reduction can also impact user behavior, with people more likely to use eco-friendly products or support sustainable companies. For example, people may be more likely to use electric cars or reusable shopping bags to reduce their environmental impact.
• Energy Availability: Energy availability can also affect user behavior, with people in areas with limited or unreliable energy access more likely to use products and services that require less energy. For example, people in areas with frequent power outages may be more likely to use solar-powered devices or battery-operated products.

8. Demographic Factors

• Age: Different age groups may have different preferences for technology and digital products, which could affect user behavior.
• Gender: Gender differences in technology use and preferences may also affect user behavior.
• Education: Education levels may impact user behavior, with higher education levels potentially leading to more frequent and effective use of digital products.
• Income: Income levels may impact the ability and willingness to purchase and use digital products, which could affect user behavior.
• Location: Users in different geographic locations may have different access to technology and preferences for digital products, which could affect user behavior.

In summary, the synchronization and patterns of cognitive, psycho-sociological, sociological, economical, political, technological, environmental, and demographic factors can lead to seasonal or rhythmic changes in user behavior that are not necessarily due to a change in the design of the product. These changes can be mistaken for a redesign when, in fact, they are due to external factors that influence user behavior. It is important to consider these factors and their potential impact on use behavior when analyzing user data and making decisions about product design and development.

Effective Approaches for Measuring the Impact of a New Design: Statistical Analysis and Time Series Modeling

There are two ways of measuring the real impact of a new design. The first approach is to assess its significance using statistical tests such as t-tests, ANOVA, chi-tests and regression analysis. These tests can provide valuable insights into the effectiveness of the new design.

However, this approach may not fully capture the temporal changes in usage patterns that can result from factors beyond the design changes. That’s why it’s recommended to use a combination of methods to ensure a comprehensive evaluation of the impact of a new design.

The second approach is to analyze the time series of usage change across time before and after the release of the new design using analysis methods such as Hidden Markov Model (HMM), wavelet transform, Autoregressive Integrated Moving Average (ARIMA) Model, and Difference-in-Differences (DID) Analysis. This approach requires careful consideration of the potential confounding factors discussed earlier. These include socio-psychological, sociological, economic, political, technological, environmental, and demographic factors that can influence usage patterns.

Moreover, this approach requires specialized analytical tools such as time series modeling, causal inference, and intervention analysis to disentangle the effects of the design change from those of other factors.

While the harder way requires more effort and expertise, it can provide a more accurate and nuanced understanding of the impact of the design change over time. Therefore, it is recommended to use a combination of both methods to ensure a comprehensive evaluation of the impact of a new design.

Conclusion

In conclusion, changes in user behavior due to website or digital product redesigns and updates can be misleading if not analyzed properly. As a quantitative UX researcher, I believe it is essential to dynamically analyze these changes using time series analyses and statistical tests to gain a comprehensive understanding of the impact of design changes on user behavior.

In my future articles, I will be exploring the topic of user behavior analysis in more depth, including how to use different statistical techniques for this purpose. One article will focus on statistical tests, while another will delve into time series analysis. By better understanding user behavior, we can improve the user experience of digital products and achieve better business outcomes.

Remember, user behavior is a crucial aspect of any digital product, not just websites. We need to analyze it carefully to make informed decisions about design changes. Let’s start analyzing user behavior to create better user experiences.

Further Reading

• Understand the statistics behind A/B testing — an intuitive way
• Analysis of Time Series Data