Background

‍
Survey Junkie is a mobile and desktop app that connects users with market research surveys, allowing them to earn money by sharing their opinions. While the platform has traditionally been seen as transactional—take surveys, get paid—we recognized there was a deeper narrative behind why 2.5 million monthly active users consistently return to engage with the platform.

At the start of 2022, we conducted a survey among our members to gain insight into their motivations, preferences, and strategies for taking surveys. From the thousands of responses we received, a clear trend emerged: many members used Survey Junkie with specific savings goals in mind. They were leveraging the platform to save for essentials like utility bills, holiday gifts, or to contribute to their "fun money" fund.
‍

‍

Hypothesis 1

Hypothesis: This discovery led to our first hypothesis: by introducing a feature that allows members to set and track goals within Survey Junkie, we could create a more personalized experience. In turn, this would make Survey Junkie a more integral part of members' daily routines, boosting both retention and engagement.‍
‍

Test: To test this hypothesis, we developed a Smoke Test with minimal functionality to gauge member interest in the feature without requiring significant development time. We conducted an A/B test over four weeks, targeting 100% of our iOS app users. Cohort A (the control group) did not see the goal-setting feature, while Cohort B was introduced to the feature, specifically after completing their first survey.

The Smoke Test prototype, utilizing very simple UI components to get an initial signal on Member engagement

‍

Results: 68.5% of members in Cohort B set a goal. This group also demonstrated higher engagement, with 9% more survey clicks per member and 17% more unique redemptions compared to the control group. By the metrics of our Smoke Test, the experiment was a clear success. However, as we examined results further down the funnel, we observed some intriguing and mixed outcomes.
‍

68.5% of Members in the test group set a goal

‍

Only 3.7% of members who set a goal were able to achieve it within the four-week period. Additionally, retention rates in Cohort B dropped by 7% at five days and 26% at 14 days. Further analysis revealed that 33% of members were setting ambitious $30 goals, which could take weeks for casual users to achieve. Compounding this, progress reminders were only displayed at 25%, 50%, 75%, and 100% increments, meaning some members might never see a tracking prompt. These findings led to the formation of our next hypothesis.

‍

Hypothesis 2

Hypothesis: We hypothesized that members may feel frustrated if they don’t see consistent progress toward their goals. By introducing static goal tracking at key reward moments, such as after completing surveys, and enhancing the experience with animated progress visuals, we could make goal progress more tangible. This approach aimed to reduce frustration and keep members engaged by clearly showcasing their achievements along the way.

‍

Test: This time, we designed a more comprehensive test, using the same goal setup flow as in the Smoke Test but adding static goal tracking on the Rewards page and on survey completion screens where members earn points. We conducted this test with 100% of our iOS audience, again applying a 50/50 A/B split.

‍

The next iteration of the goal setting test with static goal tracking in rewarding moments

‍

Results: The added touchpoints made a significant difference. While the percentage of users setting goals remained similar (65%), the real impact showed in the goal achievement metrics. 54% of members who set goals successfully reached them, and 14-day retention rates rose by an impressive 20%! Our members clearly felt a sense a newfound of accomplishment through setting goals towards the things important to them.

‍

Conclusion: Using hypothesis-driven development proved to be an effective way to de-risk our work in real-time. By testing our assumptions early through well-structured experiments, we were able to gather actionable insights that directly influenced our product decisions. This iterative approach allowed us to validate our ideas quickly, adjust course when necessary, and ensure that the features we developed truly met the needs of our members. It reinforced the importance of continuous testing and learning, ultimately helping us create a more engaging and personalized experience while minimizing the risk of investing significant resources into unproven concepts.

‍

‍

‍

‍