Background

‍
Survey Junkie is a mobile and desktop app that delivers market research surveys for users to take and get paid for sharing their opinions. Historically Survey Junkie was a very transactional product; take surveys, get paid. However, we knew there was a larger story to tell as to why our 2.5 million monthly active users were continuously drawn to the platform.

In the beginning of 2022, we put out a survey to our Members to understand their motivations, preferences, and strategies in survey taking. Through the thousands of responses we collected, we noticed a trend emerging through the data; Members seemed to have mental savings goals in mind when using Survey Junkie. Members were using Survey Junkie to save for things like utility bills, holiday presents, or to add to their "fun money." 
‍

‍

Hypothesis 1

Hypothesis: From this discovery, our first hypothesis was formed: If we created a way to set and track goals within Survey Junkie, Members would have a personalized experience, and Survey Junkie would become a more integral part of the Member's daily routine (and thus retention and engagement would increase).‍

‍

Test: To test this first hypothesis, we created a Smoke Test with very basic functionality to understand if Members would be interested in such a feature, without investing a lot of development time. We ran this as an A/B test to 100% of our iOS app users during a four-week period. Cohort A was not shown any goal setting feature (the control), and cohort B was shown the feature (specifically after their first survey complete).

The Smoke Test prototype, utilizing very simple UI components to get an initial signal on Member engagement

‍

Results: The initial results were very clear, with 68.5% of Members in cohort B setting a goal. This cohort was also taking more surveys than the control group (9% more survey clicks per Member), and redeeming more than the control group (17% more unique redeemers). From the standpoint of our Smoke Test, the test was a success! However, we noticed some very interesting and mixed results further down the funnel.
‍

68.5% of Members in the test group set a goal

‍

Only 3.7% of Members who set a goal were achieving their goals during this four-week period. Furthermore, 5 day and 14-day retention also declined for our cohort B by 7% and 26% percent, respectively. Upon further investigation, 33% of Members were setting $30 goals. A $30 goal can take weeks for a casual app user to complete, and in our Smoke Test we were only showing progress reminders at 25, 50, 75, and 100% increments (thus some Members might never see the goal tracking prompt). From our findings, our next hypothesis was formed.

‍

Hypothesis 2

Hypothesis: We hypothesized that Members get frustrated if they don’t see progress towards their goal. By providing static goal tracking at moments of reward (like survey completes) and adding animation to the trackers we could further visualize progress. This would reduce frustration and help members stay engaged by showing them their goal progress in a very tangible way.

‍

Test: We created a more robust test this time that utilized the same goal setup flow as the Smoke Test, but added static goal tracking on our Rewards page, as well as goal tracking on survey complete screens where Members received points. For this test, once again tested 100% of our iOS audience with a 50/50 A/B split.

The next iteration of the goal setting test with static goal tracking in rewarding moments

‍

Results: This test is still in progress, check back soon to see if goal setting resonates with our Members!