Tuesday, January 14, 2014

Analyzing the impact of when the user is prompted to leave a review

I recently released a new app and as is my way with the apps I release myself I've used it to do some experimentation.

It was a simple app originally created for one person but I've added some new features and released updates based on requests and feedback.

In addition to the requested updates, I've also done some experimentation with prompting the person using the app to leave a review.

Adding a prompt to ask the user to leave a review/rating is really easy and if you're registered with DVLUP then you could get 100XP for doing so.

The way that the "rate my app" code works is that a prompt like the above is shown on the 5th time that the app is opened. If the rate option is not selected then a similar prompt is shown again on the 10th time the app is launched.

I have a fundamental objection to this way of working.
When someone opens an app it's because they want to use it to achieve some task. That task is almost never to rate the app. Therefore, asking the user to do something unrelated to what they are actually trying to do is frustrated and just gets in their way. As a rule, making it harder for the user to do what they want and adding unnecessary steps is a bad thing.
It's much better to ask people to do something to help you if it isn't going to get in the way of, or distract from, what they are currently doing.

While I included the default (prompt on app launch) behaviour in the original version of my app, in version 2.0 I changed it to only show the prompt when the back button was used to close the app.

Now, with 3 weeks of data I can do some analysis of what effect this change had.

What this graph shows is that reviews seem to be more tied to downloads than anything else. This suggests to me that the people who are leaving reviews are doing so straight after downloading the app, rather than in response to the prompt displayed to the app. However, this app has seen a higher level of reviews than other similar apps so maybe the prompts are having an effect.
What this really shows me is that having the prompt shown when navigating away from the app, once the person has completed their task, rather than when the app is launched doesn't have a negative impact on the number of ratings/reviews that are actually given.

Admittedly this wasn't a very scientific study and I'm not tracking the interaction with the prompt in any detail so more could be done there.
Maybe it's too early to really tell the full impact of this difference. I'll keep an eye on it and if there's anything worth noting I'll share it here.


  1. Interesting idea. I have always been conflicted on how to ask the user for feedback for the same reason.

  2. It would be interesting to correlate the info with what the ratings were for the time frames. You could use distimo to help with that. However you would also need to factor in features or bugs added as well.