Machine Learning in Firebase: Using Predictions

Getting Started with Firebase Predictions

Firebase Predictions applies machine learning to your Google Analytics for Firebase data to create groups of users based on predicted behavior. These groups are updated daily, and can be used for targeting with notifications, remote configuration and more. Out of the box, Firebase Predictions will create groups of users that are considered likely to churn, likely not to churn, likely to spend and likely not to spend. If you track conversion analytics, you can also create predictions for these events. In this article, I’ll step you through Firebase Predictions, how to set it up, how it works, and how you can create and track your own predictions.

How does it work?

Behind Firebase Predictions is TensorFlow, Google’s framework for Machine Learning. It uses supervised learning where we read the last 100 days of analytics activity for your app and use this to generate a set of features — inputs to a Machine Learning model. From these features, a model is created to predict their output (labels in ML parlance), with the four built-in labels mentioned above. So, for example, when predicting if a user will churn, Predictions will look at the behavior of all users over the last 100 days, and use this to learn about the users that did churn during that time period. The model will then help determine which users might churn in the next 7 days. See Figure 1.

Similarly, models for users that are predicted not to churn, as well as users that will or will not spend in-app are built, and predictions determined.

Getting Started

Prior to using predictions, there are some pre-requisites that your app needs. You’ll go through these in this section.

Step 1. Add Analytics to your App
Predictions gets its data from Google Analytics for Firebase. Implementing this is very easy, and requires no code — giving you a set of common analytics out-of-the-box.

For Android, add this to your build.gradle file:

compile 'com.google.firebase:firebase-core:11.6.0'

For iOS, add this to your Podfile:

pod 'Firebase/Core'

To Learn more, including how to use code to write custom analytics for your own analytics events, check out the documentation at: https://firebase.google.com/docs/analytics/

Step 2. Enable Predictions
When using Firebase in your apps, you have a project in the cloud that is used to administer everything. You access this via the Firebase Console. So, if you’ve never written a Firebase App before, and need a project on the Firebase console, check out this article (iOS/Swift) or this one (Android) to show you how to get started. The console is designed around the principles of Firebase — Developing Apps, Stabilizing them, Measuring them with Analytics and Growing them. Predictions is considered a Grow technology, so you’ll find it in this section. See Figure 2.

Select Predictions and you’ll see a message asking you to opt-in, giving you details on how it works. Note that it presently works best for apps with at least 10,000 monthly active users. Select Yes I’m In and you’ll activate Predictions. See Figure 3.

You’ll now see a screen containing the 4 out-of-the-box predictions. As you’ve just started, there’ll be no predictions available yet. It takes on average 24 hours to create one, and longer if you haven’t been active for at least 100 days, or have less than 10,000 users.

Understanding Predictions

After a little time, your Predictions cards will populate. So, for example, in Figure 4 you can see the churn card for a popular app. In this case, 20% of the app’s users are predicted to churn in the next 7 days. That’s an alarming statistic, and it’s good to know about this before it happens.

Note the slider control, which is used to set tolerance of the prediction. When you choose Low Risk Tolerance as shown, you are sending a signal that you want to avoid false positives as much as possible, and only access those users that have a very high probability of matching the prediction. There are three tolerance levels — Low, Medium and High, with the net of users widening as you move up the levels, at a higher risk of false positives.

You can test the accuracy by opening the slider at the bottom of the card. In this case you can see that the churn prediction is demonstrated as being very accurate — typically in the 95% range. See Figure 5.

Play with the tolerance selection — you’ll notice that for some predictions, in particular custom ones, the accuracy will vary for higher tolerance, which is expected, as you’re tolerating more false positives!

If you’re confident in your prediction, and want to act on it, then you can do so with the Target Users button on the card. Let’s take a look at that in the next section.

Acting on Predictions

The users that meet the prediction are assigned a user group which updates on a daily basis. Thus, when you act on the group, you are acting on the current group, which will be updated tomorrow, and so on.

Your options in acting on the users are to:
Use Remote Config, a technology where app behavior can be changed remotely by setting a value for a distinct group of users. So, for example, a group of users that fall in a prediction could get a different value from everyone else.

Use Notifications, a technology that allows you to target users in the predictions group with a message to re-engage them. The message could include some kind of in-app currency or other incentive to bring them back into your app.

Examples of Predictions in Action

Halfbrick Studios
A great example of this is Halfbrick games who already had Firebase Remote Config implemented in their game Dan the Man and experimented with using the prediction of users that would churn to see if they could increase their retention rate. They chose to provide a gift of in-app currency to users that were predicted to two groups of users: Those that beat level 3 of their game, and those that were predicted to churn by Firebase Predictions.

They then compared this against a control group that received no reward. Here were their retention results:

The results show that there was no difference in retention by giving a reward to users that finished level 3, but there was a 20% increase in 7-day retention for those who had been predicted to churn receiving the gift.

There are two learnings from this: First, using the heuristic of people beating level 3 as being low-risk-of-churn was clearly incorrect, as there was no difference in retention for these. Second was that the churn prediction was successful, but further experimentation, including combining it with a good heuristic could be even more successful.

To this end, Firebase Predictions also allows you to create a prediction on a custom analytic. We’ll discuss that in a bit more detail below.

Rockbite
As well as increasing retention, it’s also important to optimize revenue for your app. To this end, Rockbite did an experiment using Firebase Predictions to provide the best experience for users that are predicted to spend. Their game involves two types of in-app-currency: Crystals, that can be purchased with real currency, or earned by playing the game and Chests, that can only be purchased with the aforementioned Crystals. From experience they could see that some players preferred Crystals at the top of the screen, with Chests beneath, whereas others preferred it the other way around.

Their assumption was that users who have never spent before would logically first buy crystals in order to create a virtual currency balance that could be used to buy chests and get other in-app content. Logically this makes sense, because if they didn’t have any crystals, the other content would not be available to them. So, in order to try their hypothesis, they used the following logic:

Users that fell into the predicted to spend group got Chests at top of screen and Crystals underneath. Through experimentation they found that this increased revenue by 25%.

Users that fell into the predicted not to spend group got Crystals at the top of the screen and Chests underneath. Similarly, this group increased spending by 24%.

The experiments that let to the comparative revenues mentioned above were done by combining Firebase Predictions with Firebase A/B Testing to compare the hypothesis of screen placement for the predicted audience against control groups.

Concrete Software
Many apps and games use In-App Purchasing and In-App Advertisements to generate revenue, and Concrete Software wanted to improve their user experience while simultaneously increasing their revenue. Their hypothesis: Users that spend money in app should not receive adverts. So they used the will_spend prediction in Firebase Predictions to create a user group, and delivered an update to those users that turned off interstitial adverts. For these users they saw an increase in In App Purchase revenue by 21%.

Using Notifications

If you choose to reach your user with Notifications, you’ll be taken to the Firebase notification composer screen, seen in Figure 9. Note the target settings — it’s a user segment, and then you’ll be able to use the Predict setting, and the tolerance level you want to address.

For your app to receive and render the notification, it needs to use the Firebase Cloud Messaging SDK, instructions on implementing which are here (Android) and here (iOS)

Creating your own Prediction

Using the out-of-the box predictions for churn and spend are very useful, but you’ll also likely want to build your own predictions about what is important to you. You can predict based on conversion events, so you’ll need to define custom analytics and set them up as conversion events before you can create any.

So, for example, if you have a game that gives users the option to purchase new levels, you’d likely consider that a conversion event. You would define an event (purchase_level, for example) and write the event when the user does that action. In the console you would mark this event as a conversion event.

Then, on the Predictions screen, you click the Create a Prediction and you’ll see a Create Prediction screen that allows you to name your prediction, pick the analytics event you want to predict, and choose whether you are predicting the user will do that event or will not do that event. See Figure 10 where I’m setting up to predict which users are going to purchase a new level for the game.

Then, over time, the card will be populated with predictions, and you can target them in the same way as the out-of-the-box ones. See Figure 11 for my results of users that purchase new levels!

About 3% of the users of this app are determined to be likely to purchase a new level. It might be time to give them a little nudge, potentially with some in app currency so they may buy more. Or, if you prefer, you could go the opposite route, and create a prediction for users who are not likely to purchase a level, and incentivize them somehow to change their mind!

Hints and Tips

Every app and every scenario is different, so it’s difficult to have a one-tip-fits-all strategy, but here’s a few things I’ve learned while working with predictions.

1. Use the out-of-the-box predictions as soon as you can. They are there, and they’re free. You should start looking into them right away. You may not need to take any action with them immediately, but it’s good to at least know the rough percentage of your audience that fits these categories so you can get better insight about the future of your app.
2. If you don’t want to change anything for churners/spenders right away, you can still use these user groups to understand your audience. For example, you can set a value in Remote Config for people who are going to churn. At runtime you then read this value, and use it to determine your other analytics — for example you could track potential churners through the levels of your game (distinguishing them from everyone), to gain an insight on where churners drop out, and help you avoid that in future, instead of merely guessing what the problem might be.
3. Combine with Firebase A/B Testing to run experiments. This was done very successfully in the games I mentioned above, where, instead of rolling a change out to every user that was predicted to churn, a dynamic group from predictions could be a tested upon
4. Consider the custom analytics that you capture. Too few, and you won’t gain great insight. Too many, and the information may be lost in all the data. As always iterate, iterate, iterate, and don’t be afraid to try new things, while cutting out ones that aren’t working for you.

Summary

In this article you were introduced to Firebase Predictions, and saw what they’re all about, how they work, and how they apply Machine Learning technology to your analytics. You saw how you can create your own predictions from conversion events in your analytics, and also how to act on predictions — setting Remote Config variables or re-engaging your users with notifications. You can learn more about Firebase Predictions at https://firebase.google.com/products/predictions/