CODEWITHSUNDEEP
androidfirebasefirebase-remote-configgoogle-experimentsfirebase-ab-testing
Eyal Gofer
Eyal Gofer

Reputation: 29

Low user count in firebase a/b experiments

I'm having some issue's with Firebase A/B testing on our android app. It seems that the "Users in experiment" count is about 5 times lower then the actual amount of users that should be in said experiment. Experiment started before roll out and is running over a week and still the amount of users entering it is much lower then the amount of unique users that visited the app since. The experiment is open to 100% of eligible users, no conditions or activation event has been set. (i'm cross-referencing data from a/b dashboard and google analytics)

some useful information:

Initialization code for remote config in our android app

    public void initRemoteConfig(){
    // Initialize Firebase Remote Config.
    FirebaseRemoteConfig remoteConfig = FirebaseRemoteConfig.getInstance();

    // Define Firebase Remote Config Settings.
    FirebaseRemoteConfigSettings firebaseRemoteConfigSettings =
            new FirebaseRemoteConfigSettings.Builder()
                    .build();

    // Apply config settings and default values.
    remoteConfig.setConfigSettings(firebaseRemoteConfigSettings);
    remoteConfig.setDefaultsAsync(RemoteConfigHelper.getDefaultValuesMap()).addOnCompleteListener(task -> {
        long cacheExpiration = DateUtils.DAY_IN_MILLIS;
        // If in debug on testing inside the office reduce cacheExpiration to 0
        // so that each fetch goes to the server. This should not be used in release builds.
        if (Tools.DEBUG || MetaDataHelper.getInstance(this).getSetting(R.string.in_office).equals(AppConsts.TRUE)) {
            cacheExpiration = 0;
        }

        remoteConfig.fetch(cacheExpiration)
                .addOnSuccessListener(aVoid -> {
                    // Make the fetched config available via
                    // FirebaseRemoteConfig get<type> calls.
                    remoteConfig.activate();
                    RemoteConfigHelper.isVideoAdUser = FirebaseRemoteConfig.getInstance().getBoolean(SHOW_VIDEO_AD_OVERVIEW);
                    RemoteConfigHelper.logRemoteValues();
                })
                .addOnFailureListener(e -> {
                    Crashlytics.setString(CALLBACK,"remote config fetch failed");
                    Crashlytics.logException(e);
                    // There has been an error fetching the config
                });
    });
    Loger.d(TAG, "Remote instance ID token: " + getPrefString(R.string.pref_notification_reg_id,""));
}

also we've been getting FirebaseRemoteConfigServerException on crashlytics, complete log:

Non-fatal Exception: com.google.firebase.remoteconfig.FirebaseRemoteConfigServerException
Fetch failed: The user is not authorized to access the project. Please make sure you are using the API key that corresponds to your Firebase project.
com.google.firebase.remoteconfig.internal.ConfigFetchHandler.createExceptionWithGenericMessage

(number of users receiving this exception isn't high enough to explain issue)

  1. any idea's why this is happening? our a/b testing are kinda useless until we sort drive more users into the experiments.
  2. Is using inconstant firebase user properties as conditions for experiments a good idea in general? (ie: conditioning experiment on user preferred language)

Thanks in advance :)

Upvotes: 1

Views: 952

Answers (2)

Eyal Gofer
Eyal Gofer

Reputation: 29

Sorry, this question was misleading, i entered fetch interval time in milliseconds when fetch method required it in seconds.

Upvotes: 0

amrro
amrro

Reputation: 1526

I had the same problem when updated Firebase SDK dependencies: 403 Forbidden.

I found this Readme, that describes the problem; It worked with me.

Upvotes: 0

Related Questions