Google app engine JRE Class "Black List"

Question

There is a "JRE Class White List" for the Google App Engine.

What I would really like is a "Black List" -- in other words, Java APIs that will not work on GAE. Does such a list exist? Have any developers run into problems with Java APIs on GAE?

Answer 1

I was looking for something when i came across this query and so thought to share the details on the black & white list of GAE(Google App Engine) so anyone getting such issue could address it properly. Details :-

appengine-agentruntime.jar has two instance variables as :-

private static Agent agent      
private static Set<String> blackList

We getting blackList from agent & agent = AppEngineDevAgent.getAgent(). So if we check b) appengine-agent.jar we can find agent is Class<?> implClass = agentImplLoader.loadClass("com.google.appengine.tools.development.agent.impl.AgentImpl");

And then going to AgentImpl class i.e. c) appengine-agentimpl.jar we can see blacklist variable getting populated at class load with the static initialization & it refers Whitelist for filtering the allowed classes.

static {
            initBlackList();
        }

    public static Set<String> getBlackList() {
        return blackList;
    }

    private static boolean isBlackListed(String className) {
        Set<String> whiteList = WhiteList.getWhiteList();
        return (!whiteList.contains(className))
                && (!className.startsWith("com.sun.xml.internal.bind."));
    }

Finally can check d) appengine-tools-sdk-1.8.3.jar for list of all WhiteList classes.

Conclusion: As a hack in order to use any JRE class which do not belong to this WhiteList one need to play around either with the WhiteList or with the BlackList. A possible hack would be if you unjar the appengine-agentruntime.jar library & comment the content of reject method as

public static void reject(String className) {
    /*throw new NoClassDefFoundError(className + " is a restricted class. Please see the Google " + " App Engine developer's guide for more details.");*/
  }

And then again jar it and use in your project.Hope it helps.

-----------------------------------------------------------------------------

a) appengine-agentruntime.jar :- It holds the actual Runtime class which throws exception (from reject method) for classes which do not belong to above white list.

package com.google.appengine.tools.development.agent.runtime;

import com.google.appengine.tools.development.agent.AppEngineDevAgent;
import com.google.appengine.tools.development.agent.impl.Agent;
import com.google.apphosting.utils.clearcast.ClearCast;
//REMOVED OTHER IMPORTS TO KEEP IT SHORT

public class Runtime {

    private static Agent agent = (Agent) ClearCast.cast(
            AppEngineDevAgent.getAgent(), Agent.class);

    private static Set<String> blackList = agent.getBlackList();

    public static ClassLoader checkParentClassLoader(ClassLoader loader) {
        ClassLoader systemLoader = ClassLoader.getSystemClassLoader();
        return (loader != null) && (loader != systemLoader) ? loader
                : Runtime.class.getClassLoader();
    }

    public static void recordClassLoader(ClassLoader loader) {
        agent.recordAppClassLoader(loader);
    }

    public static void reject(String className) {
        throw new NoClassDefFoundError(className
                + " is a restricted class. Please see the Google "
                + " App Engine developer's guide for more details.");
    }

    private static boolean isBlackListed(Class klass) {
        String className = klass.getName().replace('.', '/');
        return blackList.contains(className);
    }

    // REMOVED OTHER METHODS TO KEEP IT SHORT
}

b) appengine-agent.jar :-

package com.google.appengine.tools.development.agent;

import com.google.apphosting.utils.clearcast.ClearCast;
//REMOVED OTHER IMPORTS TO KEEP IT SHORT

public class AppEngineDevAgent {
    private static final String AGENT_IMPL = "com.google.appengine.tools.development.agent.impl.AgentImpl";
    private static final String AGENT_IMPL_JAR = "appengine-agentimpl.jar";
    private static final Logger logger = Logger.getLogger(AppEngineDevAgent.class.getName());
    private static Object impl;

    public static void premain(String agentArgs, Instrumentation inst) {
        URL agentImplLib = findAgentImplLib();
        URLClassLoader agentImplLoader = new URLClassLoader(
                new URL[] { agentImplLib }) {
            protected PermissionCollection getPermissions(CodeSource codesource) {
                PermissionCollection perms = super.getPermissions(codesource);
                perms.add(new AllPermission());
                return perms;
            }
        };
        try {
            Class<?> implClass = agentImplLoader
                    .loadClass("com.google.appengine.tools.development.agent.impl.AgentImpl");
            impl = ((AgentImplStruct) ClearCast.staticCast(implClass,
                    AgentImplStruct.class)).getInstance();
            AgentImplStruct agentImplStruct = (AgentImplStruct) ClearCast.cast(
                    impl, AgentImplStruct.class);
            agentImplStruct.run(inst);
        } catch (Exception e) {
            logger.log(
                    Level.SEVERE,
                    "Unable to load the App Engine dev agent. Security restrictions will not be completely emulated.",
                    e);
        }
    }

    public static Object getAgent() {
        return impl;
    }

    //REMOVED OTHER METHODS TO KEEP IT SHORT
}

c) appengine-agentimpl.jar :-

package com.google.appengine.tools.development.agent.impl;

import com.google.apphosting.runtime.security.WhiteList;
//REMOVED OTHER IMPORTS TO KEEP IT SHORT

public class BlackList {
    private static final Logger logger = Logger.getLogger(BlackList.class.getName());
    private static Set<String> blackList = new HashSet();

    static {
        initBlackList();
    }

    public static Set<String> getBlackList() {
        return blackList;
    }

    private static boolean isBlackListed(String className) {
        Set<String> whiteList = WhiteList.getWhiteList();
        return (!whiteList.contains(className))
                && (!className.startsWith("com.sun.xml.internal.bind."));
    }

    private static void initBlackList() {
        Set<File> jreJars = getCurrentJreJars();
        for (File f : jreJars) {
            JarFile jarFile = null;
            try {
                jarFile = new JarFile(f);
            } catch (IOException e) {
                logger.log(
                        Level.SEVERE,
                        "Unable to read a jre library while constructing the blacklist. Security restrictions may not be entirely emulated. "
                                + f.getAbsolutePath());
            }
            continue;

            Enumeration<JarEntry> entries = jarFile.entries();
            while (entries.hasMoreElements()) {
                JarEntry entry = (JarEntry) entries.nextElement();
                String entryName = entry.getName();
                if (entryName.endsWith(".class")) {
                    String className = entryName.replace('/', '.').substring(0,
                            entryName.length() - ".class".length());
                    if (isBlackListed(className)) {
                        blackList.add(className.replace('.', '/'));
                    }
                }
            }
        }
        blackList = Collections.unmodifiableSet(blackList);
    }

    private static Set<File> getCurrentJreJars() {
        return getJreJars(System.getProperty("java.home"));
    }

    //REMOVED OTHER METHODS TO KEEP IT SHORT
}

d) appengine-tools-sdk-1.8.3.jar :- It has a class called WhiteList which includes all allowed JRE classes.

package com.google.apphosting.runtime.security;

import java.util.Arrays;
import java.util.HashSet;
import java.util.Set;

public class WhiteList {
    private static Set<String> whiteList = new HashSet(
            Arrays.asList(new String[] {
                    "java.beans.Transient",
                    "java.lang.BootstrapMethodError",
                    "java.lang.Character$UnicodeScript",
                    "java.lang.ClassValue",
                    "java.lang.SafeVarargs",

                    //Removed other classes to keep this article short

                    "java.net.URLClassLoader",
                    "java.security.SecureClassLoader",
                    "sun.net.spi.nameservice.NameService" }));

    public static Set<String> getWhiteList() {
        return whiteList;
    }
}

Answer 2

i use Servlet in my GAE project, however it is not in the whitelist even when it will work without any problem. In fact, Google mention how to use Servlet but it is not in the whitelist

import javax.servlet.http.*;

Mentioned here:

http://code.google.com/appengine/docs/java/runtime.html

but not included here:

http://code.google.com/appengine/docs/java/jrewhitelist.html

I love GAE (because the free quota) but the documentation is a mess.

I use IntelliJ and it mark as an error when the import not in the whitelist. However, it is possible to disable it.

Answer 3

I got a card advertising this service at Google I/O:

LTech AppEngine Compatibility Analyzer

Sounds like it might be of use to you. I have not tried it, if you do try it, please come back and comment. Thanks!

Answer 4

It seems that they've taken more of a white-list approach: http://code.google.com/appengine/docs/java/jrewhitelist.html.

There is also more detail about the sandbox (what files it can access and so on) here: http://code.google.com/appengine/docs/java/runtime.html#The_Sandbox

The restrictions seem to be pretty intuitive (like restricted filesystem access, no JNI, etc).