Java Generate software configuration

Question

I am working on a project which has some .properties configuration files for datasource, MQ and some other stuff. We do also have launch shell scripts and user profile scripts. The problem i am facing is that we do actually deploy this software on 5 different environments, and of course the configuration is different for each of them. It's being a little bit hard to maintain about 30 plain text files with the configuration. Most of them are pretty much equal, like shell scripts that only have some different path references on.

Do you guys know any kind of tool i could integrate on our build script that might grab these properties from a single file or an embedded database and then generate the proper environment configuration? If it could also generate the scripts it would be even more interesting.

Thanks

Answer 1

In a previous answer, I outlined how Config4* could satisfy your needs. I decided to eat my own dog food, so I knocked up a ready-to-compile-and-run Config4*-based application that will do what you want. I am providing the code inline in this answer. Rather than reading the code via the StackOverview webpage, you might find it easier to copy-and-paste the code into files so you can view it with a text editor.

First, we need a configuration file that defines three variables:

• deploymentType (specified as a command-line argument to have the value dev, staging or prod);

• files (pairs of template files and output files);

• searchAndReplace (pairs of search and replace strings to be applied to the template files to produce the output files). The pairs of strings used depend on the value of deploymentType.

Here is an example of such a file (copy-and-paste this into templates.cfg):

deploymentType ?= ""; # specified with a command-line argument

files = [
    # template file                     output file
    # ----------------------------------------------------
     "log4j-template.properties",       "log4j.properties",
     "hello-template.sh",               "hello.sh",
];

@if (deploymentType == "dev") {
    searchAndReplace = [
        "${db.host}",                   "localhost",
        "${db.user}",                   "guest",
        "${db.log.level}",              "2",
    ];
} @elseIf (deploymentType == "staging") {
    searchAndReplace = [
        "${db.host}",                   exec("hostname"),
        "${db.user}",                   getenv("USERNAME"),
        "${db.log.level}",              "0",
    ];
} @elseIf (deploymentType == "prod") {
    searchAndReplace = [
        "${db.host}",                   "production.example.com",
        "${db.user}",                   getenv("USERNAME"),
        "${db.log.level}",              "0",
    ];
} @else {
    @error "deploymentType must be 'dev', 'staging' or 'prod'";
}

Here is the main-line of the application. You should cut-n-paste the following into InstantiateTemplateFiles.java:

import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.FileReader;
import java.io.FileWriter;
import java.io.IOException;
import java.io.PrintWriter;
import org.config4j.Configuration;
import org.config4j.SchemaValidator;
import org.config4j.ConfigurationException;

public class InstantiateTemplateFiles
{
    public static void main(String[] args)
    {
        Configuration       cfg = Configuration.create();
        SchemaValidator     sv = new SchemaValidator();
        String[]            searchAndReplace;
        String[]            files;
        String              contents;
        String              modifiedContents;
        String              templateFile;
        String              outputFile;
        int                 i;
        String[]            schema = new String[] {
            "deploymentType = string",
            "searchAndReplace=table[string,search, string,replace]",
            "files=table[string,template-file, string,output-file]",
        };

        if (args.length != 2) {
            System.err.println("\nusage: java InstantiateTemplateFiles"
                    + " meta-config-file.cfg deploymentType\n");
            System.exit(1);
        }
        try {
            //--------
            // Parse the configuration file, perform schema validation
            // and retrieve the required configuration variables.
            //--------
            cfg.insertString("", "deploymentType", args[1]);
            cfg.parse(args[0]);
            sv.parseSchema(schema);
            sv.validate(cfg, "", "");
            searchAndReplace = cfg.lookupList("", "searchAndReplace");
            files = cfg.lookupList("", "files");

            //--------
            // Do the real work
            //--------
            for (i = 0; i < files.length; i += 2) {
                Util.searchAndReplaceInFile(files[i + 0], files[i + 1],
                                            searchAndReplace);
            }
        } catch(IOException ex) {
            System.err.println("\n" + ex.getMessage() + "\n");
            System.exit(1);
        } catch(ConfigurationException ex) {
            System.err.println("\n" + ex.getMessage() + "\n");
            System.exit(1);
        }
    }
}

Finally, here is the code to perform the search-and-replace on files. This code is independent of Config4*, so you might find it useful even if you decide to build a non-Config4*-based utility. You should cut-n-paste this code into Util.java:

import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.FileReader;
import java.io.FileWriter;
import java.io.IOException;
import java.io.PrintWriter;

public class Util
{
    public static void searchAndReplaceInFile(
        String      inputFile,
        String      outputFile,
        String[]    searchAndReplacePairs) throws IOException
    {
        String      contents;
        String      modifiedContents;

        contents = Util.readTextFile(inputFile);
        modifiedContents = Util.replace(contents, searchAndReplacePairs);
        Util.writeTextFile(outputFile, modifiedContents);
    }

    public static String readTextFile(String fileName) throws IOException
    {
        BufferedReader          in;
        StringBuffer            result;
        String                  line;

        result = new StringBuffer();
        in = new BufferedReader(new FileReader(fileName));
        while ((line = in.readLine()) != null) {
            result.append(line).append("\n");
        }
        in.close();
        return result.toString();
    }

    public static void writeTextFile(String fileName, String contents)
                                                        throws IOException
    {
        PrintWriter             out;
        StringBuffer            result;
        String                  line;

        out = new PrintWriter(new BufferedWriter(new FileWriter(fileName)));
        out.print(contents);
        out.close();
    }

    public static String replace(
        String                  origStr,
        String                  searchStr,
        String                  replacementStr)
    {
        StringBuffer            result;
        int                     origStrLen;
        int                     searchStrLen;
        int                     currStart;
        int                     pIndex;

        result = new StringBuffer();
        origStrLen = origStr.length();
        searchStrLen = searchStr.length();
        currStart = 0;
        pIndex = origStr.indexOf(searchStr, currStart);
        while (pIndex != -1) {
            result.append(origStr.substring(currStart, pIndex));
            result.append(replacementStr);
            currStart = pIndex + searchStrLen;
            pIndex = origStr.indexOf(searchStr, currStart);
        }
        result.append(origStr.substring(currStart));
        return result.toString();
    }

    public static String replace(
        String                  origStr,
        String[]                searchAndReplacePairs)
    {
        int                     i;
        int                     currIndex;
        String                  subStr;
        String                  replaceStr;
        StringBuffer            result;
        SearchAndReplacePair[]  pairs;
        SearchAndReplacePair    nextPair;

        pairs = new SearchAndReplacePair[searchAndReplacePairs.length / 2];
        for (i = 0; i < searchAndReplacePairs.length; i += 2) {
            pairs[i/2] = new SearchAndReplacePair(origStr,
                                                  searchAndReplacePairs[i + 0],
                                                  searchAndReplacePairs[i + 1]);
        }

        result = new StringBuffer();
        currIndex = 0;
        nextPair = findNextPair(origStr, currIndex, pairs);
        while (nextPair != null) {
            subStr = origStr.substring(currIndex, nextPair.indexOf);
            result.append(subStr);
            result.append(nextPair.replace);
            currIndex = nextPair.indexOf + nextPair.length;
            for (i = 0; i < pairs.length; i++) {
                pairs[i].findNext(currIndex);
            }
            nextPair = findNextPair(origStr, currIndex, pairs);
        }
        subStr = origStr.substring(currIndex);
        result.append(subStr);

        return result.toString();
    }

    private static SearchAndReplacePair findNextPair(
        String                      origStr,
        int                         currIndex,
        SearchAndReplacePair[]      pairs)
    {
        int                         i;
        SearchAndReplacePair        bestSoFar;
        SearchAndReplacePair        item;

        bestSoFar = null;
        for (i = 0; i < pairs.length; i++) {
            item = pairs[i];
            if (item.indexOf == -1) {
                continue;
            }
            if (bestSoFar == null) {
                bestSoFar = item;
                continue;
            }
            if (bestSoFar.indexOf < item.indexOf) {
                continue;
            }
            if (bestSoFar.indexOf > item.indexOf) {
                bestSoFar = item;
                continue;
            }
            if (bestSoFar.length < item.length) {
                bestSoFar = item;
            }
        }
        return bestSoFar;
    }

}


class SearchAndReplacePair
{
    String      source;
    String      search;
    String      replace;
    int         length;
    int         indexOf;
    int         sourceLength;

    public SearchAndReplacePair(String source, String search, String replace)
    {
        this.source = source;
        this.sourceLength = source.length();
        this.search  = search;
        this.replace = replace;
        this.length  = search.length();
        this.indexOf = source.indexOf(search);
    }


    public void findNext(int fromIndex)
    {
        if (indexOf == -1 || indexOf + 1 == sourceLength) {
            indexOf = -1;
        } else {
            indexOf = source.indexOf(search, fromIndex);
        }
    }

}

Assuming you have downloaded and installed Config4J (from the Config4* website), you can compile the utility with the following:

CLASSPATH=.:/path/to/config4j.jar;
export CLASSPATH
javac -classpath .:/ag/projects/config4j/lib/config4j.jar *.java

Here is an example of running it:

java InstantiateTemplateFiles templates.cfg prod

If the file hello-template.sh looks like:

#!/bin/sh
DB_HOST=${db.host}
DB_USER=${db.user}
DB_LOG_LEVEL=${db.log.level}
echo Hello from $DB_USER at log level $DB_LOG_LEVEL on host $DB_HOST

then the generated hello.sh file will look like:

#!/bin/sh
DB_HOST=production.example.com
DB_USER=cjmchale
DB_LOG_LEVEL=0
echo Hello from $DB_USER at log level $DB_LOG_LEVEL on host $DB_HOST

Answer 2

You could have a look at newly announced tools4j-config which lets you handle configuration at runtime rather than build time.

Answer 3

I am the maintainer of Config4*, which is a configuration-file parser library in C++ and Java flavours. Most of the contents in a Config4* configuration file are name=value statements, but you can reference environment variables and the standard output of executing some commands like hostname. You can also have if-then-else statements in a configuration file. For example (keywords are prefixed with "@"):

@if (exec("hostname") @in ["host1", "host2", "host3"]) {
    ... # set variables to values for production environment
} @elseIf (exec("hostname") @in ["host4", "host5", "host6"]) {
    ... # set variables to values for staging environment
} @else {
    @error "Unknown host";
}

I call this adaptable configuration because a single configuration file can adapt its contents for a variety of hosts, user names, and so on. Config4* provides a trivial way to integrate command-line options with a configuration file, so it is possible to have a configuration file that adapts its contents based on the presence of a command-line option such as -env production or -env staging. For example:

env ?= ""; # set by a command-line option
if (env == "production") {
    ... # set variables to values for production environment
} @elseIf (env == "staging") {
    ... # set variables to values for staging environment
} @else {
    @error "You must specify '-env production' or '-env staging' as a command-line option";
}

I can think of two possible ways that Config4* might be of help to you.

One option is for you to embed the Config4* parser in your applications. However, although I think that is a good approach when developing new applications, I think might be tedious to retrofit Config4* to an existing application (not because the Config4* is difficult to use, but just because you will be modifying existing code that uses, say, the Java properties API or an XML API to use a different API, and such modifications tend to be tedious).

The second option better fits with the specifics of your question. You write template versions of your shell scripts and property files. These template files will use a particular syntax, such as '${variable.name}' to specify where values from a configuration file should be used. You then write a small utility application that reads a template file and a configuration file, performs the required substitutions, and then writes the transformed file to disk. You could run that utility application from your build system.

Answer 4

Maven provides this out-of-the-box: http://maven.apache.org/guides/mini/guide-building-for-different-environments.html.