Creating an encrypted log file

Question

I am creating a client side application which needs to create a log of the user activity but for various reasons this log must not be human readable.

Currently for my development I am creating a plain text log which looks something like this:

12/03/2009 08:34:21 -> User 'Bob' logged in
12/03/2009 08:34:28 -> Navigated to config page
12/03/2009 08:34:32 -> Option x changed to y

When I deploy my application, the log must not be in plain text, so all text must be encrypted. This doesn't appear to be straightforward to achieve as I need the log file to dynamically update as each entry is added.

The approach I was thinking about was to create a binary file, encrypt each log entry in isolation and then append it to the binary file with some suitable demarcation between each entry.

Does anyone know of any common approaches to this problem, I'm sure there has to be a better solution!

Answer 1

Very old question and I'm sure the tech world has made much progress, but FWIW Bruce Schneier and John Kelsey wrote a paper on how to do this: https://www.schneier.com/paper-auditlogs.html

The context is not just security but also preventing the corruption or change of existing log file data if the system that hosts the log/audit files is compromised.

Answer 2

I have the exact same need as you. Some guy called 'maybeWeCouldStealAVa' wrote a good implementation in: How to append to AES encrypted file , however this suffered from not being flushable - you would have to close and reopen the file each time you flush a message, to be sure not to lose anything.

So I've written my own class to do this:

import javax.crypto.*;
import javax.crypto.spec.IvParameterSpec;
import javax.crypto.spec.SecretKeySpec;
import java.io.*;
import java.security.*;


public class FlushableCipherOutputStream extends OutputStream
{
    private static int HEADER_LENGTH = 16;


    private SecretKeySpec key;
    private RandomAccessFile seekableFile;
    private boolean flushGoesStraightToDisk;
    private Cipher cipher;
    private boolean needToRestoreCipherState;

    /** the buffer holding one byte of incoming data */
    private byte[] ibuffer = new byte[1];

    /** the buffer holding data ready to be written out */
    private byte[] obuffer;



    /** Each time you call 'flush()', the data will be written to the operating system level, immediately available
     * for other processes to read. However this is not the same as writing to disk, which might save you some
     * data if there's a sudden loss of power to the computer. To protect against that, set 'flushGoesStraightToDisk=true'.
     * Most people set that to 'false'. */
    public FlushableCipherOutputStream(String fnm, SecretKeySpec _key, boolean append, boolean _flushGoesStraightToDisk)
            throws IOException
    {
        this(new File(fnm), _key, append,_flushGoesStraightToDisk);
    }

    public FlushableCipherOutputStream(File file, SecretKeySpec _key, boolean append, boolean _flushGoesStraightToDisk)
            throws IOException
    {
        super();

        if (! append)
            file.delete();
        seekableFile = new RandomAccessFile(file,"rw");
        flushGoesStraightToDisk = _flushGoesStraightToDisk;
        key = _key;

        try {
            cipher = Cipher.getInstance("AES/CBC/PKCS5Padding");

            byte[] iv = new byte[16];
            byte[] headerBytes = new byte[HEADER_LENGTH];
            long fileLen = seekableFile.length();
            if (fileLen % 16L != 0L) {
                throw new IllegalArgumentException("Invalid file length (not a multiple of block size)");
            } else if (fileLen == 0L) {
                // new file

                // You can write a 16 byte file header here, including some file format number to represent the
                // encryption format, in case you need to change the key or algorithm. E.g. "100" = v1.0.0
                headerBytes[0] = 100;
                seekableFile.write(headerBytes);

                // Now appending the first IV
                SecureRandom sr = new SecureRandom();
                sr.nextBytes(iv);
                seekableFile.write(iv);
                cipher.init(Cipher.ENCRYPT_MODE, key, new IvParameterSpec(iv));
            } else if (fileLen <= 16 + HEADER_LENGTH) {
                throw new IllegalArgumentException("Invalid file length (need 2 blocks for iv and data)");
            } else {
                // file length is at least 2 blocks
                needToRestoreCipherState = true;
            }
        } catch (InvalidKeyException e) {
            throw new IOException(e.getMessage());
        } catch (NoSuchAlgorithmException e) {
            throw new IOException(e.getMessage());
        } catch (NoSuchPaddingException e) {
            throw new IOException(e.getMessage());
        } catch (InvalidAlgorithmParameterException e) {
            throw new IOException(e.getMessage());
        }
    }


    /**
     * Writes one _byte_ to this output stream.
     */
    public void write(int b) throws IOException {
        if (needToRestoreCipherState)
            restoreStateOfCipher();
        ibuffer[0] = (byte) b;
        obuffer = cipher.update(ibuffer, 0, 1);
        if (obuffer != null) {
            seekableFile.write(obuffer);
            obuffer = null;
        }
    }

    /** Writes a byte array to this output stream. */
    public void write(byte data[]) throws IOException {
        write(data, 0, data.length);
    }

    /**
     * Writes <code>len</code> bytes from the specified byte array
     * starting at offset <code>off</code> to this output stream.
     *
     * @param      data     the data.
     * @param      off   the start offset in the data.
     * @param      len   the number of bytes to write.
     */
    public void write(byte data[], int off, int len) throws IOException
    {
        if (needToRestoreCipherState)
            restoreStateOfCipher();
        obuffer = cipher.update(data, off, len);
        if (obuffer != null) {
            seekableFile.write(obuffer);
            obuffer = null;
        }
    }


    /** The tricky stuff happens here. We finalise the cipher, write it out, but then rewind the
     * stream so that we can add more bytes without padding. */
    public void flush() throws IOException
    {
        try {
            if (needToRestoreCipherState)
                return; // It must have already been flushed.
            byte[] obuffer = cipher.doFinal();
            if (obuffer != null) {
                seekableFile.write(obuffer);
                if (flushGoesStraightToDisk)
                    seekableFile.getFD().sync();
                needToRestoreCipherState = true;
            }
        } catch (IllegalBlockSizeException e) {
            throw new IOException("Illegal block");
        } catch (BadPaddingException e) {
            throw new IOException("Bad padding");
        }
    }

    private void restoreStateOfCipher() throws IOException
    {
        try {
            // I wish there was a more direct way to snapshot a Cipher object, but it seems there's not.
            needToRestoreCipherState = false;
            byte[] iv = cipher.getIV(); // To help avoid garbage, re-use the old one if present.
            if (iv == null)
                iv = new byte[16];
            seekableFile.seek(seekableFile.length() - 32);
            seekableFile.read(iv);
            byte[] lastBlockEnc = new byte[16];
            seekableFile.read(lastBlockEnc);
            cipher.init(Cipher.DECRYPT_MODE, key, new IvParameterSpec(iv));
            byte[] lastBlock = cipher.doFinal(lastBlockEnc);
            seekableFile.seek(seekableFile.length() - 16);
            cipher.init(Cipher.ENCRYPT_MODE, key, new IvParameterSpec(iv));
            byte[] out = cipher.update(lastBlock);
            assert out == null || out.length == 0;
        } catch (Exception e) {
            throw new IOException("Unable to restore cipher state");
        }
    }

    public void close() throws IOException
    {
        flush();
        seekableFile.close();
    }
}

Here's an example of using it:

import org.junit.Test;
import javax.crypto.Cipher;
import javax.crypto.CipherInputStream;
import javax.crypto.spec.IvParameterSpec;
import javax.crypto.spec.SecretKeySpec;
import java.io.*;
import java.io.BufferedWriter;



public class TestFlushableCipher {
    private static byte[] keyBytes = new byte[] {
            // Change these numbers, lest other StackOverflow readers can decrypt your files.
            -53, 93, 59, 108, -34, 17, -72, -33, 126, 93, -62, -50, 106, -44, 17, 55
    };
    private static SecretKeySpec key = new SecretKeySpec(keyBytes,"AES");
    private static int HEADER_LENGTH = 16;


    private static BufferedWriter flushableEncryptedBufferedWriter(File file, boolean append) throws Exception
    {
        FlushableCipherOutputStream fcos = new FlushableCipherOutputStream(file, key, append, false);
        return new BufferedWriter(new OutputStreamWriter(fcos, "UTF-8"));
    }

    private static InputStream readerEncryptedByteStream(File file) throws Exception
    {
        FileInputStream fin = new FileInputStream(file);
        byte[] iv = new byte[16];
        byte[] headerBytes = new byte[HEADER_LENGTH];
        if (fin.read(headerBytes) < HEADER_LENGTH)
            throw new IllegalArgumentException("Invalid file length (failed to read file header)");
        if (headerBytes[0] != 100)
            throw new IllegalArgumentException("The file header does not conform to our encrypted format.");
        if (fin.read(iv) < 16) {
            throw new IllegalArgumentException("Invalid file length (needs a full block for iv)");
        }
        Cipher cipher = Cipher.getInstance("AES/CBC/PKCS5Padding");
        cipher.init(Cipher.DECRYPT_MODE, key, new IvParameterSpec(iv));
        return new CipherInputStream(fin,cipher);
    }

    private static BufferedReader readerEncrypted(File file) throws Exception
    {
        InputStream cis = readerEncryptedByteStream(file);
        return new BufferedReader(new InputStreamReader(cis));
    }

    @Test
    public void test() throws Exception {
        File zfilename = new File("c:\\WebEdvalData\\log.x");

        BufferedWriter cos = flushableEncryptedBufferedWriter(zfilename, false);
        cos.append("Sunny ");
        cos.append("and green.  \n");
        cos.close();

        int spaces=0;
        for (int i = 0; i<10; i++) {
            cos = flushableEncryptedBufferedWriter(zfilename, true);
            for (int j=0; j < 2; j++) {
                cos.append("Karelia and Tapiola" + i);
                for (int k=0; k < spaces; k++)
                    cos.append(" ");
                spaces++;
                cos.append("and other nice things.  \n");
                cos.flush();
                tail(zfilename);
            }
            cos.close();
        }

        BufferedReader cis = readerEncrypted(zfilename);
        String msg;
        while ((msg=cis.readLine()) != null) {
            System.out.println(msg);
        }
        cis.close();
    }

    private void tail(File filename) throws Exception
    {
        BufferedReader infile = readerEncrypted(filename);
        String last = null, secondLast = null;
        do {
            String msg = infile.readLine();
            if (msg == null)
                break;
            if (! msg.startsWith("}")) {
                secondLast = last;
                last = msg;
            }
        } while (true);
        if (secondLast != null)
            System.out.println(secondLast);
        System.out.println(last);
        System.out.println();
    }
}

Answer 3

Don't encrypt individual log entries separately and write them to a file as suggested by other posters, because an attacker would easily be able to identify patterns in the log file. See the block cipher modes Wikipedia entry to learn more about this problem.

Instead, make sure that the encryption of a log entry depends on the previous log entries. Although this has some drawbacks (you cannot decrypt individual log entries as you always need to decrypt the entire file), it makes the encryption a lot stronger. For our own logging library, SmartInspect, we use AES encryption and the CBC mode to avoid the pattern problem. Feel free to give SmartInspect a try if a commercial solution would be suitable.

Answer 4

I'm wondering what kind of application you write. A virus or a Trojan horse? Anyway ...

Encrypt each entry alone, convert it to some string (Base64, for example) and then log that string as the "message".

This allows you to keep parts of the file readable and only encrypt important parts.

Notice that there is another side to this coin: If you create a fully encrypted file and ask the user for it, she can't know what you will learn from the file. Therefore, you should encrypt as little as possible (passwords, IP addresses, costumer data) to make it possible for the legal department to verify what data is leaving.

A much better approach would be to an obfuscator for the log file. That simply replaces certain patterns with "XXX". You can still see what happened and when you need a specific piece of data, you can ask for that.

[EDIT] This story has more implications that you'd think at first glance. This effectively means that a user can't see what's in the file. "User" doesn't necessarily include "cracker". A cracker will concentrate on encrypted files (since they are probably more important). That's the reason for the old saying: As soon as someone gets access to the machine, there is no way to prevent him to do anything on it. Or to say it another way: Just because you don't know how doesn't mean someone else also doesn't. If you think you have nothing to hide, you haven't thought about yourself.

Also, there is the issue of liability. Say, some data leaks on the Internet after you get a copy of the logs. Since the user has no idea what is in the log files, how can you prove in court that you weren't the leak? Bosses could ask for the log files to monitor their pawns, asking to have it encoded so the peasants can't notice and whine about it (or sue, the scum!).

Or look at it from a completely different angle: If there was no log file, no one could abuse it. How about enabling debugging only in case of an emergency? I've configured log4j to keep the last 200 log messages in a buffer. If an ERROR is logged, I dump the 200 messages to the log. Rationale: I really don't care what happens during the day. I only care for bugs. Using JMX, it's simple to set the debug level to ERROR and lower it remotely at runtime when you need more details.

Answer 5

It is not clear to me wheter your concern is on the security, or the implement.

A simple implement is to hook up with a stream encryptor. A stream encryptor maintains its own state and can encrypt on the fly.

StreamEncryptor<AES_128> encryptor;
encryptor.connectSink(new std::ofstream("app.log"));
encryptor.write(line);
encryptor.write(line2);
...

Answer 6

Encrypting each log entry individually would decrease the security of your ciphertext a lot, especially because you're working with very predictable plaintext.

Here's what you can do:

1. Use symmetric encryption (preferably AES)
2. Pick a random master key
3. Pick a security window (5 minutes, 10 minutes, etc.)

Then, pick a random temporary key at the beginning of each window (every 5 minutes, every 10 minutes, etc.)

Encrypt each log item separately using the temporary key and append to a temporary log file.

When the window's closed (the predetermined time is up), decrypt each element using the temporary key, decrypt the master log file using the master key, merge the files, and encrypt using the master key.

Then, pick a new temporary key and continue.

Also, change the master key each time you rotate your master log file (every day, every week, etc.)

This should provide enough security.

Answer 7

For .Net see Microsoft Application blocks for log and encrypt functionality: http://msdn.microsoft.com/en-us/library/dd203099.aspx

I would append encrypted log entries to a flat text file using suitable demarcation between each entry for the decryption to work.

Answer 8

FWIW, the one time I needed an encrypted logger I used a symmetric key (for performance reasons) to encrypt the actual log entries.

The symmetric 'log file key' was then encrypted under a public key and stored at the beginning of the log file and a separate log reader used the private key to decrypt the 'log file key' and read the entries.

The whole thing was implemented using log4j and an XML log file format (to make it easier for the reader to parse) and each time the log files were rolled over a new 'log file key' was generated.

Answer 9

Assuming you're using some sort of logging framework, e.g., log4j et al, then you should be able to create a custom implementation of Appender (or similar) that encrypts each entry, as @wzzrd suggested.

Answer 10

This is not really my thing, I'll admit that readily, but can't you encrypt each entry individually and then append it to the logfile? If you that refrain from encrypting the timestamp, you can easily find entries your are looking for and decrypt those when needed.

My point being mainly that appending individual encrypted entries to a file does not necessarily need to be binary entries appended to a binary file. Encryption with (for example) gpg will yield ascii garble that can be appended to an ascii file. Would that solve you problem?