My xor encryption not working on complex files

Question

Just for fun I have been trying to make a file encryptor which does a bitwise xor encryption against a 128-bit key. It seems to work for plain text files using some editors, but for others and more complex files I encounter a "corrupted file" error. If I do the encryption on a text file twice, in Gedit it displays the contents of the original file like it should, but in Notepad, I just get a whole bunch of question marks.

I have written this code in Java, however the encryption part is pretty universal. The key is in the form of an array of 8 16-bit characters:

char[] key = {26372, 15219, 53931, 50406, 26072, 23469, 25002, 37812};

To encrypt the file, I have a loop which runs this code:

builder.append((char)(bR.read() ^ key[pos++]));
if(pos == 8) pos = 0; //returns to first position in key

This reads the next character from the file using a BufferedReader, xors it against the next character in the key and appends it to a StringBuilder.

Here is the entire code:

import java.awt.BorderLayout;
import java.awt.Toolkit;
import java.awt.event.ActionEvent;
import java.awt.event.ActionListener;
import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.File;
import java.io.FileReader;
import java.io.FileWriter;
import java.io.IOException;

import javax.swing.JButton;
import javax.swing.JFileChooser;
import javax.swing.JFrame;
import javax.swing.JOptionPane;
import javax.swing.JPanel;
import javax.swing.JProgressBar;
import javax.swing.JScrollPane;
import javax.swing.JTextArea;
import javax.swing.SwingUtilities;
import javax.swing.SwingWorker;
import javax.swing.UIManager;


public class Encryptor implements Runnable {

    /**
     * @param args
     */
    public static void main(String[] args) {
        try {
            SwingUtilities.invokeAndWait(new Encryptor());
        } catch(Exception e) {
            e.printStackTrace();
        }
    }

    JFrame frame;
    JButton bCancel, bEncrypt;
    JTextArea statusArea;
    JScrollPane statusScroll;
    JProgressBar progressBar;
    String fileContents, fileType, fileName;
    EncryptSwingWorker encryptWorker;
    char[] key = {
        26372, 15219, 53931, 50406, 26072, 23469, 25002, 37812};

    public void run() {

        try {
            UIManager.setLookAndFeel(UIManager.getSystemLookAndFeelClassName());
        } catch(Exception e) {}

        frame = new JFrame();

        statusArea = new JTextArea();
        statusArea.setEditable(false);
        statusScroll = new JScrollPane(statusArea);
        progressBar = new JProgressBar();
        progressBar.setValue(0);

        bCancel = new JButton("Cancel");
        bCancel.setEnabled(false);
        bCancel.addActionListener(new ActionListener() {

            @Override
            public void actionPerformed(ActionEvent arg0) {
                int option = JOptionPane.showOptionDialog(frame,
                        "Are you sure you want to cancel?",
                        "Warning", JOptionPane.YES_NO_OPTION,
                        JOptionPane.QUESTION_MESSAGE, null,
                        null, null
                );
                if(option == 0) {
                    encryptWorker.cancel(true);
                    statusArea.append("Encryption cancelled\n");
                    progressBar.setValue(0);
                    progressBar.setIndeterminate(false);
                    bCancel.setEnabled(false);
                    bEncrypt.setEnabled(true);
                }
            }

        });
        bEncrypt = new JButton("Encrypt file...");
        bEncrypt.addActionListener(new ActionListener() {

            @Override
            public void actionPerformed(ActionEvent e) {
                encryptWorker = Encryptor.this.new EncryptSwingWorker();
                encryptWorker.execute();
            }

        });

        JPanel top = new JPanel(new BorderLayout());
        top.add(bEncrypt, BorderLayout.CENTER);
        top.add(bCancel, BorderLayout.EAST);
        frame.getContentPane().add(top, BorderLayout.NORTH);
        frame.getContentPane().add(statusScroll, BorderLayout.CENTER);
        frame.getContentPane().add(progressBar, BorderLayout.SOUTH);

        frame.setSize(200, 150);
        frame.setResizable(false);
        frame.setTitle("sEncryptor");
        frame.setVisible(true);
        frame.setDefaultCloseOperation(JFrame.DISPOSE_ON_CLOSE);
        frame.setIconImage(Toolkit.getDefaultToolkit().getImage("encryptor.png"));
    }

    void encrypt() throws IOException {
        JFileChooser fc = new JFileChooser();
        int returnVal = fc.showOpenDialog(frame);

        if(returnVal == JFileChooser.APPROVE_OPTION) {
            if(fc.getSelectedFile() == null) {
                JOptionPane.showOptionDialog(frame,
                    "You did not select a file",
                    "File Open Error",
                    JOptionPane.DEFAULT_OPTION,
                    JOptionPane.WARNING_MESSAGE, null,
                    null, null
                );
                encrypt();
                return;
            }

            bCancel.setEnabled(true);
            bEncrypt.setEnabled(false);

            fileName = fc.getSelectedFile().getPath();

            statusArea.append("Calculating size...\n");
            progressBar.setIndeterminate(true);
            int size = (int)(new File(fileName).length() / 32);
            progressBar.setIndeterminate(false);
            progressBar.setMaximum(size);

            BufferedReader bR = new BufferedReader(new FileReader(fc.getSelectedFile()));
            StringBuilder builder = new StringBuilder();

            statusArea.append("Encrypting...\n");
            int pos = 0;
            for(double d = 0; bR.ready(); d += 0.03125) {
                if(encryptWorker.isCancelled()) return;
                progressBar.setValue((int)d);
                builder.append((char)(bR.read() ^ key[pos++]));
                if(pos == 8) pos = 0;
            }

            progressBar.setValue(0);
            progressBar.setIndeterminate(true);

            fileContents = builder.toString();
            bR.close();
            statusArea.append("Encryption finished\n");
            progressBar.setIndeterminate(false);
        }
    }

    void save(String file) throws IOException {
        statusArea.append("Saving...\n");
        progressBar.setIndeterminate(true);
        BufferedWriter bW = new BufferedWriter(new FileWriter(file));
        bW.write(fileContents);
        bW.close();
        progressBar.setIndeterminate(false);
        statusArea.append("Done!\n");
    }

    void saveAs() throws IOException {
        JFileChooser fc = new JFileChooser();
        int returnVal = fc.showSaveDialog(frame);

        if(returnVal == JFileChooser.APPROVE_OPTION) {
            if(fc.getSelectedFile() == null) {
                JOptionPane.showOptionDialog(frame,
                    "You did not select a file",
                    "File Open Error",
                    JOptionPane.DEFAULT_OPTION,
                    JOptionPane.WARNING_MESSAGE, null,
                    null, null
                );
                saveAs();
                return;
            }

            save(fc.getSelectedFile().getPath());
        }
    }

    class EncryptSwingWorker extends SwingWorker<Void, Void> {

        @Override
        protected Void doInBackground() throws Exception {
            encrypt();
            if(EncryptSwingWorker.this.isCancelled()) return null;
            bCancel.setEnabled(false);
            while(true) {
                Object[] options = {"Override", "Save As...", "Cancel"};
                int option = JOptionPane.showOptionDialog(frame,
                        "Override existing file?",
                        "Save", JOptionPane.YES_NO_CANCEL_OPTION,
                        JOptionPane.QUESTION_MESSAGE, null,
                        options, options[0]
                );
                if(option == 0)
                    try {
                        save(fileName);
                    } catch(IOException e1) {
                        e1.printStackTrace();
                    }
                else if(option == 1)
                    try {
                        saveAs();
                    } catch(IOException e1) {
                        e1.printStackTrace();
                    }
                else {
                    int option1 = JOptionPane.showOptionDialog(frame,
                            "The encryption will be lost if you continue\n" +
                            "Are you sure you want to cancel?",
                            "Warning", JOptionPane.YES_NO_OPTION,
                            JOptionPane.QUESTION_MESSAGE, null,
                            null, null
                    );
                    if(option1 == 1) continue;
                    break;
                }
                break;
            }
            fileContents = "";
            bEncrypt.setEnabled(true);
            return null;
        }

    }
}

Is there something I'm missing in my code, or does xoring the entire file cause problems?

Answer 1

Got it. The problem is in the file encoding, as always :). Never, ever, depend on default encoding - it will eventually backfire at you.

Since Reader and Writer classes are made for plaintext inputs/outputs, it gets messed up. In this case, the Writer doesn't know what to do with the various integer numbers it gets and ends up failing. When you fix the Writer, you must fix the Reader, too, because the encrypted data may not by only valid characters...

The best solution is to ignore characters, use bytes and FileInputStream, FileOutputStream (wrapped by their Buffered brothers if necessary), because those are made to handle binary data. That way, you'll end up reading/writing exactly the data you intend to read/write. Try to hack it together yourself, it's a great lesson. If you'll get stuck hard, look here.

The easiest solution is to change only two lines. Note that it will, again, only work for text files! Seriously, don't mess random numbers and printable chars together.

BufferedReader bR = new BufferedReader(new FileReader(fc.getSelectedFile()));

to

BufferedReader bR = new BufferedReader(new InputStreamReader(new FileInputStream(fc.getSelectedFile()), "UTF-8"));

and

BufferedWriter bW = new BufferedWriter(new FileWriter(file));

to

BufferedWriter bW = new BufferedWriter(new OutputStreamWriter(new FileOutputStream(file), "UTF-8"));