c# How to run a application faster

Question

I am creating a word list of possible uppercase letters to prove how insecure 8 digit passwords are this code will write aaaaaaaa to aaaaaaab to aaaaaaac etc. until zzzzzzzz using this code:

class Program
{
    static string path;
    static int file = 0;
    static void Main(string[] args)
    {
        new_file();
        var alphabet = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ123456789+-*_!$£^=<>§°ÖÄÜöäü.;:,?{}[]";
        var q = alphabet.Select(x => x.ToString());
        int size = 3;
        int counter = 0;

        for (int i = 0; i < size - 1; i++)
        {
            q = q.SelectMany(x => alphabet, (x, y) => x + y);
        }
        foreach (var item in q)
        {
            if (counter >= 20000000)
            {
                new_file();
                counter = 0;
            }
            if (File.Exists(path))
            {
                using (StreamWriter sw = File.AppendText(path))
                {
                    sw.WriteLine(item);
                    Console.WriteLine(item);
                    /*if (!(Regex.IsMatch(item, @"(.)\1")))
                    {
                        sw.WriteLine(item);
                        counter++;
                    }
                    else
                    {
                        Console.WriteLine(item);
                    }*/
                }
            }
            else
            {
                new_file();
            }
        }
    }

    static void new_file()
    {
        path = @"C:\" + "list" + file + ".txt";
        if (!File.Exists(path))
        {
            using (StreamWriter sw = File.CreateText(path))
            {
            }
        }
        file++;
    }
}

The Code is working fine but it takes Weeks to run it. Does anyone know a way to speed it up or do I have to wait? If anyone has a idea please tell me.

Answer 1

Performance:

size 3:   0.02s
size 4:   1.61s
size 5: 144.76s

Hints:

• removed LINQ for combination generation
• removed Console.WriteLine for each password
• removed StreamWriter
• large buffer (128k) for file writing

---

  const string alphabet = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ123456789+-*_!$£^=<>§°ÖÄÜöäü.;:,?{}[]";
  var byteAlphabet = alphabet.Select(ch => (byte)ch).ToArray();
  var alphabetLength = alphabet.Length;

  var newLine = new[] { (byte)'\r', (byte)'\n' };

  const int size = 4;
  var number = new byte[size];
  var password = Enumerable.Range(0, size).Select(i => byteAlphabet[0]).Concat(newLine).ToArray();

  var watcher = new System.Diagnostics.Stopwatch();
  watcher.Start();

  var isRunning = true;
  for (var counter = 0; isRunning; counter++)
  {
    Console.Write("{0}: ", counter);
    Console.Write(password.Select(b => (char)b).ToArray());
    using (var file = System.IO.File.Create(string.Format(@"list.{0:D5}.txt", counter), 2 << 16))
    {
      for (var i = 0; i < 2000000; ++i)
      {
        file.Write(password, 0, password.Length);
        var j = size - 1;
        for (; j >= 0; j--)
        {
          if (number[j] < alphabetLength - 1)
          {
            password[j] = byteAlphabet[++number[j]];
            break;
          }
          else
          {
            number[j] = 0;
            password[j] = byteAlphabet[0];
          }
        }
        if (j < 0)
        {
          isRunning = false;
          break;
        }
      }
    }
  }
  watcher.Stop();
  Console.WriteLine(watcher.Elapsed);
}

Answer 2

your alphabet is missing 0

With that fixed there would be 89 chars in your set. Let's call it 100 for simplicity. The set you are looking for is all the 8 character length strings drawn from that set. There are 100^8 of these, i.e. 10,000,000,000,000,000.

The disk space they will take up depends on how you encode them, lets be generous - assume you use some 8 bit char set that contains the these characters, and you don't put in carriage returns, so one byte per char, so 10,000,000,000,000,000 bytes =~ 10 peta byes?

Do you have 10 petabytes of disk? (10000 TB)?

[EDIT] In response to 'this is not an answer':

The original motivation is to create the list? The shows how large the list would be. Its hard to see what could be DONE with the list if it was actualised, i.e. it would always be quicker to reproduce it than to load it. Surely whatever point could be made by producing the list can also be made by simply knowing it's size, which the above shows how to work it out.

There are LOTS of inefficiencies in you code, but if your questions is 'how can i quickly produce this list and write it to disk' the answer is 'you literally cannot'.

[/EDIT]

Answer 3

Try the following modified code. In LINQPad it runs in < 1 second. With your original code I gave up after 40 seconds. It removes the overhead of opening and closing the file for every WriteLine operation. You'll need to test and ensure it gives the same results because I'm not willing to run your original code for 24 hours to ensure the output is the same.

class Program
{
    static string path;
    static int file = 0;
    static void Main(string[] args)
    {
        new_file();
        var alphabet = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ123456789+-*_!$£^=<>§°ÖÄÜöäü.;:,?{}[]";
        var q = alphabet.Select(x => x.ToString());
        int size = 3;
        int counter = 0;

        for (int i = 0; i < size - 1; i++)
        {
            q = q.SelectMany(x => alphabet, (x, y) => x + y);
        }

        StreamWriter sw = File.AppendText(path);

        try
        {
            foreach (var item in q)
            {
                if (counter >= 20000000)
                {
                    sw.Dispose();
                    new_file();
                    counter = 0;
                }
                sw.WriteLine(item);
                Console.WriteLine(item);
            }
        }
        finally
        {
            if(sw != null)
            {
                sw.Dispose();
            }
        }
    }

    static void new_file()
    {
        path = @"C:\temp\list" + file + ".txt";
        if (!File.Exists(path))
        {
            using (StreamWriter sw = File.CreateText(path))
            {
            }
        }
        file++;
    }
}