How to store and serialize a List to avoid long strings

Question

I am having some problems storing information. The problem comes because when saving lists I get very long strings even if the information looks short.

Lets say I have 100 houses and I want to save how many people are on each house. I use this:

Houses.cs

[System.Serializable]
public class Houses
{
    public int ID { get; set; }    
    public int PEOPLE { get; set; }

    public Houses(int _ID, int _PEOPLE)
    {
        ID = _ID;
        PEOPLE = _PEOPLE ;   
    }
}

Then in another script I have:

public List<Houses> HousesList = new List<Houses>();
void Awake()
{
    HousesList.Add(new Houses(0,0));
    //repeat until 100 (or more) I use a loop to initialize it
}

Then to add people I say:

HousesList[3].PEOPLE+=5;

Then to save I use the BinaryFormatter/MemoryStream way but even if the list has only one ID and one value for the people the resulting serialized string has 4,000-5,000 characters.

To store a small amount of data I use the same system but only with one List:

Houses.Add(new Houses(0,2,0,0,1)); // house 1, house 2... house 5

But this way even if there are 100 houses the string is short but is confusing with so many numbers. And later I can have problems with save/load if I add more houses.

Is there any way to manage this kind of data and save it in a much shorter string?

Thank you.

Answer 1

You could even simply save it as a list of lines each lines of the form

id people

so you have a file like e.g.

0 3
1 4
2 5
...

and later you can parse it via

var lines = fileContent.Split('/n');

and then on each line

var parts = line.Split(' ');
int id = int.TryParse(parts[0]) ? id : -1;
int people = int.TryParse(parts[1]) ? people : -1;

So in a code e.g.

[Serializable]
public class House
{
    public int Id;
    public int People;

    public House(int id, int people)
    {
        Id = id;
        People = people;   
    }
}

List<House> Houses = new List<House>();

public void Save()
{
    var stringBuilder = new StringBuilder();

    foreach(var house in Houses)
    {
        stringBuilder.Append(house.Id).Append(' ').Append(house.People).Append('/n');
    }

    // Now use the file IO method of your choice e.g.
    File.WriteAllText(filePath, stringBuilder.ToString(), Encoding.UTF8);   
}

public void Load()
{
    // clear current list
    Houses.Clear();

    // Use the file IO of choice e.g.
    string readText = File.ReadAllText(path);

    var lines = readText.Split('/n', StringSplitOptions.RemoveEmptyEntries);

    foreach (var line in lines)
    {
        var parts = line.Split(' ');

        // skip wrong formatted line
        if(parts.Length != 2) continue;

        int id = int.TryParse(parts[0]) ? id : -1;
        int people = int.TryParse(parts[1]) ? people : -1;

        if(id < 0 || people < 0) continue;    

        Houses.Add(new House(id, people));
    }
}

and now you have your houses with id and people count.

The string length depends on your values ofcourse but would be something like (for 100 houeses with people count > 9 and < 100)

house IDs + seperator + people  + /n
10+89*2   + 100       + 100 * 2 + 100

≈ 588 characters | bytes

Answer 2

Here's my attempt with Json.Net + GZipStream

Json length: characters=2181, bytes=2181
gzipped.Length=437

class Program
{
  public class Houses
  {
    public int Id { get; } // Btw. No need for setters
    public int People { get; } 

    public Houses(int id, int people)
    {
      Id = id;
      People = people;
    }
  }

  static void Main(string[] args)
  {
    var houses = new List<Houses>();
    for (int i = 0; i < 100; i++)
      houses.Add(new Houses(i, i));

    var json = Newtonsoft.Json.JsonConvert.SerializeObject(houses);
    byte[] inputBytes = Encoding.UTF8.GetBytes(json);

    Console.WriteLine($"Json length: characters={json.Length}, bytes={inputBytes.Length}");

    byte[] gzipped;
    using (var outputStream = new MemoryStream())
    {
      using (var gZipStream = new GZipStream(outputStream, CompressionMode.Compress))
      {
        gZipStream.Write(inputBytes, 0, inputBytes.Length);
      }
      gzipped = outputStream.ToArray();

    }
    Console.WriteLine($"gzipped.Length={gzipped.Length}");
  }
}

Answer 3

BinaryFormatter is not known for being an efficient serializer; in fact, you really, really shouldn't use it; frankly ever. There are a lot of known concerns with it.

If space is you whimsy, protobuf is probably a good bet; protobuf looks to be about 596 bytes for this; in this case I'm using protobuf-net:

596 bytes
100 items
95, 95
96, 96
97, 97
98, 98
99, 99

with code:

using ProtoBuf;
using System.Collections.Generic;
using System.IO;
using System.Linq;

static class P
{
    static void Main()
    {
        var houses = new List<Houses>();
        for (int i = 0; i < 100; i++)
            houses.Add(new Houses(i, i));
        using(var ms = new MemoryStream())
        {
            Serializer.Serialize(ms, houses);
            System.Console.WriteLine($"{ms.Length} bytes");
            ms.Position = 0;
            var clone = Serializer.Deserialize<List<Houses>>(ms);
            System.Console.WriteLine($"{clone.Count} items");
            foreach(var item in clone.Skip(95))
            {
                System.Console.WriteLine($"{item.Id}, {item.People}");
            }
        }
    }
}
[ProtoContract(SkipConstructor = true)]
public class Houses
{
    [ProtoMember(1)]
    public int Id { get; set; }
    [ProtoMember(2)]
    public int People { get; set; }
    public Houses(int id, int people)
    {
        Id = id;
        People = people;
    }
}