CODEWITHSUNDEEP
c#.netml.netml.net-model-builder
Mario
Mario

Reputation: 14800

ML.Net Data.SaveAsText CSV file cannot be loaded by Model Builder

I export my current ML.Net Data to a CSV with this function

using (var stream = File.Create("c:\\temp\\aidata.csv"))
             mlContext.Data.SaveAsText(trainData, stream);

The saved data looks ok, but when I try to load the CSV with the ML.Net Model Builder I get this error:

Data Error: Unable to infer column types of the file provided.

Note: I also have vector columns in the CSV

The CSV file looks like this (I have removed some colums to be smaller).

If I load the data file with mlContext.Data.LoadFromTextFile("c:\temp\aidata.csv"); it loads it without any error, but the Model Builder fails to load it.

Upvotes: 0

Views: 1699

Answers (2)

CSDev
CSDev

Reputation: 3235

As a matter of fact the file can not be .csv just because it's saved with the extension. It needs transformation like this:

static class MLCsvHelper
{
    private class ColumnDefinition
    {
        private readonly int end;

        public string Name { get; }
        public int Start { get; }
        public int Count { get; }

        public ColumnDefinition(string name, int start, int count) =>
            (Name, Start, Count, end) = (name, start, count, start + count - 1);

        public override string ToString() =>
            $"{Name}:\"{Start}:{end}\"";
    }

    public static void Patch(string file, out string csv)
    {
        csv = Path.ChangeExtension(file, "patched.csv");
        var lines = File.ReadAllLines(file);

        var columns = lines.TakeWhile(line => line.Contains("#@"))
            .Where(line => line.Contains("col=")).Select(line => GetColumn(line))
            .ToArray();

        var data = lines.SkipWhile(line => line.Contains("#@")).Skip(1)
            .Select(line => line.Split('\t')).ToArray();

        var res = new[] { string.Join("\t", columns.Select(column => column.Name)) }
            .Concat(data.Select(item => string.Join("\t", columns.Select(column => GetValue(column, item)))));

        File.WriteAllLines(csv, res.ToArray());
    }

    private static ColumnDefinition GetColumn(string line)
    {
        var items = line.Split(new[] { '=', ':' });
        var name = items[1];
        var range = items.Last().Split('-');
        var start = int.Parse(range.First());
        var last = int.Parse(range.Last());
        var count = last - start + 1;
        return new ColumnDefinition(name, start, count);
    }

    private static string GetValue(ColumnDefinition column, string[] data)
    {
        var chunk = data.Skip(column.Start).Take(column.Count);
        var value = string.Join("\t", chunk);
        if (chunk.Skip(1).Any())
            value = $"\"{value}\"";
        return value;
    }
}

MLCsvHelper.Patch("zvVEYT", out var csv);

Upvotes: 1

D. Dahlberg
D. Dahlberg

Reputation: 156

As @jdweng said in comments. The file you provided is not in a correct csv format (columns and values are seperated by ";"). It does however look like tsv format(columns and values are seperated by tabs).

It should work if you try saving your text as a tsv file instead.

Also, the ML.Net video example is using tsv format.

Upvotes: 0

Related Questions