CODEWITHSUNDEEP
javafileebcdiccomp-3
BaN3
BaN3

Reputation: 435

handling comp3 and ebcidic conversion in java to ASCII for large files

I am trying to convert comp3 and EBCIDIC characters in my java code but im running into out of memory exception as the amount of data handled is huge about 5 gb. my code is currently as follows:

byte[] data = Files.readAllBytes(path);

this is resulting in an out of memory exception which i can understand, but i cant use a file scanner as well since the data in the file wont be split into lines.

Can anyone point me in the correct direction on how to handle this

Note: the file may contain records of different length hence splitting it based on record length seams not possible.

Upvotes: 0

Views: 767

Answers (2)

user207421
user207421

Reputation: 310957

I'm running into out of memory exception as the amount of data handled is huge about 5 gb.

You only need to read one record at a time.

My code is currently as follows:

byte[] data = Files.readAllBytes(path);

This is resulting in an out of memory exception which i can understand

Me too.

but i cant use a file scanner as well since the data in the file wont be split into lines.

You mean you can't use the Scanner class? That's not the only way to read a record at a time.

In any case not all files have record delimiters. Some have fixed-length records, some have length words at the start of each record, and some have record type attributes at the start of each record, or in both cases at least in the fixed part of the record.

I'll have to split it based on an attribute record_id at a particular position(say at the begining of each record) that will tell me the record length

So read that attribute, decode it if necessary, and read the rest of the record according to the record length you derive from the attribute. One at a time.

I direct your attention to the methods of DataInputStream, especially readFully(). You will also need a Java COMP-3 library. There are several available. Most of the rest can be done by built-in EBCDIC character set decoders.

Upvotes: 0

Bruce Martin
Bruce Martin

Reputation: 10543

As Bill said you could (should) ask for the data to be converted to display characters on the mainframe and if English speaking you can do a ascii transfer.

Also how are you deciding where comp-3 fields start ???

You do not have to read the whole file into memory, you can still read the file in blocks, This method will fill an array of bytes:

protected final int readBuffer(InputStream in, final byte[] buf)
throws IOException {

    int total = 0;
    int num = in.read(buf, total, buf.length);

    while (num >= 0 && total + num < buf.length) {
        total += num;
        num = in.read(buf, total, buf.length - total);
    }
    return num;
}

if all the records are the same length, create an array of the record length and the above method will read one record at a time.

Finally the JRecord project has classes to read fixed length files etc. It can do comp-3 conversion. Note: I am the author of JRecord.

Upvotes: 0

Related Questions