CODEWITHSUNDEEP
javanio
user3756506
user3756506

Reputation: 451

How to correctly read byte java file

I want to read fast line by line big csv files (approx ~ 1gb) in UTF-8. I have created a class for it, but it doesn't work properly. UTF-8 decodes Cyrillic symbol from 2 bytes. I use byte buffer to read it, for example, it has 10 bytes length. So if symbol composed from 10 and 11 bytes in the file it wouldn't be decoded normally :(

Tipical error

public class MyReader extends InputStream {

  private FileChannel channel;
  private ByteBuffer buffer = ByteBuffer.allocate(10);
  private int buffSize = 0;
  private int position = 0;
  private boolean EOF = false;
  private CharBuffer charBuffer;

  private MyReader() {}

  static MyReader getFromFile(final String path) throws IOException {
    MyReader myReader = new MyReader();
    myReader.channel = FileChannel.open(Path.of(path),
        StandardOpenOption.READ);
    myReader.initNewBuffer();
    return myReader;
  }
  private void initNewBuffer() {
    try {
      buffSize = channel.read(buffer);
      buffer.position(0);
      charBuffer = Charset.forName("UTF-8").decode(buffer);
      buffer.position(0);
    } catch (IOException e) {
      throw new RuntimeException("Error reading file: {}", e);
    }
  }
  @Override
  public int read() throws IOException {
    if (EOF) {
      return -1;
    }
    if (position < charBuffer.length()) {
      return charBuffer.array()[position++];
    } else {
      initNewBuffer();
      if (buffSize < 1) {
        EOF = true;
      } else {
        position = 0;
      }
      return read();
    }
  }
  public char[] readLine() throws IOException {
    int readResult = 0;
    int startPos = position;
    while (readResult != -1) {
      readResult = read();
    }
    return Arrays.copyOfRange(charBuffer.array(), startPos, position);
  }
}

Upvotes: 0

Views: 85

Answers (2)

Joop Eggen
Joop Eggen

Reputation: 109547

First: the gain is questionable.

The Files class has many nice and quite production fast methods.

Bytes with high bit 1 (< 0) are part of a UTF-8 multibyte sequence. With high bits 10 they are continuation bytes. Sequences might be upto 6 bytes nowadays (I believe).

So the next buffer starts with some continuation bytes, they belong to the previous buffer.

The programming logic I gladly leave to you.

Upvotes: 0

user3756506
user3756506

Reputation: 451

Bad solution, but it works)

private void initNewBuffer() {
    try {
      buffSize = channel.read(buffer);
      buffer.position(0);
      charBuffer = StandardCharsets.UTF_8.decode(buffer);
      if (buffSize > 0) {
        byte edgeByte = buffer.array()[buffSize - 1];
        if (edgeByte == (byte) 0xd0 ||
            edgeByte == (byte) 0xd1 ||
            edgeByte == (byte) 0xc2 ||
            edgeByte == (byte) 0xd2 ||
            edgeByte == (byte) 0xd3
        ) {
          channel.position(channel.position() - 1);
          charBuffer.limit(charBuffer.limit()-1);
        }
      }
      buffer.position(0);
    } catch (IOException e) {
      throw new RuntimeException("Error reading file: {}", e);
    }
  }

Upvotes: 1

Related Questions