How to parse PDF files in map reduce programs?

Question

I want to parse PDF files in my hadoop 2.2.0 program and I found this, followed what it says and until now, I have these three classes:

PDFWordCount: the main class containing map and reduce functions. (just like native hadoop wordcount sample but instead of TextInputFormat I used my PDFInputFormat class.

PDFRecordReader extends RecordReader: Which is the main work here. Especially I put my initialize function here for more illustration.

public void initialize(InputSplit genericSplit, TaskAttemptContext context)
      throws IOException, InterruptedException {
      System.out.println("initialize");
      System.out.println(genericSplit.toString());
    FileSplit split = (FileSplit) genericSplit;
    System.out.println("filesplit convertion has been done");
    final Path file = split.getPath();
    Configuration conf = context.getConfiguration();
    conf.getInt("mapred.linerecordreader.maxlength", Integer.MAX_VALUE);
    FileSystem fs = file.getFileSystem(conf);
    System.out.println("fs has been opened");
    start = split.getStart();
    end = start + split.getLength();
    System.out.println("going to open split");
    FSDataInputStream filein = fs.open(split.getPath());
    System.out.println("going to load pdf");
    PDDocument pd = PDDocument.load(filein);
    System.out.println("pdf has been loaded");
    PDFTextStripper stripper = new PDFTextStripper();
    in =
    new LineReader(new ByteArrayInputStream(stripper.getText(pd).getBytes(
        "UTF-8")));
    start = 0;
    this.pos = start;
    System.out.println("init has finished");
  }

(You can see my system.out.printlns for debugging. This method fails in converting genericSplit to FileSplit. Last thing I see in console, is this:

hdfs://localhost:9000/in:0+9396432

which is genericSplit.toString()

PDFInputFormat extends FileInputFormat: which just creates new PDFRecordReader in createRecordReader method.

I want to know what is my mistake?

Do I need extra classes or something?

padmalcom · Accepted Answer

Reading PDFs is not that difficult, you need to extend the class FileInputFormat as well as the RecordReader. The FileInputClass should not be able to split PDF files since they are binaries.

public class PDFInputFormat extends FileInputFormat {

  @Override
  public RecordReader createRecordReader(InputSplit split,
    TaskAttemptContext context) throws IOException, InterruptedException {
      return new PDFLineRecordReader();
  }

  // Do not allow to ever split PDF files, even if larger than HDFS block size
  @Override
  protected boolean isSplitable(JobContext context, Path filename) {
    return false;
  }

}

The RecordReader then performs the reading itself (I am using PDFBox to read PDFs).

public class PDFLineRecordReader extends RecordReader {

private Text key = new Text();
private Text value = new Text();
private int currentLine = 0;
private List lines = null;

private PDDocument doc = null;
private PDFTextStripper textStripper = null;

@Override
public void initialize(InputSplit split, TaskAttemptContext context)
        throws IOException, InterruptedException {

    FileSplit fileSplit = (FileSplit) split;
    final Path file = fileSplit.getPath();

    Configuration conf = context.getConfiguration();
    FileSystem fs = file.getFileSystem(conf);
    FSDataInputStream filein = fs.open(fileSplit.getPath());

    if (filein != null) {

        doc = PDDocument.load(filein);

        // Konnte das PDF gelesen werden?
        if (doc != null) {
            textStripper = new PDFTextStripper();
            String text = textStripper.getText(doc);

            lines = Arrays.asList(text.split(System.lineSeparator()));
            currentLine = 0;

        }

    }
}

    // False ends the reading process
@Override
public boolean nextKeyValue() throws IOException, InterruptedException {

    if (key == null) {
        key = new Text();
    }

    if (value == null) {
        value = new Text();
    }

    if (currentLine < lines.size()) {
        String line = lines.get(currentLine);

        key.set(line);

        value.set("");
        currentLine++;

        return true;
    } else {

        // All lines are read? -> end
        key = null;
        value = null;
        return false;
    }
}

@Override
public Text getCurrentKey() throws IOException, InterruptedException {
    return key;
}

@Override
public Text getCurrentValue() throws IOException, InterruptedException {
    return value;
}

@Override
public float getProgress() throws IOException, InterruptedException {
    return (100.0f / lines.size() * currentLine) / 100.0f;
}

@Override
public void close() throws IOException {

    // If done close the doc
    if (doc != null) {
        doc.close();
    }

}

Hope this helps!

How to parse PDF files in map reduce programs?

Answers (2)

Related Questions