CODEWITHSUNDEEP
rparsingpdf
matsuo_basho
matsuo_basho

Reputation: 3030

Parsing PDF by paragraph in R using pdftools

I am trying to parse a PDF document by paragraph in R. I have the PDF saved on my local machine. Thus, please download the sample pdf from the Apple website.

require(pdftools)

apple <- pdf_text('apple.pdf')

apple[[26]]

The issue is that if we examine the 26th page, each line terminates with an '\r\n'. This is no different than the \r\n between the end of the first paragraph (in italics) and the Overview and Highlights paragraph. In the PDF, it does appear that 2 lines are skipped, but the object in R doesn't reflect that.

I cannot figure out whether this is a function of this particular package, or whether in fact the conversion to text eliminates these paragraph markers. I haven't been able to set up import using other methods (ex. using the tm package)

Upvotes: 2

Views: 1746

Answers (1)

treysp
treysp

Reputation: 713

I think it's an underlying property of the document (not of the general text conversion process or of pdftools).

If you use your mouse to select text across paragraph breaks, it doesn't pick up the blank lines, suggesting that they are part of the PDF's layout metadata and not the text itself (though I don't actually know anything about PDF file specs):

selected paragraph break

Your best bet may be coming up with heuristic rule-sets to identify paragraph breaks. I'm thinking something like:

  • Previous line ends with a period then \r\n
  • Paragraph title line is short, ends without a period, then \r\n
  • First sentence of paragraph starts with a capital letter and takes up the full line

Upvotes: 2

Related Questions