CODEWITHSUNDEEP
pdfelasticsearchelasticsearch-plugin
jbigdata.fr
jbigdata.fr

Reputation: 31

ElasticSearch 5.0.0 ingest-attachment plugin issues to index PDF

See this post

My Env:

{   "name" : "node-0",
    "cluster_name" : "ES500-JBD-0",  
    "cluster_uuid" : "q_akJRkrSI-glTwT5vfH4A",  
  "version" : {
    "number" : "5.0.0",
    "build_hash" : "253032b",
    "build_date" : "2016-10-26T04:37:51.531Z",
    "build_snapshot" : false,
    "lucene_version" : "6.2.0"   },
  "tagline" : "You Know, for Search"
}

Index & pipeline creation (Edit 3):

curl -XPUT 'vm01.jbdata.fr:9200/_ingest/pipeline/attachment' -d '{
  "description" : "Extract attachment information",
  "processors" : [
    {
      "attachment" : {
        "field" : "data",
        "indexed_chars" : -1
      }
    }
  ]
}'

Mapping creation (Edit 4) with french :

curl -XPUT 'vm01.jbdata.fr:9200/ged-idx-00' -d '{
  "mappings" : {
    "ged_type_0" : {
      "properties" : {
         "attachment.data" : {
            "type": "text",
            "analyzer" : "french"
            }
         }
      }
   }
}'

ES specific config (Edit 1 & Edit 2):

$ bin/elasticsearch-plugin list
ingest-attachment

From config/elasticsearch.yml

plugin.mandatory: ingest-attachment

CommandS to index a PDF:

1/ A "raw" PDF.

curl -H 'Content-Type: application/pdf' -XPUT vm01.jbdata.fr:9200/ged-idx-00?pipeline=attachment -d @/tmp/zookeeperAdmin.pdf

{"error":{"root_cause":[{"type":"settings_exception","reason":"Failed to load settings from [%PDF-1.4%��� ... 0D33957F>]>>startxref76764%%EOF; line: 1, column: 2]"}},"status":500}

2/ A "B64ed" PDF.

aPath='/tmp/zookeeperAdmin.pdf'
aB64content=$(base64 $aPath | perl -pe 's/\n/\\n/g')
echo $aB64content > /tmp/zookeeperAdmin.pdf.b64
curl -XPUT "http://vm01.jbdata.fr:9200/ged-idx-00?pipeline=attachment" -d '{
    "file" : "content" : "'$aB64content'"
}'

{"error":{"root_cause":... "reason":"failed to parse source for create index","caused_by":{"type":"json_parse_exception","reason":"Unexpected character (':' (code 58)): was expecting comma to separate Object entries\n at [Source: org.elasticsearch.transport.netty4.ByteBufStreamInput@65a254b6; line: 2, column: 25]"}},"status":400}

How to use correctly the ingest-attachment plugin ton index PDF ?

Upvotes: 0

Views: 1374

Answers (2)

Ilia P
Ilia P

Reputation: 646

In fact, it's quite difficult to extract text from PDF properly, often you have to extract inline images or render the whole page and OCR it depending on the text extracted from the page and it's content (for example you have to analyse whether encoding is right or not). You simply can not tune Tika to use any custom logic inside parsing process, neither you can't do so with Ingest Attachment. If you're aiming at a good quality PDF parsing - Ingest Attachment is not what you're looking for, you have to do it yourself.

Read the full story here: https://blog.ambar.cloud/ingest-attachment-plugin-for-elasticsearch-should-you-use-it/

Upvotes: 1

Sylvain
Sylvain

Reputation: 11

From my experience, the file needs to be encoded in Base64, so your option 2 should be the good way to go.

About your last attempt:

curl -XPUT "http://vm01.jbdata.fr:9200/ged-idx-00?pipeline=attachment" -d '{
    "file" : "content" : "'$aB64content'"
}'

The provided JSON is malformed ("a" : "b" : "c"), hence the error.

As specified in your pipeline creation, you only need a data field, so the following should do the trick:

curl -XPUT "http://vm01.jbdata.fr:9200/ged-idx-00?pipeline=attachment" -d '{
    "data" : "'$aB64content'"
}'

Upvotes: 1

Related Questions