CODEWITHSUNDEEP
elasticsearchkibanafilebeattshark
Alfred Balle
Alfred Balle

Reputation: 1195

Importing PCAP into Elasticsearch

I'm trying out Elasticsearch for the very first time.

I've downloaded Elasticsearch and Kibana and everything seems to run fine. I can visit http://localhost:5601 and view Kibana without errors.

I've made some traces with wireshark/tshark and converted it into Elasticsearch format with:

tshark -r test_trace.pcap -T ek > test_trace.pcap.json

Now I'm trying to import that .json into Elasticsearch, but it seems to fail:

curl -s -H "Content-Type: application/x-ndjson" -XPOST "localhost:9200/foo/_bulk" --data-binary "@/Users/test-elastic/test_trace.pcap.json"

I'm getting no errors or any output, but visiting Kibana shows index_not_found_exception and running:

curl 'http://127.0.0.1:9200/foo/_search/?size=10&pretty=true'

Outputs

{
  "error" : {
    "root_cause" : [
      {
        "type" : "index_not_found_exception",
        "reason" : "no such index",
        "resource.type" : "index_or_alias",
        "resource.id" : "foo",
        "index_uuid" : "_na_",
        "index" : "foo"
      }
    ],
    "type" : "index_not_found_exception",
    "reason" : "no such index",
    "resource.type" : "index_or_alias",
    "resource.id" : "foo",
    "index_uuid" : "_na_",
    "index" : "foo"
  },
  "status" : 404
}

How can I import my data correctly and view it in Elasticsearch and Kibana?

The JSON file is 195MB, converted from 10MB PCAP file. Output of first lines in json-file is:

{"index" : {"_index": "packets-2019-02-15", "_type": "pcap_file", "_score": null}}
{"timestamp" : "1549540104875", "layers" : {"frame": {"frame_frame_interface_id":...

UPDATE

After removing -s in curl I'm getting output:

HTTP/1.1 413 Request Entity Too Large

Now I've tried to use split to split the files into mulitple smaller files.

Testing import again now gives me multiple errors with:

..."reason":"failed to parse","caused_by":{"type":"json_parse_exception","reason":"Duplicate field 'ip_ip_addr'\n at [Source: org.elasticsearch.common.bytes.BytesReference$MarkSupportingStreamInputWrapper@5d2f82db; line: 1, column: 1300...

UPDATE

I used the following command on my test_trace.pcap.json to get smaller files:

split -l 10000 -a 10 test_trace.pcap.json.pcap.json ./tmp/test_trace.pcap

Then I got lots of files and tested import wit the first file:

./tmp/test_trace.pcapaaaaaaaaaa

The file type in my .json is:

"frame_frame_protocols": "sll:ethertype:ip:sctp"

and there are indeed multiple ip_ip_addr fields, as I have source and destination ip addresses in the traces.

Upvotes: 3

Views: 6502

Answers (1)

Val
Val

Reputation: 217544

Your JSON file already contains the index into which the data is supposed to be indexed, i.e. packets-2019-02-15, so your query should simply be:

curl 'http://127.0.0.1:9200/packets-2019-02-15/_search/?size=10&pretty=true'

However, I doubt that you can send a 195MB file in one go, I suggest you split it and load it in chunks

Upvotes: 1

Related Questions