CODEWITHSUNDEEP
amazon-web-servicesamazon-ec2amazon-kinesisamazon-kinesis-agent
PickleRick
PickleRick

Reputation: 31

AWS Kinesis - Kinesis Agent not parsing records

I've created a kinesis stream, installed a kinesis agent on my EC2 instance and configured it to read files from a certain directory.

Kinesis Agent Configuration file:

{
  "cloudwatch.emitMetrics": true,
  "kinesis.endpoint": "kinesis.eu-central-1.amazonaws.com",
  "firehose.endpoint": "firehose.eu-central-1.amazonaws.com",
  "awsAccessKeyId": "ACCESSKEYID",
  "awsSecretAccessKey": "SECRETACCESSKEY",

  "flows": [
    {
      "filePattern": "/home/ec2-user/workspace/connect_s3/documents/*.log",
      "kinesisStream": "EntryPoint",
      "partitionKeyOption": "RANDOM"
    }
]
}

I've also given my EC2 instance the required IAM policies for Kinesis (I added the access keys later on for debug purposes).

Then I started my agent, added files and tailed the /var/log/aws-kinesis-agent/aws-kinesis-agent.log file, but nothing happens.

My agent starts correctly, but no matter how many files I add to the /home/ec2-user/workspace/connect_s3/documents/ directory, the agent can't seem to parse anything; it returns:

Tailer Progress: Tailer has parsed 0 records (0 bytes), transformed 0 records, skipped 0 records, and has successfully sent 0 records to destination.

I've been thinking it might be a permission problem, but I changed all permissions and everything is in 755 file permission.

However, in my AWS console, the steam metrics seem to be monitoring that things are happening IMAGE HERE -> (https://i.sstatic.net/n2ily.png)

As anyone encountered problems similar to this one ? In advance, thank you for reading.

Upvotes: 2

Views: 1192

Answers (1)

PickleRick
PickleRick

Reputation: 31

I found the problem literally seconds after posting this (sorry, it took me hours total to fix the problem).

For anyone having similar issues & if you're try this from an EC2 instance:

  • Remove your Source directory (filePattern) from /ec2-user -> put it in /var/
  • Add the "initialPosition": "START_OF_FILE" argument.

Here's my final agent.json file state:

{
  "cloudwatch.emitMetrics": true,
  "kinesis.endpoint": "kinesis.eu-central-1.amazonaws.com",
  "firehose.endpoint": "firehose.eu-central-1.amazonaws.com",
  "awsAccessKeyId": "********",
  "awsSecretAccessKey": "**************",

  "flows": [
    {
      "filePattern": "/var/documents/*.txt",
      "kinesisStream": "EntryPoint",
      "partitionKeyOption": "RANDOM",
      "initialPosition": "START_OF_FILE"
    }
]
}

Also, original answer came from here:

Kinesis agent not parsing the file

Upvotes: 1

Related Questions