Connecting to AWS Kinesis with erlang library kinetic

Question

I am trying to connect to kinesis using the erlang library kinetic. https://github.com/AdRoll/kinetic... my development.config has my aws key and secret in it, however I am not sure what the metadata_base_url should be or what else I need in order to make it work...currently i have:

%% -*- erlang -*-
[{kinetic,
    [{args, [
        % All of these values are optional
        % kinetic will get all of the context from the instance
        {metadata_base_url, "https://kinesis.us-east-1.amazonaws.com"},
        {aws_access_key_id, "mykey"},
        {aws_secret_access_key, "mysecret"},
        {iam_role, "kinetic"},

        {lhttpc_opts, [{max_connections, 5000}]}
    ]}]
}].

Below is my results when trying to start it...

kinetic (master) $ make
==> lhttpc (get-deps)
==> jiffy (get-deps)
==> meck (get-deps)
==> kinetic (get-deps)
==> lhttpc (compile)
==> jiffy (compile)
==> meck (compile)
==> kinetic (compile)
Compiled src/kinetic.erl
kinetic (master) $ erl -pa ebin -pa deps/*/ebin -s inets -s crypto -s ssl -s lhttpc -config development -s kinetic
Erlang R16B03-1 (erts-5.10.4) [source] [64-bit] [smp:8:8] [async-threads:10] [hipe] [kernel-poll:false] [dtrace]

Eshell V5.10.4  (abort with ^G)
1> 
=INFO REPORT==== 2-Dec-2014::11:51:31 ===
    application: kinetic
    exited: {{shutdown,
                 {failed_to_start_child,kinetic_config,
                     {{badmatch,{error,403}},
                      [{kinetic_config,new_args,1,
                           [{file,"src/kinetic_config.erl"},{line,127}]},
                       {kinetic_config,update_data,1,
                           [{file,"src/kinetic_config.erl"},{line,42}]},
                       {kinetic_config,init,1,
                           [{file,"src/kinetic_config.erl"},{line,55}]},
                       {gen_server,init_it,6,
                           [{file,"gen_server.erl"},{line,304}]},
                       {proc_lib,init_p_do_apply,3,
                           [{file,"proc_lib.erl"},{line,239}]}]}}},
             {kinetic,start,[normal,[]]}}
    type: temporary

When removed the base_url config...

=ERROR REPORT==== 2-Dec-2014::12:41:30 ===
{failed_connect,[{to_address,{"169.254.169.254",80}},{inet,[inet],etimedout}]}

=INFO REPORT==== 2-Dec-2014::12:41:30 ===
    application: kinetic
    exited: {{shutdown,
                 {failed_to_start_child,kinetic_config,
                     {{badmatch,
                          {error,
                              {failed_connect,
                                  [{to_address,{"169.254.169.254",80}},
                                   {inet,[inet],etimedout}]}}},
                      [{kinetic_config,new_args,1,
                           [{file,"src/kinetic_config.erl"},{line,127}]},
                       {kinetic_config,update_data,1,
                           [{file,"src/kinetic_config.erl"},{line,42}]},
                       {kinetic_config,init,1,
                           [{file,"src/kinetic_config.erl"},{line,55}]},
                       {gen_server,init_it,6,
                           [{file,"gen_server.erl"},{line,304}]},
                       {proc_lib,init_p_do_apply,3,
                           [{file,"proc_lib.erl"},{line,239}]}]}}},
             {kinetic,start,[normal,[]]}}
    type: temporary

Answer 1

It seems that in case of running kinetic application outside of ec2 cluster you need to specify region in config:

[{kinetic,
    [{args, [
         {region, "us-east-1"}, %% just an example
         ...
    ]}]
}].

and use fixed version of kinetic which won't be trying to discover region.

Second solution is to set metadata_base_url option to your http service which on get request for "/latest/meta-data/placement/availability-zone" will respond with your region.

I've never used aws and some of statements might be improper.

Answer 2

The error message says that the access to https://kinesis.us-east-1.amazonaws.com/latest/meta-data/placement/availability-zone return an error 403: Forbidden. Reading http documentation it means that the authentication was correct but that you do not have the right access to this ressource.