CODEWITHSUNDEEP
amazon-kinesislocalstack
JavaTec
JavaTec

Reputation: 1044

localstack Kinesis does not start in docker

We're running Docker on Windows 10. We have a simple Java Springboot application, which should interact with AWS Kinesis and S3. For local testing, we want to pull the localstack Docker latest image and install and run. But the startup for Kinesis throws error whereas S3 works fine. Below are the details:

docker compose:
version: '3.1'
services:
  localstack:
    image: "localstack/localstack"
    container_name: localstack_demo
    environment:
      SERVICES: kinesis, s3, sqs
      DEBUG: 1
      DATA_DIR: /tmp/localstack
      KINESIS_STREAM_SHARDS: 1
      KINESIS_ERROR_PROBABILITY: 0.0
      KINESIS_STREAM_NAME: my-stream
    ports:
      - "4567-4584:4567-4584"
      - "8055:8080"
    volumes:
      - './.localstack:/tmp/localstack'
      - '/var/run/docker.sock:/var/run/docker.sock'

When we try to hit the default URL for Kinesis, http://localhost:4568, we get the following error: (please note that default URL for S3 responds as expected):

localstack_demo | 2019-09-24T22:35:27:ERROR:localstack.services.generic_proxy: Error forwarding request: the JSON object must be str, bytes or bytearray, not 'NoneType' Traceback (most recent call last):
localstack_demo |   File "/opt/code/localstack/localstack/services/generic_proxy.py", line 240, in forward
localstack_demo |     path=path, data=data, headers=forward_headers)
localstack_demo |   File "/opt/code/localstack/localstack/services/kinesis/kinesis_listener.py", line 22, in forward_request
localstack_demo |     data = json.loads(to_str(data))
localstack_demo |   File "/usr/lib/python3.6/json/__init__.py", line 348, in loads
localstack_demo |     'not {!r}'.format(s.__class__.__name__))
localstack_demo | TypeError: the JSON object must be str, bytes or bytearray, not 'NoneType'

My docker installation:

$ docker info
Containers: 3
 Running: 2
 Paused: 0
 Stopped: 1
Images: 169
Server Version: 18.09.9
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 4.14.141-boot2docker
Operating System: Boot2Docker 18.09.9 (TCL 8.2.1)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 989.4 MiB
Name: default
ID: QYOP:66HM:CLYX:77QX:UJES:37LA:JTFL:ER2U:2ZDA:HNNM:JWNN:CBOA
Docker Root Dir: /mnt/sda1/var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
 provider=virtualbox
Insecure Registries:
 127.0.0.0/8

Any pointers would be greatly appreciated

Upvotes: 5

Views: 2607

Answers (2)

Alex Shtromberg
Alex Shtromberg

Reputation: 837

So, I'm trying to verify(bring strong evidence) the answer, but apparently default kinesis provider doesn't work correctly on my Silicon M1(if you know what I mean). I noticed that kinesis provider kinesis-mock, the default one, is crashing immediately after the start...

Solution that works to me is to switch kinesis provider with environmental variables:

- KINESIS_PROVIDER=kinesalite

And the config may looks like from the answer before with addition:

version: '3.1'
services:
  localstack:
    image: "localstack/localstack"
    container_name: localstack_demo
    environment:
      - SERVICES=kinesis,s3,sqs
      - DEBUG=1
      - DATA_DIR=/tmp/localstack
      - KINESIS_STREAM_SHARDS=1
      - KINESIS_ERROR_PROBABILITY=0.0
      - KINESIS_STREAM_NAME=my-stream
      - KINESIS_PROVIDER=kinesalite
    ports:
      - "4567-4584:4567-4584"
      - "8055:8080"
    volumes:
      - './.localstack:/tmp/localstack'
      - '/var/run/docker.sock:/var/run/docker.sock'

Hope it helped

Upvotes: 6

Mark Davies
Mark Davies

Reputation: 1497

I think this may be the way you are formatting your docker compose YAML file, here is a possible fix for what you're trying to achieve here:

version: '3.1'
services:
  localstack:
    image: "localstack/localstack"
    container_name: localstack_demo
    environment:
      - SERVICES=kinesis,s3,sqs
      - DEBUG=1
      - DATA_DIR=/tmp/localstack
      - KINESIS_STREAM_SHARDS=1
      - KINESIS_ERROR_PROBABILITY=0.0
      - KINESIS_STREAM_NAME=my-stream
    ports:
      - "4567-4584:4567-4584"
      - "8055:8080"
    volumes:
      - './.localstack:/tmp/localstack'
      - '/var/run/docker.sock:/var/run/docker.sock'

Hope this helps

Upvotes: 6

Related Questions