CODEWITHSUNDEEP
spring-bootspring-cloudspring-cloud-stream
Keith Bennett
Keith Bennett

Reputation: 947

Spring Cloud Kinesis startup failing after latest Spring Cloud upgrade when running locally

Before deploying our Spring Cloud microservices (packaged as Docker images) to AWS, we test them locally against LocalStack. As such, we have beans defined like the following in our code base that are autoconfigured:

  @Bean("amazonKinesisAsyncMyBean")
  @Primary
  @Conditional(value = {LocalStackProfileCondition.class})
  public AmazonKinesisAsync localAmazonKinesisAsync(
      final LocalStackProfileProperties localStackProfileProperties,
      final LocalStackProperties localStackProperties,
      AWSCredentialsProvider awsCredentialsProvider) {
    System.setProperty(SDKGlobalConfiguration.AWS_CBOR_DISABLE_SYSTEM_PROPERTY, "true");
    System.setProperty(SDKGlobalConfiguration.DISABLE_CERT_CHECKING_SYSTEM_PROPERTY, "true");
    return AmazonKinesisAsyncClientBuilder.standard()
        .withEndpointConfiguration(
            new AwsClientBuilder.EndpointConfiguration(
                generateUriString(
                    localStackProperties.getProtocol().toLowerCase(),
                    localStackProperties.getHost(),
                    localStackProperties.getKinesis().getPort()),
                localStackProfileProperties.getClient().getRegion()))
        .withCredentials(awsCredentialsProvider)
        .build();
  }

With Spring Boot 2.3.5.RELEASE, Spring Cloud Hoxton.SR4, and Spring Cloud Stream Binder Kinesis 2.0.1.RELEASE, we are able to successfully test our Docker containers locally against LocalStack. However, after upgrading to Spring Cloud Hoxton.SR9 we are failing with the following stack trace:

Caused by: com.amazonaws.SdkClientException: Unable to execute HTTP request: Connect to squid.acme.com:8080 [squid.acme.com/x.x.x.x, squid.acme.com/x.x.x.x, squid.acme.com/x.x.x.x] failed: connect timed out
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1207)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1153)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530)
    at com.amazonaws.services.kinesis.AmazonKinesisClient.doInvoke(AmazonKinesisClient.java:2809)
    at com.amazonaws.services.kinesis.AmazonKinesisClient.invoke(AmazonKinesisClient.java:2776)
    at com.amazonaws.services.kinesis.AmazonKinesisClient.invoke(AmazonKinesisClient.java:2765)
    at com.amazonaws.services.kinesis.AmazonKinesisClient.executeDescribeStream(AmazonKinesisClient.java:875)
    at com.amazonaws.services.kinesis.AmazonKinesisClient.describeStream(AmazonKinesisClient.java:846)
    at org.springframework.cloud.stream.binder.kinesis.provisioning.KinesisStreamProvisioner.createOrUpdate(KinesisStreamProvisioner.java:135)
    at org.springframework.cloud.stream.binder.kinesis.provisioning.KinesisStreamProvisioner.provisionProducerDestination(KinesisStreamProvisioner.java:91)
    at org.springframework.cloud.stream.binder.kinesis.provisioning.KinesisStreamProvisioner.provisionProducerDestination(KinesisStreamProvisioner.java:57)
    at org.springframework.cloud.stream.binder.AbstractMessageChannelBinder.doBindProducer(AbstractMessageChannelBinder.java:223)
    ... 33 common frames omitted

Our local development environment is walled off from the internet, thus the connection timeout via the proxy. Regardless, the problem in the first place is that after upgrading Spring Cloud it appears the Kinesis binder is trying to connect to AWS to get the shard list. How can I ensure that these requests get routed to LocalStack and not AWS when I am running with the new configuration?

UPDATE 1:

After updating Spring Cloud Stream Kinesis Binder to 2.0.3.RELEASE, following is the stack trace I now receive:

Caused by: com.amazonaws.SdkClientException: Unable to execute HTTP request: Connect to squid.acme.com:8080 [squid.acme.com/x.x.x.x, squid.acme.com/x.x.x.x, squid.acme.com/x.x.x.x] failed: connect timed out
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1207)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1153)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530)
    at com.amazonaws.services.kinesis.AmazonKinesisClient.doInvoke(AmazonKinesisClient.java:2809)
    at com.amazonaws.services.kinesis.AmazonKinesisClient.invoke(AmazonKinesisClient.java:2776)
    at com.amazonaws.services.kinesis.AmazonKinesisClient.invoke(AmazonKinesisClient.java:2765)
    at com.amazonaws.services.kinesis.AmazonKinesisClient.executeListShards(AmazonKinesisClient.java:1557)
    at com.amazonaws.services.kinesis.AmazonKinesisClient.listShards(AmazonKinesisClient.java:1528)
    at org.springframework.cloud.stream.binder.kinesis.provisioning.KinesisStreamProvisioner.getShardList(KinesisStreamProvisioner.java:141)
    at org.springframework.cloud.stream.binder.kinesis.provisioning.KinesisStreamProvisioner.getShardList(KinesisStreamProvisioner.java:122)
    at org.springframework.cloud.stream.binder.kinesis.provisioning.KinesisStreamProvisioner.createOrUpdate(KinesisStreamProvisioner.java:167)
    at org.springframework.cloud.stream.binder.kinesis.provisioning.KinesisStreamProvisioner.provisionProducerDestination(KinesisStreamProvisioner.java:93)
    at org.springframework.cloud.stream.binder.kinesis.provisioning.KinesisStreamProvisioner.provisionProducerDestination(KinesisStreamProvisioner.java:59)
    at org.springframework.cloud.stream.binder.AbstractMessageChannelBinder.doBindProducer(AbstractMessageChannelBinder.java:223)
    ... 33 common frames omitted

UPDATE 2:

After setting org.springframework logging level to DEBUG, I was able to capture the following Conditions Evaluation Report:

============================
CONDITIONS EVALUATION REPORT
============================


Positive matches:
-----------------

   ContextCredentialsAutoConfiguration matched:
      - @ConditionalOnClass found required class 'com.amazonaws.auth.AWSCredentialsProvider' (OnClassCondition)

   KinesisBinderConfiguration matched:
      - @ConditionalOnMissingBean (types: org.springframework.cloud.stream.binder.Binder; SearchStrategy: all) did not find any beans (OnBeanCondition)

   KinesisBinderConfiguration#dynamoDBStreams matched:
      - @ConditionalOnMissingBean (types: com.amazonaws.services.dynamodbv2.AmazonDynamoDBStreams; SearchStrategy: all) did not find any beans (OnBeanCondition)

   KinesisBinderConfiguration.KinesisBinderHealthIndicatorConfiguration matched:
      - @ConditionalOnClass found required class 'org.springframework.boot.actuate.health.HealthIndicator' (OnClassCondition)
      - @ConditionalOnEnabledHealthIndicator management.health.defaults.enabled is considered true (OnEnabledHealthIndicatorCondition)

   KinesisBinderConfiguration.KinesisBinderHealthIndicatorConfiguration#kinesisBinderHealthIndicator matched:
      - @ConditionalOnMissingBean (names: kinesisBinderHealthIndicator; SearchStrategy: all) did not find any beans (OnBeanCondition)


Negative matches:
-----------------

   KinesisBinderConfiguration#amazonKinesis:
      Did not match:
         - @ConditionalOnMissingBean (types: com.amazonaws.services.kinesis.AmazonKinesisAsync; SearchStrategy: all) found beans of type 'com.amazonaws.services.kinesis.AmazonKinesisAsync' amazonKinesisAsyncMyBean (OnBeanCondition)

   KinesisBinderConfiguration#cloudWatch:
      Did not match:
         - @ConditionalOnMissingBean (types: com.amazonaws.services.cloudwatch.AmazonCloudWatchAsync; SearchStrategy: all) found beans of type 'com.amazonaws.services.cloudwatch.AmazonCloudWatchAsync' amazonCloudWatchAsync (OnBeanCondition)
      Matched:
         - @ConditionalOnProperty (spring.cloud.stream.kinesis.binder.kpl-kcl-enabled) matched (OnPropertyCondition)

   KinesisBinderConfiguration#dynamoDB:
      Did not match:
         - @ConditionalOnMissingBean (types: com.amazonaws.services.dynamodbv2.AmazonDynamoDBAsync; SearchStrategy: all) found beans of type 'com.amazonaws.services.dynamodbv2.AmazonDynamoDBAsync' amazonDynamoDBAsync (OnBeanCondition)

   KinesisBinderConfiguration#dynamoDBLockRegistry:
      Did not match:
         - @ConditionalOnProperty (spring.cloud.stream.kinesis.binder.kpl-kcl-enabled=false) found different value in property 'spring.cloud.stream.kinesis.binder.kpl-kcl-enabled' (OnPropertyCondition)

   KinesisBinderConfiguration#kinesisCheckpointStore:
      Did not match:
         - @ConditionalOnProperty (spring.cloud.stream.kinesis.binder.kpl-kcl-enabled=false) found different value in property 'spring.cloud.stream.kinesis.binder.kpl-kcl-enabled' (OnPropertyCondition)

   KinesisBinderConfiguration#kinesisProducerConfiguration:
      Did not match:
         - @ConditionalOnMissingBean (types: com.amazonaws.services.kinesis.producer.KinesisProducerConfiguration; SearchStrategy: all) found beans of type 'com.amazonaws.services.kinesis.producer.KinesisProducerConfiguration' kinesisProducerConfiguration (OnBeanCondition)
      Matched:
         - @ConditionalOnProperty (spring.cloud.stream.kinesis.binder.kpl-kcl-enabled) matched (OnPropertyCondition)


Exclusions:
-----------

    None


Unconditional classes:
----------------------

    None

Upvotes: 1

Views: 518

Answers (1)

Artem Bilan
Artem Bilan

Reputation: 121560

Let's look into that condition one more time:

 KinesisBinderConfiguration#amazonKinesis:
      Did not match:
         - @ConditionalOnMissingBean (types: com.amazonaws.services.kinesis.AmazonKinesisAsync; SearchStrategy: all) found beans of type 'com.amazonaws.services.kinesis.AmazonKinesisAsync' amazonKinesisAsync (OnBeanCondition)

Pay attention to the amazonKinesisAsync bean name. Doesn't look like it belongs to the mentioned localAmazonKinesisAsync bean in your configuration. Probably your conditions doesn't work somehow and another AmazonKinesisAsync bean wins which already points to the squid.acme.com:8080 endpoint...

Upvotes: 1

Related Questions