Blue Green deployment with multiple Micro Services with internal calls

Question

I have a 8 spring boot micro services which internally call each other. The calling dns's of other micro services, define in the application.properties file of each service.

Suppose, micro service A represent by A -> a.mydns.com and B-> b.mydns.com etc

So basically each micro service consist of a ELB and two HA Proxies (distribute in two zones) and 4 App servers (distribute in two zones).

Currently I am creating the new Green servers (app servers only) and switch the live traffic from HA Proxy level. In this case, while the new version of the micro services are testing, it expose to the live customers also.

Ideally, the approach should be, creating the entire server structure including ELB's and HA Proxies for each micro service right?

But then how come I face the challenge of testing it with a test dns. I can map the ELB to a test dns. But then how about the external micro service dns's which hard coded in side the application.properties file?

What would be the approach I should take in such scenario?

Answer 1

I personally would go down a simpler route using a test DNS entry for the green deployment which is then swapped out for the live DNS entry when you have fully verified your green deployment is good.

So what do I mean by this:

You state that your live deployments have the following DNS entries:

• a.mydns.com
• b.mydns.com

I would suggest that you create a pattern where each micro-service deployment also gets a test dns entry:

• test.a.mydns.com
• test.b.mydns.com

When deploying the "green" version of your micro-service, you deploy everything (including the ELB) and map the CNAME of the ELB to the test DNS entry in Route 53. This means you have the green version ready to go, but not being used by your live application. The green version has it's own DNS entry, so you can run your full test-suite against the test.a.mydns.com domain.

If (and only if) the test suite passes, you swap the CNAME entry for a.mydns.com to be the ELB that was created as part of your green deployment. This means that your existing micro-services simply start talking to your green deployment once DNS propagates. If there is an issue, simply reverse the DNS update to the old CNAME entry and you have fully rolled-back.

It requires a little bit of co-ordination here, but you should be able to automate the whole thing with something like Jenkins and the AWS CLI.

Answer 2

Typically for B/G testing you wouldn't use different dns for new functions, but define rules, such as every 100th user gets send to the new function or only ips from a certain region or office have access to the new functionality, etc.

Assuming you're using AWS, you should be able to create an ALB in front of the ELBs for context based routing in which you should be able define rules for your routing to either B or G. In this case you have to separate environments functioning independently (possibly using the same DB though).

For more complicated rules, you can use tools such as leanplum or omniture inside your spring boot application. With this approach you have one single environment hosting old and new functionality and later you'd remove the code that is outdated.

Answer 3

Ideally, the approach should be, creating the entire server structure including ELB's and HA Proxies for each micro service right?

This is not necessarily true. The deployment (blue green or canary, no matter what your deployment strategy is) should be transparent to it's consumers (in your case other 7 microservices). That means, your services DNS name (Or IP) to which other services interacts should stay the same. IMHO, in the event of a microservice deployment, you shouldnt have to think about other services in the ecosystem as long as you are keeping your part of the contract; after all that's the whole point of "micro"services. As other SOer pointed out, if you can't deploy your one microservice without making changes to other services, that is not a microservice, it's just a monolith talking over http.

I would suggest you to read this article https://www.thoughtworks.com/insights/blog/implementing-blue-green-deployments-aws

I am quoting relevant parts here

Multiple EC2 instances behind an ELB

If you are serving content through a load balancer, then the same technique would not work because you cannot associate Elastic IPs to ELBs. In this scenario, the current blue environment is a pool of EC2 instances and the load balancer will route requests to any healthy instance in the pool. To perform the blue-green switch behind the same load balancer you need to replace the entire pool with a new set of EC2 instances containing the new version of the software. There are two ways to do this -- automating a series of API calls or using AutoScaling groups.

There are other creatives ways like this too

DNS redirection using Route53

Instead of exposing Elastic IP addresses or long ELB hostnames to your users, you can have a domain name for all your public-facing URLs. Outside of AWS, you could perform the blue-green switch by changing CNAME records in DNS. In AWS, you can use Route53 to achieve the same result. With Route53, you create a hosted zone and define resource record sets to tell the Domain Name System how traffic is routed for that domain.

To answer other question.

But then how about the external micro service dns's which hard coded in side the application.properties file?

If you are doing this, I would suggest you to read about 12factor app; especially the config part. You should take a look at service discovery options too, if you haven't already done so.

I have a feeling that, what you have here is a spaghetti of not-so-micro-services. If it is a greenfield project and if your timeline-budget allows, I would suggest you to look in to containerizing your application along with it's infrastructure (a single word: Dockerizing) and use any container orchestration technology like kubernetes, Docker swarm or AWS ECS (easiest of all, provided you are already on AWS-land), I know this is out of scope of this question, just a suggestion.

Answer 4

I would suggest dockerizing your microservices (easy with spring-boot), and then using ECS (Elastic Container Service) and ELB (Elastic Load Balancer) with application loadbalancers. (can be internal, or internet faced).

ECS and ELB then utilizes your microservices /health endpoints when you deploy new versions.

Then you could implement a more sophisticated HealthIndicator in spring-boot, to determine whether or not the application is healthy (and therefor ready to recieve incomming requests). Only when the new application is healthy, is it put into service, and the old one(s) are put to sleep.

Then test all your business logic on a test environment, and because of Docker, you're running the exact same image on all environment, you shouldn't need to be running (any) tests when deploying to production. (Because it has already been tested, and if it boots up, you're good to go).