CODEWITHSUNDEEP
deploymentproduction-environmentrelease-managementab-testingcanary-deployment
Vikas Gupta
Vikas Gupta

Reputation: 10904

Canary vs. A/B release strategy

I was going through the different types of release strategy and was confused between the Canary and A/B strategy. Both of them seems to be similar.

Everywhere I read on Canary is "Allows to test deployment by releasing the new version to a small group of them." and on A/B is "An A/B testing strategy targets a specific group of customers."

Then where the differences lie between them and what are the use cases of both?

References: https://azure.microsoft.com/en-in/overview/kubernetes-deployment-strategy/

Upvotes: 25

Views: 16171

Answers (4)

Daniel D
Daniel D

Reputation: 11

If an app has a new feature that is to be released after internal testing, canary method decreases the risk of impact of unpredicted behaviour (hidden bug or misconfiguration). When the new feature is confirmed to work well, the feature may give different performance depending on parameters of that feature. To find out which set of parameters gives better performance we use A/B testing.

A typicall example in e-commerce is releasing new web page with new campaign of a product (canary deployment), then rearanging images, links or even modifying the contents of an advertisement on that web page to find out how viewers respond to the campaign (A/B testing).

Upvotes: 0

lhkirfuxbnmvklsn
lhkirfuxbnmvklsn

Reputation: 1

I believe already enough correct information has been provided in this thread. However, there is another simple differentiation WHEN you want to use either of them, so it may help you to differ between the two as their are indeed very similiar - but they serve different purposes:

  • Canary testing is concerned with production performance (version-oriented)

  • A/B testing is concerned about effectiveness of new features (effect-oriented)

Happy deploying!

Upvotes: 0

Srikanth
Srikanth

Reputation: 41

A/B Testing

In simple terms A/B testing is a way to compare two versions of something to determine which performs better .
In an A/B test, some percentage of your users automatically receives “version A” and other receives “version B.
It is a controlled experiment process. To run the experiment user groups are split into 2 groups. Group “A,” often called the “control group,” continues to receive the existing product version, while Group “B,” often called the “treatment group”, receives a different experience, based on a specific hypothesis about the metrics to be measured
At the end the results from 2 groups which is a variety of metrics is compared to determine which version performed better.

Canary Testing

Canary Testing is a way to reduce risk and validate new software by releasing software to a small percentage of users. With canary testing, you can deliver new features to certain groups of users at a time.
Since the new feature is only distributed to a small number of users, its impact is relatively small and changes can be reversed quickly should the new code prove to be buggy.
It is a technique to reduce the risk of introducing a new software version in production by slowly rolling out the change to a small subset of users before rolling it out to the entire infrastructure and making it available to everybody.
While canary releases are a good way to detect problems and regressions, A/B testing is a way to test a hypothesis using variant implementations.

Blue/Green Deployment

Blue-green deployment is a software deployment strategy which utilizes two production environments (a “blue environment” and a “green environment”) in order to make the software deployment process easier and safer.
The two production environments are kept as identical as possible, and when new code is deployed, it is pushed to the environment that is currently inactive. Once the new changes have been tested in production, a router can then switch to point to the environment where the new changes are live, making for a smooth cut-over.
One of the main benefits of blue-green deployments is disaster recovery. Because there are two identical environments for production, if new changes are rolled out to one (say the blue version) and any issues are discovered, a router can just switch back to the other environment (green version) which has the old version of the code with zero downtime.
Blue-green deployment can be used for canary testing by simply having the router direct a percentage of your traffic to new version of the code to see how it performs with live traffic, before rolling out the change to 100% of your users.

Upvotes: 4

suren
suren

Reputation: 8766

A/B test's purpose is usually to see users' response (In a way, how much they like it) to a new UI, feature, etc. But you know that the new version works. So, you actually send randomly both versions of the application to all of them. It can be 50-50, 80-20, 90-10, anything. Sometimes the functionality is not even relevant. You might want to see which version attracts more clients and stuff like that.

Canary is more focused on how well works the new feature. Or if it actually works. It usually will be 90-10, 80-20, A >> B. Never 50-50, because if it goes wrong, you don't want half of your users to have a bad experience. So you are not positive if the new version is going to work as expected.

The most important difference (and this is what almost no one talks about) is that a canary testing has session affinity. So it doesn't send both versions to all users, but randomly sends some users to the new version, and keeps them on the same version.

Upvotes: 50

Related Questions