CODEWITHSUNDEEP
airflowcamundaconductor
Jason Chen
Jason Chen

Reputation: 23

Microservices orchestration choices

I am exploring the possible solutions for orchestrating my flows across multiple services via some infrastructure. Searching shows me a few options such as Conductor, Camunda, Airflow etc.

I am wondering what would fit my use case better

  1. One of my service is in Java, the other is in Python
  2. I need to pass info to the Java service, then take the output and pass it to the Python service
  3. Final output is then published to another queue

It feels like Conductor is a good choice, but would love to hear your inputs!

Upvotes: 2

Views: 2794

Answers (3)

Pallavi Sengupta
Pallavi Sengupta

Reputation: 193

Introduction to Container Orchestration

The practice of automating the administration of container-based microservice applications across different clusters is known as container orchestration. Within corporations, this notion is gaining popularity. In addition, a variety of Container Orchestration technologies have become indispensable in the deployment of microservice-based applications.

Software development in the modern era is no longer monolithic. Instead, it generates component-based apps that run across many containers. These adaptable and scalable containers work together to accomplish a specified purpose or microservice. Depending on the complexity of the application and other requirements like load balancing, they may span many clusters.

Containers encapsulate application code as well as its dependencies. To function efficiently, they receive the resources they require from physical or virtual hosts. When complicated systems are built as containers, clustering them for deployment requires adequate management and priority.

How to Choose a Container Orchestration Tool?

We've looked at a number of Orchestration Tools that you may examine when selecting which is ideal for your business. To do so, make sure to understand your company's requirements and operations. Then you'll be able to more readily weigh the benefits and drawbacks of each option.

Kubernetes

Kubernetes has a lot of features and is ideally suited for container and cluster management at the corporate level. Kubernetes is managed by a number of platforms, including Google, AWS, Azure, Pivotal, and Docker. As the containerized workload grows, you have a lot of options.

The biggest disadvantage is that it does not work with Docker Swarm and Compose CLI manifests. It might also be difficult to understand and set up. Despite these flaws, it is one of the most used systems for cluster deployment and management.

Docker Swarm

For individuals who are already familiar with Docker Compose, Docker Swarm is a better option. It's easy to use and doesn't require any additional software. Unlike Kubernetes and Amazon ECS, however, Docker Swarm lacks sophisticated features such as built-in logging and monitoring. As a result, it is better suited to small-scale businesses that are just starting started with containers.

Amazon ECS

If you're already familiar with Amazon Web Services, Amazon ECS is a great way to install and configure clusters. It's a quick and easy method to get started, and it scales to match demand. It also connects with a number of other AWS services. It's also excellent for small teams with limited resources for container maintenance.

One of its disadvantages is that it is incompatible with nonstandard deployments. It also contains ECS-specific configuration files, which complicates debugging.

Upvotes: -3

Bas Harenslak
Bas Harenslak

Reputation: 3094

All listed tools might technically be able to execute your workflow (I have no experience working with Conductor & Camunda). A few characteristics on which a decision is usually made are:

  • open vs closed source
  • how do you define workflows? (e.g. Python code in Airflow. Others use e.g. JSON/XML/something custom)
  • does it come with a UI?
  • can it scale out in case my workloads start growing?
  • is it agnostic to any technology or limited to running certain technologies? (e.g. Oozie is built for scheduling jobs on Hadoop)
  • other requirements could be e.g. security, logging, monitoring, etc.

There are many orchestration-tool-comparisons on the internet, e.g. 1 or 2.

Upvotes: 0

rob2universe
rob2universe

Reputation: 7628

All options can fulfill the requirement stated. Think about further / future requirements. Is it only a data pipe? Is it about orchestrating a larger end-to end business process? Do you need support for long-running processes? Is end-to-end transparency in a graphical form a benefit? Is graphical process modelling in BPMN2 standard going to be a benefit? Are there going to be audit or reporting requirements? Or is it going to be a simple, isolated, technical solution?

This article gives a great overview of tools in the market and what their primary use cases are: https://blog.bernd-ruecker.com/understanding-the-process-automation-landscape-9406fe019d93

Upvotes: 1

Related Questions