Skip to main content

The rise of Agile Integration, Integration is not DEAD nor LAME!

I wanted to blog about this for a very long time, but because of work and being too lazy to turn on my laptop on the weekends, now I finally have a couple of hours to sit down and start blogging.

"Integration is DEAD."
"No, no, let's talk about microservice!!"
"What is that again? Integration, you mean SOA? It's lame... ".

These makes the majority of the percentage when I talk about Integration. In fact, if you happen to be in any the software conference, you see the rooms full when it's about "container" and "microservice." I understand the nature of a developer, always seeking for the latest and greatest technology out there. BUT!! In my opinion, I don't think Integration is going AWAY if you containerize your application. Neither does it will DISAPPEAR in a microservice architecture. In fact, Integration just gets more complicated and tricky when you try to break an application into smaller pieces and have them running all over the place in the distributed environment!

It was hard to deal with when we have to orchestrate, choreograph and compose all the coarse-grained services together, and imaging letting tinier, smaller and more refined services talks to each other. Yeah, trying to get an oversized heavy-weight strong bull to co-ordinate and communicate is one thing. I don't think to have to conduct thousands of small cute duckling, (I know, they are now smaller to handle one by one, light and easy to replace) is going be any easier. :-)

Maybe the word Integration was abusively used by too many vendors (Us too ;-D ),  and it is too often associated with ESB, which IMHO should be one of the Integration strategies. ESB can be overly complicated with a wrapped around RPCs.   Integration is about connecting, composing between services, neglecting integration when implementing microservice architecture (MSA) is going to be disastrous. I am sure a lot of smart people out there designing the MSA has already taken this into account. But they probably did not realize that because Integration is already deeply ingrained into their minds, even in microservice we still want loose coupling. Again, in my opinion, the wall between application development and integration are getting thinner and thinner, by that I mean you do integration in microservice way and apply integration in your microservice. So, yes, integration needs to change, adapt and adjust accordingly but it never goes away:
  • Current deployment method
    • As now services are updated more frequent than ever before, we need to quickly respond to the change, in the integration layer. We can package it into the container and standardize the deployment method across the company. 
  • Speed and automated delivery
    • A quicker respond to developer, continuous integration and delivery, faster time to markets can be achieved by adopting the DevOps trend. Integration should also be part of this cycle. Not OUTSIDE! 
  • Flexible for changes
    • Making sure the message can be route to the correct service, allowing automatic discovery of services, with load balance capability that shares the load no matter it's synchronized or event base call. Providing and compose with flexible contracts like APIs between service can also minimize the impact on the changes. 
  • More robust towards failure
    • In the integration, it's all about the isolation of failure, and handles the errors, increase service resiliency. Also the ability for it to adopt version changes, and service lifecycle management is also one of the important factors in Integration.
Some concepts of doing integration in MSA comparing with monolithic application are poles apart. For example, continue from the bull and duckling theme. When you try to get a couple stubborn, massive bulls together and adjust position takes a lot of energy, and often you will need to be creative and smart to improve according to their position. Therefore the pipelines have to be smart and do many manipulations. Whereas the ducklings are easy to move around and cheap to replace. If one got sick, naughty or does not have the right color you need, simply toss it and get a newly hatched egg. (Alright, I don't mean any animal cruelty here, it's just a metaphor. ). The pipe that brings these microservice can be dumb or simple. Because each duckling is cheap to create. 

In my eye, microservice should never go freely create on wherever and whenever it should be at least logically bound. I call this application domain. By circling these microservice, we will be able to see a much clear picture of inbound microservice integration and outbound microservice integration. 

The inbound microservices integration will be a lot more flexible in protocols, data formats because of the possibility to connect to various datasources and SaaS application. Since they are bounded within the same application domain, it's likely the data models are shared between these microservices. They tend to be more coupled with each other (although service autonomy is where we are trying to get. They are probably handled by the same person.) There will be a couple of microservices responsible for composing some of the other microservices together too. The logic inside should be simple and straight forward. And each of the microservice should easily take care of the protocol and data extract itself. 

On the other hand, the outbound microservices integration should be strictly loosely coupled with inbound microservice. Preferably access commonly managed endpoints from each application domain, yes, I mean APIs. It is mostly device or client bound. As it often required to put together a more coarse-grain service like the old ESB days for the client application or from a process. 
Both inbound and outbound integration can be synchronous and asynchronous, all depends on the need of your system. So don't limit yourself to just one option. I strongly recommend the Enterprise Integration Pattern Book, as I said, the integration problem is still there. The patterns are extremely useful in this manner. 

Agile Integration is Red Hat's opinionated way of doing the integration. And most importantly it's backed up by Fuse (Apache Camel) and OpenShift. 

It talks about 3 main pillars, which resonate my integration believes in MSA world.
  • Distributed Integration 
    • Lightweight, support distributed deployment
    • Pattern Based Integration
    • Reusable Connectors
  • Containers
    • Cloud native solutions
    • Lean artifacts, individually deployable
    • Container based scaling and high availability
  • API
    • Well defined, re-usable, and well managed end-points 
    • Ecosystem leverage

So! I hope my murmur on this blog post will at least makes you think "Integration is NOT LAME". And start thinking about it when next time you create your next microservice.


Jian Feng said…
Great post!
They are very helpful thoughts.

Popular posts from this blog

Red Hat JBoss Fuse - Getting Started with Fuse Integration Service 2.0 Tech preview

I just realized that I did not do a getting started for Fuse Integration Service 2.0 Tech preview before I did the pipeline demo, thanks for those of you who reminded me! :)

To get started with FIS 2.0, for people who has just getting to know the technology, here is how I interpret it. Basically, it's divide into two aspect,

1. Integration development, FIS uses Apache Camel as the core technology that creates, orchestrate, compose microservices into a super lightweight thin integration layer, and become the API provider and service orchestrator through exposing RESTful or messaging service endpoints. And you can choose to either package and run it with Spring-Boot or Karaf.

2. Application Deployment and Management, FIS takes advantages of OpenShift platform, and allows you to separately deploy the micro-integration service among distributed environment, at the same time takes care of the failover, high availability, load balancing and service lookup problem for you.

So, now we know …

Red Hat JBoss Fuse/A-MQ - Fuse and A-MQ Version 6.3 GA is released!

Fuse and A-MQ 6.3 GA has just went out. Maybe, you would think this is just only a minor version release why should I care? Hold your thoughts on that! Because they have done a lot of improvements and also added many new features into this release.

Besides various bug fixes and making sure Fuse Fabric is much more stable. There are two major change in this version update:

New Tooling in JBoss Developer Studio (JBDS) 9.1 GA. Newer Apache Camel version – Camel v2.17. I was really impressed by the work put in to make developing Camel application much simpler. First is the installation of tooling itself. Now it has a all-in-one installer so you don't need to worry about which plugins you need to check. See the videos below to see the new "Getting Started" of Fuse 6.3.

And If you notice from the above video, the presentation of camel route in JBDS has also updated. It fixed some of the miss representation of logic and making it easier to read.

Old Camel Route
New Camel Route
On …

Fuse Integration Service - Setup JBDS and create first quickstart application

Before we go and start creating our first application, I want to show you how to setup your JBoss Developer Studio, create a small application from the quickstart example and then running it on Fuse Integration Service.

I am using JBoss Developer Studio version 9, you can find it here.
After download the

double-click it, and start installing with default values.

After successful installation, we will need install the plugins for Fuse, on JBoss Central view, select software update, select enable early access.

And select JBoss Fuse Development for the plugin,

Click on install, and we are all set to go!

First thing first, we want to create a Fuse project to deploy on the base of Fuse Integration Service, which is OpenShift. If you have not installed it, please go back to my previous post for instructions. So on your JBDS, right click and start creating the project. Select new, maven project, if you have installed the plugin correctly, you should …