It's been a year and half since I blogged about "Agile Integration architecture" (Gosh, time just flies). With the "microservices" and "cloud-native" hype, I was especially curious on how all these new concept and technology affect us on how to architect the integration systems. If you ever pay close attention to all the latest and greatest news from the Kubernetes community, I am sure you will hear a lot about the new "Service Mesh". And rumor has it that this is how integration can/should be done in cloud native world, but, is that so? Anyone who has ever worked on an integration project would tell you, it's a LOT more COMPLEX and can get worst overtime. I did a talk with Christian Posta in Red Hat Tech Exchange coming from a more comprehensive view of how different Red Hat technologies are applied under different patterns when building integration solutions. In fact he also did a great blog about it.


Since then, another topics has been brought up numerous time, what about Serverless, how would it impact integration? Does it mean the death of services?  If we have all Microservices connected via Serviceless mechanism in service mesh, does that mean we go back to the old days of writing all the integration transformation logics in our application again? Obviously, this is complicated question to answer, and the new features keeps popping up, I am going to try to give my best to try and explain how I see everything fits into Agile Integration vision. Honestly, there is no exact right answer when it's about architecting system, it requires constant refactoring, my thought is to give a more general and flexible way of doing it that has less impact when change and being able to adapt change quickly. (That is what I call Agile Integration).

Service Mesh 

I remember last time, I started the agile integration reference architecture simply because the lack of organization of Microservices, and it still stand today. JUST because we have service mesh, doesn't mean it will magically solve the spaghetti connectivity if you are not being careful. To set the ground, what service mesh helps is to relief developer from dealing with repetitive, boring work in a distributed environment. Yes. Being cloud native with Microservices, means the system is more vulnerable to chain reaction disaster if not designed correctly, as you are working with small bots and gears in the system, that you never know how any small missed-handle can impact the entire system. Therefore we need to make each component more robust, failure proof and ideally damage recoverable. And Service Mesh helps you with getting the BASICs ready by adding a sidecar next to your running application. For every Microservices you create, no matter what they do, core business or content orchestration, they all need some kind of failure proof. Service Mesh gives you all that, from an application networking level. So that as a developer you will no longer have to wrap every single one of your Microservices with the circuit breaker, error retry and even handling version, deployment routing, and apply authorization,  those that has NOTHING to do with what the actually app is responsible for. These are often common rules that applies to the whole cluster. And service mesh is best for that, as these common policies are detached from actual application, can be managed centrally and apply individual on the sidecar so the microservice running your app is protected behind it.

Another great feature with service mesh, is to ability to trace and observe the incoming request for the entire system. This will certainly help developer/operation have more insight into the complex spider web-like connectivity.

Dealing with REAL Life Integration
But when it comes to actually application implementation, some folks thinks we can now rely on Service Mesh to configure and connect. MAYBE, if you are developing only a couple of small microservices, you can probably get away from that. BUT that is not what's happening in real life. For instance, formatting the data to the right granularity (Splitting/Aggregation), routing base on processing outcome of some content, a more complex orchestration of services call that requires precise rollback and business handling (Saga) , collection of events triggering, we can't possibly write all these into the Service Mesh config yml file (I am just being practical :p). So, I am sure now we have mutual agreement that integration logic still needs be written somewhere, DON'T tried to do it in your microservice with the business logic. That is why we need the conceptual layer in agile integration. They are the composite and core layer. Remember,  I mean CONCEPTUAL not physically on top of another, but separating the responsibility of the microservices so it's easier to locate, maintain and organize your applications. Taken from the lesson learnt from SOA. You will still need some kind of integration patterns to compose service that has the right granularity for the receiving end.

Monitoring and tracing is crucial in ANY integration system, and let's face it, there are NOT JUST http calls. Majority of times, they are events, and honestly an event driven reactive system is 100 times more flexible and modularize than sticking a bunch of API together. To be able to collect these data are needed beyond simple request tracing.


Serverless
I had to talk about this, the whole serverless development, deployment and runtime concept can take agile integration in another level not just conceptually but physically too. The idea of quickly spin up integration piece of logic, the ability to scale freely up and down. That is what I call true agility. I was thrilled to see Nicola demonstrate Camel-K on KNative.  Camel just opens up a wide range of connect possibility for the serverless call. To me the core spirits of server less is (For now)

• Being able to quickly produce/start application without complex configuration and heavy runtime. 
• Elastic resource allocation responding to loads. 

And what integration helps to bring to the table is the ability to allow the system to responds more events from boarder range of endpoints, for instance, IoT, SOAP and messaging and other protocol that are not simple http calls.


This is just a HIGH level overview of how I see Integration are shaped in the future of a serverless cloud native world. Of course I have not touch upon many topics like APIs, Event driven architecture, self service ability. They deserve separate blogs. Might do another later this week. Again, these are just my 2cents coming out from a more practical real life point of view.