Skip to main content

Reference architecture of Agile Integration

OK. I said it. Integration is still around, but in different form from my last blog post. So what does modern integration looks like? Looking at how agile scrum has taken over traditional waterfall development framework, by enable shorter deliver cycles, faster feedback, and having the flexibility to rapidly adapt changes. I believe it’s time for traditional integration to be agile again. By breaking up traditional ESB into distributed microservices.

A little recap of what should be in Agile integration:
  • Distributed Integration
    • Lightweight, support distributed deployment
    • Pattern Based Integration
    • Reusable Connectors
    • Microservices
  • Containers
    • Cloud native solutions
    • Lean artifacts, individually deployable
    • Container based scaling and high availability
  • API
    • Well defined, re-usable, and well managed end-points
    • Ecosystem leverage

Base on these three principle, I was asked to create a reference architecture for it. After giving some thoughts into it, drawing up numerous boxes and diagram, something clicked when I watched a 3 year old playing. And here, this is my answer…

Yup. That is it! Reference architecture of Agile Integration. Architecture is often needed during the design phase of a software project, it provides a structure and backbone for the software project. Once it’s done, it’s done, any architectural modification will be difficult and can bring disastrous result.  We all knew “one thing will never change in software development is Change”.  Maybe it’s because regulation requirement, market demands or simply learning more about the business domain. So I started to think, can we build an architecture that is flexible for change, and can be shaped into the needs of a project as needed. So base on that concept, this is my reference architecture for Agile Integration, the modern integration application development that enables flexibility from many angles.  

There are the major components in my reference architecture diagram for Agile Integration:
PaaS - Provides a foundation for fast pace software development by allowing developer to self-service, as well as enable automated provision.It will also boost operational efficiencies by making the entire environment DevOps ready.

Security/Identity management(IAM) -  Handles basic authentication and authorization for application interfaces and platforms.

Automation - Automates both the processes of building applications, and rolling deployments strategies for upgrades. Together with continuous integration tools,  it can achieve continuous delivery of the application softwares.
Logging and Tracing - What was suppose to be easy in monolithic world, has now become one of the challenge in distributed microservice architecture for integration. As it has now become harder to have a unified view of how things are doing. It is important to have a way to see logging as a whole. And has tracing capability for different activities.   

Containerized packages - Container technology allows us to build immutable portable lightweight application package, that is language agnostic. The configuration are independent from this builds allowing same package to deploy quickly in multiple environment.

Container management - In large scale system it is essential to have a convenient management platform to manage all the containers running the applications. Taking care of the monitoring, discovery, recovery and failover of the running containers.

Microservices - Large applications/services are broken down into easy maintainable pieces and can be independently develop and deploy in distributed environment. Each of them should be built with failure in mind. Microservices

Load Balance/Service discovery/Network management  - Microservice are built for flexibility, according to current system load, the load should be automatically balance to the each running instance. All services should be registered and can be found in the system without complex setup. Ideally a proxy endpoint for external users to hide away the deployment complexity.  

Layered architecture -  Logically organize microservice into a sets of layers, each has it’s own responsibility to avoid duplication of effort and make it easier to replace later on. Currently there are 4 layers
  • Gateway layer - provides simple gateway routing capability such as versioning, dealing with different platform of devices.
  • Composite layer - important middle tier that handles composition of multiple microservices. The do more complex routing from processing the content data itself and sometimes handle more complex data aggregation or normalization.  
  • Base layer - like the name which is most likely represents the basic components of the system. Handles data retrieval or business logic processing.
  • Anti-corruption layer - This layer handles interface to legacy application or anything that work against microservice quick and flexible principle. This layer is built in protection wall to your system, by doing the transforming job and translate between two very distinct implementation of the system.   

Application domains - This one is more like a pattern then architecture. But since we mention about layers in software I also wanted to touch a little bit on this. Because it’s all about reining the microservice into a more organized way. It’s a good practice to have each application domain defined (using same set of data model). And have other external integration microservice to wire between each domains. One other good thing about microservice is that you have the flexibility to move each small microservice to different domains as which it fits better with.

API management - Managing the APIs by reinforced access policy, collecting statistics around the usage of APIs. Also building a healthy eco-system among the developer and users of APIs. Further to allow direct revenue source from the APIs.

There are certainly some limitation in today’s technology to allow a complete pliable architectural form. But I believe this model can bring flexibility into the system without significant impact to the system when changes occur. I will explain more details in my upcoming posts.


Popular posts from this blog

Red Hat Fuse - Announcing Fuse 7 Tech preview 3 release.

Red Hat Fuse 7.0 technical preview three is out today! On the pathway to become one of the best cloud-native integration platform, Fuse gives developer freedom to choose how they want to develop the integration solution, where they want to deploy it and capabilities to address new integration personas that do not have development experience.
By supporting the three major runtime, developer is free to work on the runtime of their choice.By supporting standalone and cloud deployment, it simplifies the complexity to distinguish between these environments, allowing application to deploy freely among the environment of your choice. All levels of developers are welcome, you can either dive deep into creating customize complex integration logic, or using the new low code platform to quickly build a simple integration. In this Tech Preview release you get it all.
Fuse StandaloneSpring-boot for microserviceKaraf 4 for OSGi loverJBoss EAP for JavaEE developersFuse on OpenShiftPlugins for easy co…

JBoss EAP 6 - 效能調校 (一) DataSource 的 Connection Pool

效能沒有什麼Best Practice, 反正能調整的就那些。 通常,一個程式的效能大概有70-80% 都跟程式怎麼寫的其實比較有關係。

最近我最疼愛的小貓Puji 因為膀胱結石開刀的時候過世了,心情很差請原諒我的口氣沒有很好,也沒有心情寫部落格。

Puji R.I.P.



JBoss 的 SubsystemDatasource WebWeb Service EJB Hibernate JMSJCAJVM 調校OS (作業系統)

先來看一下 DataSource Subsystem, DataSource 的部分主要是針對Connection Pool 做調校。

通常,程式都會需要跟資料庫界接,電腦在本機,尤其是在記憶體的運算很快,但是一旦要外部的資源連接,就是會非常的耗資源。所以現在的應用程式伺服器都會有個Pool 放一些先連接好的 資料庫connection,當程式有需要的時候就可以馬上提供,而不用花那些多餘的資源去連接資料庫。

這就是為什麼要針對Connection Pool 去做調校。

以下會討論到的參數,都是跟效能比較有關係,Datasource 還有很多參數,像是檢核connection 是否正確的,我都不會提到。如果你追求的是非常快速的效能,那我建議你一個檢核都不要加。當然,這樣就會為伺服器上面執行的程式帶來風險。這就是你要在效能與正確,安全性上面的取捨了。 (套句我朋友說的話,不可能又要馬兒好,又要馬兒不吃草的..)

最重要的調校參數就是 Connection 的 Pool 數量。(也就是那個Pool 裡面要放幾條的connection.) 這個參數是每一個應用程式都不一樣的。


Connection Pool 最少會存留的connection 數量


Connection Pool 最多可以開啓的 connection 數量


事先將connection pool 裡面建立好min-pool-size 的connection.

我的建議是觀察一下平常程式要用到的量設定為 min-pool-size 。

JBoss Fuse - Fuse workshop 101 - Part One

On my way to Hong Kong for a day of workshop on JBoss Fuse, and as I go through my Slide deck, I cannot find any decent easy workshop for beginners. Therefore I decide make a workshop that is easy for Camel first timer to get their hands dirty.

The first of part of the workshop is an introduction to Camel, it first goes through what is exactly inside JBoss Fuse.

For part one of the workshop, it takes your through the very basic of Camel, one of the very important component inside JBoss Fuse.
Every Camel need to have a runtime container to run in, inside camel we call it a CAMEL CONTEXT.  Inside every Camel context, you can define lots of camel route and registry, don't worry about what those are, we will explain later.

So inside out blueprint xml, you will see a tag called camelContext.

Next up is camel route, they are a chain of command or process defined by you, as a developer.
Inside the camel route, there are consumer endpoints that listens to the incoming messages, producers …