Skip to main content

JBoss Fuse - JBoss Fuse or ETL?

While I was hosting a workshop on JBoss Fuse, I got a question from on of the partners, why not use ETL?

"ETL (Extract, Transform and Load) is a process in data warehousing responsible for pulling data out of the source systems and placing it into a data warehouse."

To me it's rare to compare between Fuse to an ETL solution, because I think these two should work together to compliment each other. They are solving different problems when integrating. JBoss Fuse is a light weight more agile implementation of ESB. When trying to introduce SOA or even micro-service into your system, JBoss Fuse will help in many ways. I found this excellent decision tree diagram to decide when is best to use what.

The most obvious difference between an ESB and ETL if you need your real-time processing or if it is ok to have the data prepared couple of hours beforehand? To me ETL is more like an DATA tier integration solution, if you have 2 or more databases with massive amount of data that you want to join to produce some kind of report, then yes, ETL works best. So yes, if you ever hear things like, data, massive amount and report, then ETL is probably your best bet. 

On the other-hand, integrating in an enterprise is not just simply joining data. There are many departments, systems in an enterprise, each provide different services to end users, to provide suitable information, ETL can soon get very complicated, as different end users requires various kinds, granularity of data also their need of how recent the data needs to be, too. You will soon run into this very complex, tight coupling architecture. From developer's point of view, to get data they need, they will also need broad understanding of the other database architecture. And there are exceeding amount of service now move to cloud, to integrate these services in data level is going to be a challenge. That's why we need a layer that hide all the detail implementation for us to allow loose coupling between systems, much more agile and hides unnecessary informations. 

JBoss Fuse ESB can help you in this layer of services, developer no longer have to worried about what kind of protocols, these information are from, because of the prebuilt connectors in Camel. And the Enterprise Integration Pattern of Camel can help developer freely route the data between base services, or even use it to provide or combined services. Since data can come into may forms, Camel also has different built in mappers to transform between data types. And moving your services to the cloud. 

In real life there is no one size fit all solution, it is our job to find what solution fits best with our needs, and this is how I view ETL and ESB should work together. 

Here you can see there are a numbers of datasource, it could be file, database, or messages, when it comes to providing data for BI and reports, because it needs to process a large amount of data, so it's best to use ETL to generate in batch time. to hide the complexity, some service or functions requires less amount of data can be provide through JBoss Fuse ESB to ensure it expose the right amount of data and better with a restful endpoint call. The Data generated by ETL can also be extracted and provide services to others in the enterprise through JBoss ESB, again to make sure it's loose couple, we can expose them in the formate of CSV data file or XML file. Then when it comes to integrating services on the cloud or expose our service on cloud. With JBoss Fuse and achieve high availability and load balance in our service through use of Fabric Gateway. Also we can orchestrate these already built in service to provide more functionality to users.

In summary, JBoss Fuse and ETL solve different aspect when dealing with integration, one focus on Data tier integration, which is perfect for Big Data or reports, but when it comes to decoupling services and make then more agile, reuse of code, taking your enterprise to the cloud. JBoss Fuse is a much better solution.

Comments

Popular posts from this blog

Fuse Integration Service - Setup JBDS and create first quickstart application

Before we go and start creating our first application, I want to show you how to setup your JBoss Developer Studio, create a small application from the quickstart example and then running it on Fuse Integration Service.

I am using JBoss Developer Studio version 9, you can find it here.
After download the

jboss-devstudio-9.0.0.GA-installer-eap.jar
double-click it, and start installing with default values.

After successful installation, we will need install the plugins for Fuse, on JBoss Central view, select software update, select enable early access.


And select JBoss Fuse Development for the plugin,


Click on install, and we are all set to go!

First thing first, we want to create a Fuse project to deploy on the base of Fuse Integration Service, which is OpenShift. If you have not installed it, please go back to my previous post for instructions. So on your JBDS, right click and start creating the project. Select new, maven project, if you have installed the plugin correctly, you should …

Red Hat JBoss Fuse/A-MQ - Fuse and A-MQ Version 6.3 GA is released!

Fuse and A-MQ 6.3 GA has just went out. Maybe, you would think this is just only a minor version release why should I care? Hold your thoughts on that! Because they have done a lot of improvements and also added many new features into this release.

Besides various bug fixes and making sure Fuse Fabric is much more stable. There are two major change in this version update:

New Tooling in JBoss Developer Studio (JBDS) 9.1 GA. Newer Apache Camel version – Camel v2.17. I was really impressed by the work put in to make developing Camel application much simpler. First is the installation of tooling itself. Now it has a all-in-one installer so you don't need to worry about which plugins you need to check. See the videos below to see the new "Getting Started" of Fuse 6.3.



And If you notice from the above video, the presentation of camel route in JBDS has also updated. It fixed some of the miss representation of logic and making it easier to read.

Old Camel Route
New Camel Route
On …

RHTE - Supercharge your integration services

Red Hat Tech Exchange has taken place in Vietnam, Ho Chi Minh city two weeks ago, it is a great event held by Red Hat in Asia Pacific Region. It is open to all Red Hat partners who are interested in learning what Red Hat is doing recently, see what the trend of the open source world, basically it is a great event to share your knowledge and experience, to meet other enthusiastic people.

I am very fortunate to talk in this great event, to talk about the things I have been working on and even discuss it with many. Also got lots of great ideas too. So here are the slide.

My first talk was with Thomas Qvarnström about how to handle large size data in JBoss Fuse and how JBoss Data Grid can help in the situation.

Here is the agenda of the talk, we will be talk about this in the up coming webinar on 24th Sept.

Integration often involves storing, retrieving, and transforming data. Using a traditional database in your integration is likely to becomes a bottleneck that is expensive and hard to …