Skip to main content

JBoss Fuse - Parsing CSV file playing with Websocket in Camel Part one

As I was jogging this morning, I went pass a vegetable warehouse, because it was really early in the morning, I saw the veg distributor were biding for tons and tons of vegetables, they were yelling and shouting. Love the atmosphere, and I saw a monitor that shows all kinds of information about the goods they delivered. So I begin to wonder, can I do this with Camel? 




OK, so here is the case, when ever a truck comes in with loads of goods, when the driver signs the goods over, a cvs file contains the product information will be feed into our system via wireless transmission. Because we need to work with these data, our first step is then to covert the individual items in csv file into Java Objects. Since there are multiple items in one csv file, so it's a perfect timing for us to use the splitter pattern in Camel. 


After we split each product, we can use the built-in Bindy Data format to convert the single line csv item into a Pojo. For Bindy, it is used for converting non-structured data to/from Java Bean. The sources of converting could be as follows, 

  • CSV records
  • Fixed-length records
  • FIX messages
  • or almost any other non-structured data


Before you start coding, make sure you have the right dependency by adding the camel-bindy dependency in. 

  org.apache.camel
  camel-bindy
  2.12.0.redhat-611412

First specify where you will collect the csv file, with a file endpoint, with our example, we listen from a csv file and delete the file after done

And then start splitting it using the pattern, uses new line as the separator,



And then convert it into Pojo by unmarshalling the csv file with bindy. As you can see we refer it to a Java Object named product. 

    
Inside the object,there are few things needs to be pre-configured like the annotation that defines binding mappings. First, need to set the separator between each column, and then set which position it is in the csv file. 
package org.blogdemo.websocketdemo;

import org.apache.camel.dataformat.bindy.annotation.CsvRecord;
import org.apache.camel.dataformat.bindy.annotation.DataField;

@CsvRecord(separator = "\\,")
public class Product {

	@DataField(pos = 1, required = true)
	String productId;
	
	@DataField(pos = 2, required = true)
	String productName;
	
	@DataField(pos = 3, required = true)
	String farmerName;
	
	@DataField(pos = 4, required = true)
	String quantity;
	
	@DataField(pos = 5, required = true)
	String harvestDate;
	
	//........


	@Override
	public String toString() {
		//......
	}
	
	
	
}


Here is the video taking you how to do it step by step. 
After we receive the data, we are going to display the information on the monitor through Websocket and also store the data somewhere in the system too in Part Two.


Comments

This comment has been removed by the author.
Hi,
I tried to use bindy to parse UTF-8 fixed length file I get errors. It seems that characters that need multiple bytes to encode confuse the parser/scanner.

How to deal with multiple bytes characters with bindy?

or we need to use something else?

Thanks in advance
Christina Lin said…
What do you mean by encoding multiple byte? Can you please given an example or tell me what you are trying to do?
Let's say that input file is fixed-lenght format with 600 characters. One field in this file contain Thai character that is 2 bytes / character.
Then total bytes of each line may exceed 600 bytes.
Indy check total bytes (not character) then it error.

I managed to workaround this by write my own Processor with java.

Thanks!
Keith Smith said…
Hi
Your sample is helpful. I am able to create a route that handle a fixed length file no problem. Now I am looking for a sample to read a fixed length with first row as header, then rest are detail rows. I have been trying to annotate "hasHeader" in the primary model and "isHeader" in the header model but no luck, would you kindly provide me some directions?

Thanks

Popular posts from this blog

Red Hat Fuse - Announcing Fuse 7 Tech preview 3 release.

Red Hat Fuse 7.0 technical preview three is out today! On the pathway to become one of the best cloud-native integration platform, Fuse gives developer freedom to choose how they want to develop the integration solution, where they want to deploy it and capabilities to address new integration personas that do not have development experience.
By supporting the three major runtime, developer is free to work on the runtime of their choice.By supporting standalone and cloud deployment, it simplifies the complexity to distinguish between these environments, allowing application to deploy freely among the environment of your choice. All levels of developers are welcome, you can either dive deep into creating customize complex integration logic, or using the new low code platform to quickly build a simple integration. In this Tech Preview release you get it all.
Fuse StandaloneSpring-boot for microserviceKaraf 4 for OSGi loverJBoss EAP for JavaEE developersFuse on OpenShiftPlugins for easy co…

JBoss EAP 6 - 效能調校 (一) DataSource 的 Connection Pool

效能沒有什麼Best Practice, 反正能調整的就那些。 通常,一個程式的效能大概有70-80% 都跟程式怎麼寫的其實比較有關係。

最近我最疼愛的小貓Puji 因為膀胱結石開刀的時候過世了,心情很差請原諒我的口氣沒有很好,也沒有心情寫部落格。

Puji R.I.P.

=======================正文=======================

這個題目很多人叫我寫,可是這題目好大,這分明就是整死我咩~
所以我會分幾段慢慢寫。

JBoss 的 SubsystemDatasource WebWeb Service EJB Hibernate JMSJCAJVM 調校OS (作業系統)

先來看一下 DataSource Subsystem, DataSource 的部分主要是針對Connection Pool 做調校。

通常,程式都會需要跟資料庫界接,電腦在本機,尤其是在記憶體的運算很快,但是一旦要外部的資源連接,就是會非常的耗資源。所以現在的應用程式伺服器都會有個Pool 放一些先連接好的 資料庫connection,當程式有需要的時候就可以馬上提供,而不用花那些多餘的資源去連接資料庫。

這就是為什麼要針對Connection Pool 去做調校。

以下會討論到的參數,都是跟效能比較有關係,Datasource 還有很多參數,像是檢核connection 是否正確的,我都不會提到。如果你追求的是非常快速的效能,那我建議你一個檢核都不要加。當然,這樣就會為伺服器上面執行的程式帶來風險。這就是你要在效能與正確,安全性上面的取捨了。 (套句我朋友說的話,不可能又要馬兒好,又要馬兒不吃草的..)

最重要的調校參數就是 Connection 的 Pool 數量。(也就是那個Pool 裡面要放幾條的connection.) 這個參數是每一個應用程式都不一樣的。

min-pool-size 

Connection Pool 最少會存留的connection 數量

max-pool-size 

Connection Pool 最多可以開啓的 connection 數量

prefill

事先將connection pool 裡面建立好min-pool-size 的connection.

我的建議是觀察一下平常程式要用到的量設定為 min-pool-size 。
加上…

JBoss Fuse - Fuse workshop 101 - Part One

On my way to Hong Kong for a day of workshop on JBoss Fuse, and as I go through my Slide deck, I cannot find any decent easy workshop for beginners. Therefore I decide make a workshop that is easy for Camel first timer to get their hands dirty.

The first of part of the workshop is an introduction to Camel, it first goes through what is exactly inside JBoss Fuse.

For part one of the workshop, it takes your through the very basic of Camel, one of the very important component inside JBoss Fuse.
Every Camel need to have a runtime container to run in, inside camel we call it a CAMEL CONTEXT.  Inside every Camel context, you can define lots of camel route and registry, don't worry about what those are, we will explain later.


So inside out blueprint xml, you will see a tag called camelContext.



Next up is camel route, they are a chain of command or process defined by you, as a developer.
Inside the camel route, there are consumer endpoints that listens to the incoming messages, producers …