Skip to main content

JBoss Fuse - Parsing CSV file playing with Websocket in Camel Part one

As I was jogging this morning, I went pass a vegetable warehouse, because it was really early in the morning, I saw the veg distributor were biding for tons and tons of vegetables, they were yelling and shouting. Love the atmosphere, and I saw a monitor that shows all kinds of information about the goods they delivered. So I begin to wonder, can I do this with Camel? 




OK, so here is the case, when ever a truck comes in with loads of goods, when the driver signs the goods over, a cvs file contains the product information will be feed into our system via wireless transmission. Because we need to work with these data, our first step is then to covert the individual items in csv file into Java Objects. Since there are multiple items in one csv file, so it's a perfect timing for us to use the splitter pattern in Camel. 


After we split each product, we can use the built-in Bindy Data format to convert the single line csv item into a Pojo. For Bindy, it is used for converting non-structured data to/from Java Bean. The sources of converting could be as follows, 

  • CSV records
  • Fixed-length records
  • FIX messages
  • or almost any other non-structured data


Before you start coding, make sure you have the right dependency by adding the camel-bindy dependency in. 

  org.apache.camel
  camel-bindy
  2.12.0.redhat-611412

First specify where you will collect the csv file, with a file endpoint, with our example, we listen from a csv file and delete the file after done

And then start splitting it using the pattern, uses new line as the separator,



And then convert it into Pojo by unmarshalling the csv file with bindy. As you can see we refer it to a Java Object named product. 

    
Inside the object,there are few things needs to be pre-configured like the annotation that defines binding mappings. First, need to set the separator between each column, and then set which position it is in the csv file. 
package org.blogdemo.websocketdemo;

import org.apache.camel.dataformat.bindy.annotation.CsvRecord;
import org.apache.camel.dataformat.bindy.annotation.DataField;

@CsvRecord(separator = "\\,")
public class Product {

	@DataField(pos = 1, required = true)
	String productId;
	
	@DataField(pos = 2, required = true)
	String productName;
	
	@DataField(pos = 3, required = true)
	String farmerName;
	
	@DataField(pos = 4, required = true)
	String quantity;
	
	@DataField(pos = 5, required = true)
	String harvestDate;
	
	//........


	@Override
	public String toString() {
		//......
	}
	
	
	
}


Here is the video taking you how to do it step by step. 
After we receive the data, we are going to display the information on the monitor through Websocket and also store the data somewhere in the system too in Part Two.


Comments

This comment has been removed by the author.
Hi,
I tried to use bindy to parse UTF-8 fixed length file I get errors. It seems that characters that need multiple bytes to encode confuse the parser/scanner.

How to deal with multiple bytes characters with bindy?

or we need to use something else?

Thanks in advance
Christina Lin said…
What do you mean by encoding multiple byte? Can you please given an example or tell me what you are trying to do?
Let's say that input file is fixed-lenght format with 600 characters. One field in this file contain Thai character that is 2 bytes / character.
Then total bytes of each line may exceed 600 bytes.
Indy check total bytes (not character) then it error.

I managed to workaround this by write my own Processor with java.

Thanks!
Keith Smith said…
Hi
Your sample is helpful. I am able to create a route that handle a fixed length file no problem. Now I am looking for a sample to read a fixed length with first row as header, then rest are detail rows. I have been trying to annotate "hasHeader" in the primary model and "isHeader" in the header model but no luck, would you kindly provide me some directions?

Thanks

Popular posts from this blog

Red Hat JBoss Fuse/A-MQ - Fuse and A-MQ Version 6.3 GA is released!

Fuse and A-MQ 6.3 GA has just went out. Maybe, you would think this is just only a minor version release why should I care? Hold your thoughts on that! Because they have done a lot of improvements and also added many new features into this release.

Besides various bug fixes and making sure Fuse Fabric is much more stable. There are two major change in this version update:

New Tooling in JBoss Developer Studio (JBDS) 9.1 GA. Newer Apache Camel version – Camel v2.17. I was really impressed by the work put in to make developing Camel application much simpler. First is the installation of tooling itself. Now it has a all-in-one installer so you don't need to worry about which plugins you need to check. See the videos below to see the new "Getting Started" of Fuse 6.3.



And If you notice from the above video, the presentation of camel route in JBDS has also updated. It fixed some of the miss representation of logic and making it easier to read.

Old Camel Route
New Camel Route
On …

Fuse Integration Service - Setup JBDS and create first quickstart application

Before we go and start creating our first application, I want to show you how to setup your JBoss Developer Studio, create a small application from the quickstart example and then running it on Fuse Integration Service.

I am using JBoss Developer Studio version 9, you can find it here.
After download the

jboss-devstudio-9.0.0.GA-installer-eap.jar
double-click it, and start installing with default values.

After successful installation, we will need install the plugins for Fuse, on JBoss Central view, select software update, select enable early access.


And select JBoss Fuse Development for the plugin,


Click on install, and we are all set to go!

First thing first, we want to create a Fuse project to deploy on the base of Fuse Integration Service, which is OpenShift. If you have not installed it, please go back to my previous post for instructions. So on your JBDS, right click and start creating the project. Select new, maven project, if you have installed the plugin correctly, you should …

Red Hat JBoss Fuse - Getting Started with Fuse Integration Service 2.0 Tech preview

I just realized that I did not do a getting started for Fuse Integration Service 2.0 Tech preview before I did the pipeline demo, thanks for those of you who reminded me! :)

To get started with FIS 2.0, for people who has just getting to know the technology, here is how I interpret it. Basically, it's divide into two aspect,

1. Integration development, FIS uses Apache Camel as the core technology that creates, orchestrate, compose microservices into a super lightweight thin integration layer, and become the API provider and service orchestrator through exposing RESTful or messaging service endpoints. And you can choose to either package and run it with Spring-Boot or Karaf.


2. Application Deployment and Management, FIS takes advantages of OpenShift platform, and allows you to separately deploy the micro-integration service among distributed environment, at the same time takes care of the failover, high availability, load balancing and service lookup problem for you.


So, now we know …