Skip to main content

JBoss Fuse - Running Cron Jobs with Camel, uploading files using SFTP. Part one

In the old days when I was a developer in an finance company, we used to have hundreds of night time daily jobs, weekly and even yearly jobs. These job often required a lot of integration between systems, and can be very hard to manage, and what happen if the job requires intensive CPU usage and needed scale out?
With the Quartz2 component in camel, it provides a scheduled delivery of messages or executing business logics, even interact with other systems. Here I am going to show you show use the Quartz2 component that does load balance and upload a file to a FTP Server. 

If you ever played with Camel, you'll know there is a component call Timer, it  is used to generate message exchanges when a timer fires. So why use Quartz instead of timer? 
This is explained in Camel's document, 

Timer is a simple, non persistence timer using the JDK's in built timer mechanism.
Quartz uses the Quartz library which uses a database to store timer events and supports distributed timers and cron notation.

That means timer is more like a fire and forget mechanism, whereas Quartz provides a more complete schedule solution, it allows you configure clustering to do load balance and handle fail overs.

In this demo project, it tells you how setup a cron job using Quartz2 component with 2 node running that does the load balancing, and upload a file to a FTP Server through SFTP.  Also uses FTP component to poll from server and printout the content of the file. I will also add another timer component just to compare. :)


You can find all the camel component documentation here.

First create a Fuse Blueprint Project.
There are 2 dependencies you need to add to your pom.xml 

 
     org.apache.camel
     camel-quartz2
     2.12.0.redhat-610379
 
 
     org.apache.camel
     camel-ftp
     2.12.0.redhat-610379
   
 
  postgresql
  postgresql
  9.0-801.jdbc4
  

And now we are going to the core of what makes this demo interesting for you, setting up Quartz, but if you are familiar with the Quartz already, it would be exactly the same. There are 2 main kinds of Job store in Quartz, Ram and JDBC, Ram is more similar to Timer and cannot provide cluster config, therefore, we need to setup the Database Job Store, so all node have a centralized place to get job information. And also enable clustering by etting the "org.quartz.jobStore.isClustered" property to "true". I am using a PostgreSQL Database for this, but you can modify it to what ever database you want. 

Download the Quartz Jar and find the database setup script under docs/dbTables, find the sql file that you are using, I will use the tables_postgres.sql


create a new database in PostgreSQL and run the script.


In your project under src/main/resources/ create a folder org/quartz and file name "quartz.properties". 

# Main Quartz configuration
org.quartz.scheduler.skipUpdateCheck = true
org.quartz.scheduler.instanceName = DatabaseClusteredScheduler
org.quartz.scheduler.instanceId = AUTO
org.quartz.scheduler.jobFactory.class = org.quartz.simpl.SimpleJobFactory
org.quartz.jobStore.class = org.quartz.impl.jdbcjobstore.JobStoreTX
org.quartz.jobStore.driverDelegateClass = org.quartz.impl.jdbcjobstore.PostgreSQLDelegate
org.quartz.jobStore.dataSource = quartzDataSource
org.quartz.jobStore.tablePrefix = QRTZ_
org.quartz.jobStore.isClustered = true
org.quartz.threadPool.class = org.quartz.simpl.SimpleThreadPool
org.quartz.threadPool.threadCount = 10
org.quartz.jobStore.clusterCheckinInterval = 20000


# JobStore: JDBC jobStoreTX
org.quartz.dataSource.quartzDataSource.driver = org.postgresql.Driver
org.quartz.dataSource.quartzDataSource.URL = jdbc:postgresql://localhost:5432/quartz2
org.quartz.dataSource.quartzDataSource.user = quartz2
org.quartz.dataSource.quartzDataSource.password = quartz2123
org.quartz.dataSource.quartzDataSource.maxConnections = 10

Noted in the properties files, it uses isClustered = true, because we want to make sure our 2 nodes can correctly load balance. Same as in Quartz, there are CronTrigger and SimpleTrigger, since SimpleTrigger is similar with Timer, I will use CronTrigger. 

The URI format are as follows, 

quartz2://timerName?options
quartz2://groupName/timerName?options
quartz2://groupName/timerName?cron=expression
quartz2://timerName?cron=expression

The cron expression documentation can be found by here.

quartz2://myGroup/myTimer?cron=0+0/1+09-23+?+*+MON-SAT

my expression says,  Run the job in every minutes from Monday to Saturday 9am till 11pm. 
After all these are done,


        
        
            MyJob-${date:now:yyyyMMdd HH:mm:ss}.txt
        
        
            ${date:now:yyyyMMdd HH:mm:ss} Hello THis is the content of text 
        
        
    

You can also find the FTP component here, but the URI is very simple too, 

ftp://[username@]hostname[:port]/directoryname[?options]
sftp://[username@]hostname[:port]/directoryname[?options]
ftps://[username@]hostname[:port]/directoryname[?options]

In normal cases, fill in the informations you have. And the second route is very similar, it listens to FTP and print out the content. 

        
        

I've also added a timer route, that basically upload a file every 30 seconds. 



        
            MyTimer-${date:now:yyyyMMdd HH:mm:ss}.txt
        
        
            ${date:now:yyyyMMdd HH:mm:ss} Hello This is the content from Timer 
        
    

This video tells you how write a simple program,



I will deploy this project onto fabric in part 2. And show the load balancing of the cluster job.



Comments

Saiteja Parna said…
Very good!! Excellent....

Popular posts from this blog

Red Hat JBoss Fuse - Getting Started with Fuse Integration Service 2.0 Tech preview

I just realized that I did not do a getting started for Fuse Integration Service 2.0 Tech preview before I did the pipeline demo, thanks for those of you who reminded me! :)

To get started with FIS 2.0, for people who has just getting to know the technology, here is how I interpret it. Basically, it's divide into two aspect,

1. Integration development, FIS uses Apache Camel as the core technology that creates, orchestrate, compose microservices into a super lightweight thin integration layer, and become the API provider and service orchestrator through exposing RESTful or messaging service endpoints. And you can choose to either package and run it with Spring-Boot or Karaf.


2. Application Deployment and Management, FIS takes advantages of OpenShift platform, and allows you to separately deploy the micro-integration service among distributed environment, at the same time takes care of the failover, high availability, load balancing and service lookup problem for you.


So, now we know …

Red Hat JBoss Fuse/A-MQ - Fuse and A-MQ Version 6.3 GA is released!

Fuse and A-MQ 6.3 GA has just went out. Maybe, you would think this is just only a minor version release why should I care? Hold your thoughts on that! Because they have done a lot of improvements and also added many new features into this release.

Besides various bug fixes and making sure Fuse Fabric is much more stable. There are two major change in this version update:

New Tooling in JBoss Developer Studio (JBDS) 9.1 GA. Newer Apache Camel version – Camel v2.17. I was really impressed by the work put in to make developing Camel application much simpler. First is the installation of tooling itself. Now it has a all-in-one installer so you don't need to worry about which plugins you need to check. See the videos below to see the new "Getting Started" of Fuse 6.3.



And If you notice from the above video, the presentation of camel route in JBDS has also updated. It fixed some of the miss representation of logic and making it easier to read.

Old Camel Route
New Camel Route
On …

Fuse Integration Service - Setup JBDS and create first quickstart application

Before we go and start creating our first application, I want to show you how to setup your JBoss Developer Studio, create a small application from the quickstart example and then running it on Fuse Integration Service.

I am using JBoss Developer Studio version 9, you can find it here.
After download the

jboss-devstudio-9.0.0.GA-installer-eap.jar
double-click it, and start installing with default values.

After successful installation, we will need install the plugins for Fuse, on JBoss Central view, select software update, select enable early access.


And select JBoss Fuse Development for the plugin,


Click on install, and we are all set to go!

First thing first, we want to create a Fuse project to deploy on the base of Fuse Integration Service, which is OpenShift. If you have not installed it, please go back to my previous post for instructions. So on your JBDS, right click and start creating the project. Select new, maven project, if you have installed the plugin correctly, you should …