Skip to main content

JBoss Fuse - Running Cron Jobs with Camel, uploading files using SFTP. Part one

In the old days when I was a developer in an finance company, we used to have hundreds of night time daily jobs, weekly and even yearly jobs. These job often required a lot of integration between systems, and can be very hard to manage, and what happen if the job requires intensive CPU usage and needed scale out?
With the Quartz2 component in camel, it provides a scheduled delivery of messages or executing business logics, even interact with other systems. Here I am going to show you show use the Quartz2 component that does load balance and upload a file to a FTP Server. 

If you ever played with Camel, you'll know there is a component call Timer, it  is used to generate message exchanges when a timer fires. So why use Quartz instead of timer? 
This is explained in Camel's document, 

Timer is a simple, non persistence timer using the JDK's in built timer mechanism.
Quartz uses the Quartz library which uses a database to store timer events and supports distributed timers and cron notation.

That means timer is more like a fire and forget mechanism, whereas Quartz provides a more complete schedule solution, it allows you configure clustering to do load balance and handle fail overs.

In this demo project, it tells you how setup a cron job using Quartz2 component with 2 node running that does the load balancing, and upload a file to a FTP Server through SFTP.  Also uses FTP component to poll from server and printout the content of the file. I will also add another timer component just to compare. :)


You can find all the camel component documentation here.

First create a Fuse Blueprint Project.
There are 2 dependencies you need to add to your pom.xml 

 
     org.apache.camel
     camel-quartz2
     2.12.0.redhat-610379
 
 
     org.apache.camel
     camel-ftp
     2.12.0.redhat-610379
   
 
  postgresql
  postgresql
  9.0-801.jdbc4
  

And now we are going to the core of what makes this demo interesting for you, setting up Quartz, but if you are familiar with the Quartz already, it would be exactly the same. There are 2 main kinds of Job store in Quartz, Ram and JDBC, Ram is more similar to Timer and cannot provide cluster config, therefore, we need to setup the Database Job Store, so all node have a centralized place to get job information. And also enable clustering by etting the "org.quartz.jobStore.isClustered" property to "true". I am using a PostgreSQL Database for this, but you can modify it to what ever database you want. 

Download the Quartz Jar and find the database setup script under docs/dbTables, find the sql file that you are using, I will use the tables_postgres.sql


create a new database in PostgreSQL and run the script.


In your project under src/main/resources/ create a folder org/quartz and file name "quartz.properties". 

# Main Quartz configuration
org.quartz.scheduler.skipUpdateCheck = true
org.quartz.scheduler.instanceName = DatabaseClusteredScheduler
org.quartz.scheduler.instanceId = AUTO
org.quartz.scheduler.jobFactory.class = org.quartz.simpl.SimpleJobFactory
org.quartz.jobStore.class = org.quartz.impl.jdbcjobstore.JobStoreTX
org.quartz.jobStore.driverDelegateClass = org.quartz.impl.jdbcjobstore.PostgreSQLDelegate
org.quartz.jobStore.dataSource = quartzDataSource
org.quartz.jobStore.tablePrefix = QRTZ_
org.quartz.jobStore.isClustered = true
org.quartz.threadPool.class = org.quartz.simpl.SimpleThreadPool
org.quartz.threadPool.threadCount = 10
org.quartz.jobStore.clusterCheckinInterval = 20000


# JobStore: JDBC jobStoreTX
org.quartz.dataSource.quartzDataSource.driver = org.postgresql.Driver
org.quartz.dataSource.quartzDataSource.URL = jdbc:postgresql://localhost:5432/quartz2
org.quartz.dataSource.quartzDataSource.user = quartz2
org.quartz.dataSource.quartzDataSource.password = quartz2123
org.quartz.dataSource.quartzDataSource.maxConnections = 10

Noted in the properties files, it uses isClustered = true, because we want to make sure our 2 nodes can correctly load balance. Same as in Quartz, there are CronTrigger and SimpleTrigger, since SimpleTrigger is similar with Timer, I will use CronTrigger. 

The URI format are as follows, 

quartz2://timerName?options
quartz2://groupName/timerName?options
quartz2://groupName/timerName?cron=expression
quartz2://timerName?cron=expression

The cron expression documentation can be found by here.

quartz2://myGroup/myTimer?cron=0+0/1+09-23+?+*+MON-SAT

my expression says,  Run the job in every minutes from Monday to Saturday 9am till 11pm. 
After all these are done,


        
        
            MyJob-${date:now:yyyyMMdd HH:mm:ss}.txt
        
        
            ${date:now:yyyyMMdd HH:mm:ss} Hello THis is the content of text 
        
        
    

You can also find the FTP component here, but the URI is very simple too, 

ftp://[username@]hostname[:port]/directoryname[?options]
sftp://[username@]hostname[:port]/directoryname[?options]
ftps://[username@]hostname[:port]/directoryname[?options]

In normal cases, fill in the informations you have. And the second route is very similar, it listens to FTP and print out the content. 

        
        

I've also added a timer route, that basically upload a file every 30 seconds. 



        
            MyTimer-${date:now:yyyyMMdd HH:mm:ss}.txt
        
        
            ${date:now:yyyyMMdd HH:mm:ss} Hello This is the content from Timer 
        
    

This video tells you how write a simple program,



I will deploy this project onto fabric in part 2. And show the load balancing of the cluster job.



Comments

Saiteja Parna said…
Very good!! Excellent....

Popular posts from this blog

Red Hat Fuse - Announcing Fuse 7 Tech preview 3 release.

Red Hat Fuse 7.0 technical preview three is out today! On the pathway to become one of the best cloud-native integration platform, Fuse gives developer freedom to choose how they want to develop the integration solution, where they want to deploy it and capabilities to address new integration personas that do not have development experience.
By supporting the three major runtime, developer is free to work on the runtime of their choice.By supporting standalone and cloud deployment, it simplifies the complexity to distinguish between these environments, allowing application to deploy freely among the environment of your choice. All levels of developers are welcome, you can either dive deep into creating customize complex integration logic, or using the new low code platform to quickly build a simple integration. In this Tech Preview release you get it all.
Fuse StandaloneSpring-boot for microserviceKaraf 4 for OSGi loverJBoss EAP for JavaEE developersFuse on OpenShiftPlugins for easy co…

JBoss EAP 6 - 效能調校 (一) DataSource 的 Connection Pool

效能沒有什麼Best Practice, 反正能調整的就那些。 通常,一個程式的效能大概有70-80% 都跟程式怎麼寫的其實比較有關係。

最近我最疼愛的小貓Puji 因為膀胱結石開刀的時候過世了,心情很差請原諒我的口氣沒有很好,也沒有心情寫部落格。

Puji R.I.P.

=======================正文=======================

這個題目很多人叫我寫,可是這題目好大,這分明就是整死我咩~
所以我會分幾段慢慢寫。

JBoss 的 SubsystemDatasource WebWeb Service EJB Hibernate JMSJCAJVM 調校OS (作業系統)

先來看一下 DataSource Subsystem, DataSource 的部分主要是針對Connection Pool 做調校。

通常,程式都會需要跟資料庫界接,電腦在本機,尤其是在記憶體的運算很快,但是一旦要外部的資源連接,就是會非常的耗資源。所以現在的應用程式伺服器都會有個Pool 放一些先連接好的 資料庫connection,當程式有需要的時候就可以馬上提供,而不用花那些多餘的資源去連接資料庫。

這就是為什麼要針對Connection Pool 去做調校。

以下會討論到的參數,都是跟效能比較有關係,Datasource 還有很多參數,像是檢核connection 是否正確的,我都不會提到。如果你追求的是非常快速的效能,那我建議你一個檢核都不要加。當然,這樣就會為伺服器上面執行的程式帶來風險。這就是你要在效能與正確,安全性上面的取捨了。 (套句我朋友說的話,不可能又要馬兒好,又要馬兒不吃草的..)

最重要的調校參數就是 Connection 的 Pool 數量。(也就是那個Pool 裡面要放幾條的connection.) 這個參數是每一個應用程式都不一樣的。

min-pool-size 

Connection Pool 最少會存留的connection 數量

max-pool-size 

Connection Pool 最多可以開啓的 connection 數量

prefill

事先將connection pool 裡面建立好min-pool-size 的connection.

我的建議是觀察一下平常程式要用到的量設定為 min-pool-size 。
加上…

Fuse - Contract First API Design with Apicurio and Fuse/Camel - Part One

This is part one of my two-article series that demonstrates the approach of implementing contract-first API design using Apicurioand Red Hat Fuse.

It covers how to create an OpenAPI standard document as the contract between API providers and consumers using Apicurio. It also shows how to quickly create mock tests using Red Hat Fuse.

There are two common approaches of creating these APIs.
Code FirstContract First Coming from a old time ESB developer, these are not new. We have been doing this forever. Before, it was the WSDL that define the contract of the service. we were doing a lot more code first, for me it's simply because it's much easier for me to write couple of Java classes and generate the WSDL for my consumer. 

It's often pretty straightforward if the consumer of your application has finalized how they want the service to be like. But you and I all know this is not often the case. So I had to go back to my code, and make the changes accordingly and pray I did not …