Alohomora

WSO2 ESB - A Quick Glance at the Capabilities

It's a big but connected world with huge number of entities communicating by different languages and different protocols. In a service oriented architecture there is a set of such entities/components providing and consuming different services.
For these heterogeneous entities to communicate there need be a person in the middle who can speak with all of them regardless of the languages they speak and protocols they follow. Also, in order to deliver a useful service for a consumer there need to be a person to orchestrate the services provided from different entities. This person better be fast and able to handle concurrency pretty well.
Abstractly speaking Enterprise Service Bus(ESB) is that person and WSO2 ESB is the best opensource option out there if you need one!

Following are the main functionalities WSO2 ESB provides [2]

1. Service mediation

2. Message routing

3. Data transformation

4. Data transportation

5. Service hosting

Even though ESB could cover most of the integration use cases that you might need to implement, there are many extension points you could use in case you are unable to implement your use case with built-in capabilities.

You can download WSO2 ESB at [1] and play with it! [2] is a great article that you must read that would quickly walk you through what WSO2 ESB has to offer!

[1] http://wso2.com/products/enterprise-service-bus/
[2] http://wso2.com/library/articles/2017/07/what-is-wso2-esb/

SFTP protocol over VFS transport in WSO2 ESB 5.0.0

The Virtual File System (VFS) transport is used by WSO2 ESB to process files in the specified source directory. After processing the files, it moves them to a specified location or deletes them.

Let's look at a sample scenario how we can use this functionality of WSO2 ESB.

Let's say you need to periodically check a file system location on a given remote server and if a file is available you need to send an email attaching that file and move that file to some other file system location. This can be achieved as follows.

1. Let's first configure your remote server so that ESB could securely communicate with it over SFTP.
First create a public-private key pair with ssh.

run ssh-keygen command.

eg:
manurip@manurip-ThinkPad-T540p:~/Documents/Work/Learning/blogstuff$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/manurip/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/manurip/.ssh/id_rsa.
Your public key has been saved in /home/manurip/.ssh/id_rsa.pub.
The key fingerprint is:
c3:57:b2:82:ee:d3:b3:74:55:bf:9c:93:b7:7a:2e:df manurip@manurip-ThinkPad-T540p
The key's randomart image is:
+--[ RSA 2048]----+
| |
| |
| . . . |
| o + . . |
| . S o . .|
| . + . . +|
| ... . *.|
| ...o. . .=|
| ...o .*+E|
+-----------------+

Now open your .ssh folder which is located at /home/user in linux and open the file id_rsa.pub which contains the public key. Copy that and log in to your remote server and copy that and paste it in ~/.ssh/authorized_keys file.

2. Now, let's configure ESB.

First we need to enable VFS transport receiver so that we can monitor and receive the files from our remote server. To do that uncomment the following line in ESB-home/repository/conf/axis2/axis2.xml

Also, we need to be able to send a mail. For that, uncomment the following line as well from the same file. Also fill in the configuration. In case you will be using a gmail address to send mail, the configuration would be as following.

<transportSender name="mailto" class="org.apache.axis2.transport.mail.MailTransportSender">
<parameter name="mail.smtp.host">smtp.gmail.com</parameter>
<parameter name="mail.smtp.port">587</parameter>
<parameter name="mail.smtp.starttls.enable">true</parameter>
<parameter name="mail.smtp.auth">true</parameter>
<parameter name="mail.smtp.user">test@gmail.com</parameter>
<parameter name="mail.smtp.password">password</parameter>
<parameter name="mail.smtp.from">test@gmail.com</parameter>
</transportSender>

3. Now, create the following proxy service and sequence and save them in ESB-home/repository/dpeloyment/server/synapse-configs/default/proxy-services and ESB-home/repository/dpeloyment/server/synapse-configs/default/sequences respectively.

Here is the proxy service

Here, if your private key is in a different location(means its not at ~/.ssh/) or the name is different(i.e. name is not id_rsa) you will need to provide it as a parameter as follows.

<parameter name="transport.vfs.SFTPIdentities">/path/id_rsa_custom_name</parameter>

Here you can see that we have referred to sendMailSequence in our proxy service via the sequence mediator. The sendMailSequence will be as follows.

5. Now we are good to go! Go ahead and start WSO2 ESB. And log in to your remote server and create an xml file(say test.xml) in /home/user/test/source which the location we gave as the value for transport.vfs.FileURI property. Soon after doing that you will see that it gets moved to /home/user/dest which the location we gave as the value for transport.vfs.MoveAfterProcess property. Also an email with test.xml attached will be sent to the email address you specified in your sendMailSequence.xml.

Also if you added the log mediators I have put in the proxy service and sendMailSequence you should see the similar logs as follows in the wso2carbon.log.

[2016-12-13 22:04:28,510] INFO - LogMediator log = ====VFS Proxy====
[2016-12-13 22:04:28,511] INFO - LogMediator sequence = sendMailSequence

References
[1] https://docs.wso2.com/display/ESB500/VFS+Transport
[2] http://uthaiyashankar.blogspot.com/2013/07/wso2-esb-usecases-reading-file-from.html

Dynamically provisioning Jenkins slaves with Jenkins Docker plugin

In Jenkins we have the master slave architecture where we have configured one machine as master, and some other machines as slaves. We can have a preferred number of executors in each of these machines. Following illustrates that deployment architecture.

In this approach, the concurrent builds in a given Jenkins slave are not isolated. All the concurrent builds in a given slave would be running in the same environment. If we need several builds to be run inside the same slave those builds should need same environment and actions should have taken to avoid issues such as port conflicts. This prevents us from utilizing the resources in a given slave.

With Docker we can address the above problems which are caused by the inability to isolate the builds. Jenkins Docker plugin allows a docker host to dynamically provision a slave, run a single build, then tear-down that slave. Following illustrates the deployment architecture.

I'll list down the steps to follow to get this done.

First let's see what needs to be done in Jenkins master.
1. Install Jenkins in one node which would be the master node. To install Jenkins, you could either run Jenkins jar directly or deploy the jar in tomcat.

2. Install Jenkins Docker Plugin[1]

Now lets see how to configure nodes which you are using to put up slave containers in.

3. Install Docker engine in each of the nodes. Please note that due to a bug[2] in Docker plugin you need to use a docker version below 1.12. Note that I was using Docker plugin version 0.16.1.

eg:
echo deb [arch=amd64] https://apt.dockerproject.org/repo ubuntu-trusty main > /etc/apt/sources.list.d/docker.list

apt-get update

apt-get install docker-engine=1.11.0-0~trusty

4. Add the current user to the docker group - not a required step. If this is not done you will need to use root privileges(use sudo) to issue docker commands. And once step 3 is done anyone with the keys can give any instructions to Docker daemon. No need of sudo or being in docker group.

You can test if the installation is successful by running hello-world container
docker run hello-world

5. This is not a mandatory step but if you need to protect the docker daemon, by following [3] create a CA, server and client keys.

(Note that by default Docker runs via a non-networked Unix socket. It can also optionally communicate using an HTTP socket, and in order to do our job we need it to be able to communicate through an HTTP socket. And for Docker to be reachable via the network in a safe manner, you can enable TLS by specifying the tlsverify flag and pointing Docker’s tlscacert flag to a trusted CA certificate which is what are doing in this step)

6. configure /etc/default/docker as follows.
DOCKER_OPTS="--tlsverify --tlscacert=/path/to/ca.pem --tlscert=/path/to/server-cert.pem --tlskey=/path/to/server-key.pem -H tcp://0.0.0.0:2376"

Now let's see what are the configurations to be done in Jenkins master. We need jenkins master know about the nodes we previously configured to run slave containers in.

7. Go to https://yourdomain/jenkins/configure.

What Docker plugin does is adding Docker as a jenkins cloud provider. So each node we have will be a new “cloud”. Therefore for each node, throught “Add new cloud” section, add a clould of the type “Docker”. Then we need to fill configuration options as appropriate. Note that the Docker URL should be something like https://ip:2376 or https://thedomain:2376 where ip/thedomain are the ip or the domain of the node you are adding.

8. If you did follow step 3, in credentials section, we need to “Add” new credentials of the type “Docker certificates directory”. This directory should contain the server keys/CA/certs. Please note that you will need to have the ca,cert, client key names exactly as ca.pem, cert.pem and key.pem because I think those names are hardcoded in docker plugin source code therefore if custom names are put it won't work (I experienced it!)

9. You can press “Test Connection” button to test if the docker plugin could successfully communicate with our remote docker host. If it is successful, the docker version of the remote host should appear once the button is pressed. Note that if you have docker 1.12* installed, you will still see the the connection is successful but once you try building a job, you will get an exception since docker plugin has an issue with that version.

10. Under “Images” section, we need to add our docker image by “Add Docker template”. Note that you must have this image in your nodes you previously configured or need to have it in docker hub so that it can be pulled.

Here also there are some other configurations to be done. Under, “Launch method” choose, “Docker SSH Computer Launcher” and add the credentials of the docker container which is created by our docker image. Note that these are NOT the credentials for the node itself but the credentials of our dynamically provisioned docker jenkins slaves.

Here, you can add a label to your docker image. This is a normal jenkins label which can be used to bind jobs to a given label.

11. Ok now we are good to try running one of our jenkins build jobs in a Docker container! Bind the job you prefer to a docker image using the label you previously put and click "Build Now"!

You should see something similar to following. (Look at the bottom left corner)

Here we can see a new node named "docker-e86492df7c41" where "docker" is the name I put for the docker cloud I had created and "e86492df7c41" is the ID of the docker container which was dynamically spawned to build the project.

[1] https://issues.jenkins-ci.org/browse/JENKINS-36080
[2] https://wiki.jenkins-ci.org/display/JENKINS/Docker+Plugin
[3] https://docs.docker.com/engine/security/https/

WSO2 ESB Foreach mediator example

Say you have a payload. For example,

Here, you have an array with the root elemet "data". And you need to perform some processing for each of these objects in the array and send the resulting payload to an endpoint. I'm going to use foreach mediator to achieve this.
Lets say following is the resulting payload you want.

This payload contains, a root element "info" under which there is an array of objects having,
1. id element which is created using value of "name" element from original payload, converted to uppercase , value of "id" element from original payload, and the string "ID"
2. classid element which is the value of "id" element from the original payload
3. The "name" from original payload in uppercase.
4. university element which is hard-coded
5. department element which is also hard-coded

And you need to send this resulting payload to some endpoint in JSON format.

Inside foreach mediator I use script mediator to convert the name to uppercase. Then using payload factory mediator I create the payload. And outside the foreach mediator I use send mediator to send the resulting payload to the endpoint.

Here is the sequence.
Let's use the following API to call my-in-seq.
Add both above API and sequence to synapse config. We can send a curl request as follows and see how it works.

References
[1] https://docs.wso2.com/display/ESB490/ForEach+Mediator

Creating a Carbon Component using WSO2 Carbon Component Archetype

A Carbon Component is an OSGi bundle which has a dependency on Carbon Kernel.
Using WSO2 Carbon Component Archetype which has been published to maven central, you can create a simple Carbon Component with one command.

Eg:
mvn archetype:generate -DarchetypeGroupId=org.wso2.carbon -DarchetypeArtifactId=org.wso2.carbon.archetypes.component -DarchetypeVersion=5.0.0 -DgroupId=org.sample -DartifactId=org.sample.project -Dversion=1.0.0 -Dpackage=org.sample.project

Above command will create a Carbon Component with the following structure.

org.sample.project
├── pom.xml
└── src
└── main
└── java
└── org
└── sample
└── project
├── GreeterImpl.java
├── Greeter.java
└── internal
├── DataHolder.java
└── ServiceComponent.java

This Carbon Component consumes an OSGi service which is exposed from Carbon Core and also registers an OSGi service of its own.

If you do not pass the parameters for the project then the default values will be applied.

You can find the source for this archetype here.

Creating a generic OSGi bundle using WSO2 Carbon Bundle Archetype

You can create a generic OSGi bundle using one command with WSO2 Carbon Bundle Archetype.
Even though the name is WSO2 Carbon Bundle Archetype which has been published to maven central, this is a generic OSGi bundle.

Eg:
Execute the following command. The groupID of the archetype is org.wso2.carbon,
The artifactID of the archetype is org.wso2.carbon.archetypes.bundle and the version of the archetype is 5.0.0

mvn archetype:generate -DarchetypeGroupId=org.wso2.carbon -DarchetypeArtifactId=org.wso2.carbon.archetypes.bundle -DarchetypeVersion=5.0.0 -DgroupId=org.sample -DartifactId=org.sample.project -Dversion=1.0.0 -Dpackage=org.sample.project

This will create a project with the following structure.

org.sample.project
├── pom.xml
└── src
├── main
│ └── java
│ └── org
│ └── sample
│ └── project
│ ├── Greeter.java
│ └── internal
│ └── Activator.java
└── test
└── java
└── org
└── sample
└── project
└── GreeterTest.java

If you do not pass the parameters for the project then the default values will be applied.

The source for the archetype can be found here.

Unmarshalling an XML file with JAXB

This is a simple example for unmarshalling an XML file.

This is the XML file we are going to unmarshall
This is the schema of the above XML file
Now let's create Order.java
Here for each complex type I have created an inner class.
And for the element "item-name" which has the maxOccurs=unbounded, I have used a List.
Another thing is @XmlAccessorType(XmlAccessType.FIELD) is needed only if the annotated fields inside the class have public setters. If you don't have public setters for the annotated fields inside the class you don't need to annotate the class.

Now let's write the code for unmarshalling.

Google Summer of Code 2015

This year also I'm participating Google Summer of Code with Eclipse foundation.

My project is Implementing translation between Proj4 and Well-Known-Text in Geotrellis which is a high performance geoprocessing engine and programming toolkit.

Project description in Google Melange can be found here.

Writing a simple web service and a client with Axis2

1. The service

Here a simple POJO service will be created. For simplicity the service will have only two operations.

1. Save an order which consists of an index(String) and a price(Double)
The order will be saved to an xml file.

2. Retrieve the price given the index.

First create a Java Project.

Now inside package orderprocessing.data I create the Order POJO.

Inside the package orderprocessing.service I create OrderProcessingService Class.

In order to save orders to an XML file I create XMLWriterDOM.java.
To retrieve orders from the XML file I create XMLReader.java
Create an XML file orders.xml somewhere and change the "savePath", and "retrievePath" attributes inside the above two classes according to that.

Now create a folder META-INF at the src folder level and create services.xml inside META-INF.

Here, If you do not specify the message receiver, then the default message receiver which is RawXMLINOutMessageReceiver will perform the operation[1]

And since the service implementation class is in Java, all the public methods in that service will be exposed. No need to specifically include them in services.xml[1]

This is the folder structure of my service.

Now export the project as a jar. In eclipse this can be done by, right click on the project->export
Save it to somewhere and change it's extension to .aar (eg: if your jar is OrderProcessingService.jar rename it as OrderProcessingService.aar ) and save it to axis2-1.6.2/repository/services
(I am using axis2-1.6.2)

Start the axis2server. (In linux run axis2server.sh inside axis2-1.6.2/bin)

Now if you go to http://localhost:8080/axis2/services/ it should be shown as below.

And you can access the wsdl at, http://localhost:8080/axis2/services/OrderProcessingService?wsdl

2. The client

Create another Java project.

Now in order to use the service we need to create the stub.

Go to axis2-1.6.2/bin in the terminal and run,
Now create another package orderprocessing.client inside your client project and create OrderProcessingClient.java

This is my folder structure of the client

Run the client and see if it works.

References

[1] http://axis.apache.org/axis2/java/core/docs/axis2config.html#Service_Configuration

Making SBT(Simple Build Tool) use JDK 7 having both JDK 7 and 8 installed

I was following a scala tutorial and I was using SBT as the build tool. This SBT version required JDK 7 but I had environmental variables set for JDK8 even though I had both installed on my machine.

To make SBT use JDK7 without changing JAVA_HOME environmental variable, this is what I did.

First created a new environmental variable JAVA_HOME7(any name which is not already being used) with the path to JDK7 as the value.

Then replaced JAVA_HOME in the sbt.bat file(which is inside the sbt folder inside program files folder) with JAVA_HOME7.

MATLAB - Applying a function for each element in an array using arrayfun

I'm not a big matlab fan. For a project I needed to apply a function for each element in an array. For that arrayfun can be used.

Here is a simple example.

Say you have 3 arrays as follows.

A = [1,2,3,4]
B = [5,6,7,8]
C = [9,10,11,12]

And you need to calculate the value of a function myf, for A[i], B[i] and C[i] (i=1,2,3,4) which returns A[i] x B[i] + C[i]

You can do it as follows.

First define your function considering a single element.

function output=myf(a,b,c)
output=a*b + c;
end

Then,

A = [1,2,3,4]
B = [5,6,7,8]
C = [9,10,11,12]

D = arrayfun(@myf,A,B,C)

Remember to have "@" before myf. It denotes a handle in matlab. If you forget it, matlab will try to evaluate the function instead of considering its handle, which may result in "Not enough input arguments" error.

Because a bit of Art must be a part :) - No 2

The Cycle

He cried in a rhythm when he was born
It’s the joke among his peers
When he sings the wind slows down
Hares freeze and turn their ears

He is singing, just singing
He never knew the meaning of the song
He doesn't try, he never did
Bewitching the world, sings along

Charming young ladies,
Dancing like glimmering flames
Neither they ever knew
What’s the meaning? Is it deep or lame?

The lyricists, did they know?
Their only aim, do words rhyme?
Are they hollow, are they rich?
Who cares? It’s not a crime

Some wrote, he sang, all danced
The cycle went on and on…
No one asked why,
‘cause no danger was yet known…

Scroll to a given item in a dataview - Sencha Touch

In my Sencha Touch app I had a component dataview. I had included an index bar and I needed to scroll to the relevant items when I click on index bar. This is what I did.

Here recordIndex is the index of the component in dataview that we need to scroll to.

Adding a custom theme for the android preview screen in a Sencha Touch - Cordova app

In android on app launch, a preview screen is shown. This is before the splash screen. In a Sencha - Cordova app even if we have added the splash screen plugin, before the splash screen this preview window will appear. In my last post I have written how to change the theme of this preview screen in a pure Sencha Touch app. Here I'm writing how to add a custom theme for Sencha Touch+cordova app.

First create an xml file inside,

project-folder/cordova/platforms/android/res/values

The name of the XML file could be anything. Let's call it themes.xml. Then define your theme inside it.

Eg:
I'm going to override the NoTitleBar theme.. To do that this is what we should put inside themes.xml

Now, in the android manifest XML file which is inside,

project-folder/cordova/platforms/android

change the theme as follows.
android:theme="@style/MyTheme"

Black screen before splash screen in Sencha Touch android app?

I built a native Sencha Touch app using Sencha cmd.

I was having a problem. When I open the app before my splash screen loads a black screen was appearing. I found that the reason for this black screen is that on app launch, Android displays a simple preview window (based on your activity theme) as an immediate response to the user action. Then the preview window crossfades with the actual UI, once that has fully loaded [1].

I could not find a "nice" way to get rid of this black preview window. This is the only solution I found.

1. Go to AndroidManifest.xml inside the path_to_sencha_cmd/Sencha/Cmd/4.0.4.84/stbuild/st-res/android/
(I was using the version 4.0.4.84. The path depends on that.)

2. Inside application tag change the theme as android:theme="@android:style/Theme.Light"

If you do not want the title bar use this instead of the above one.
android:theme="@android:style/Theme.Light.NoTitleBar"

I don't think this is a good way to solve the problem. But I could not find a better way rather than changing Sencha cmd code.

References

[1] https://plus.google.com/+AndroidDevelopers/posts/VVpjo7KDx4H

Sencha native build - App doesn't go beyond app loading screen after installing in phone

I has developed a sencha touch application and I wanted to do a native build using Sencha cmd. I went through a really bad time trying to find out what exactly to do.

First, there is a difference in below two builds.

1. Sencha app build native

2. Sencha app package build

The first one creates the actual application using sencha native packager. The second creates the source to be put into a packager(eg: Cordova). Say you are going to build for android. Either way you will get an apk which can be installed in an android phone. But if you are planning to access native APIs then either you should do the native build or you should do the package build plus another wrapper.

I first packaged my app using "sencha app package build". But later I wanted to access native APIs(for example accessing the network status of the device).Therefore I ran sencha app build native command. I got a nice apk and I installed it in my phone.

Problems I faced and how I solved them

1. The app got stuck on app loading screen and none of the pictures I had on app loading screen were visible.

There is a file called app.json inside the application folder. You need to put paths to all the JS, CSS assests and resources inside this file.

2. Even after solving above problem app was still stuck at app loading screen.

I ran "Sencha app build production" command. This creates a minimized version of the app inside

./build/production folder. I pointed localhost to this. Then when app loads I could see the errors on console. The problem had been that I had included my "sencha-touch-all.js" file inside my index.html.(<script type="text/javascript" src="touch/sencha-touch-all.js"></script>)

Actually there is no need to put this inside index.html because we have already included this inside app.json as a JS assest. In the native build this file is not inside the app anymore but index file requires this because I have put it there. Once I removed this from index.html and did the native build again, my app didn't stuck at app loading screen anymore.

Because a bit of Art must be a part :) - No 1

Rage & Joy

I’m staring at it,
Yes, it’s me, the creator...
It’s my precious,
I’ll die again for it, and I am the keeper…

Long queues every day
To worship, thousands of prayers…
And modern artists (they say!)
Bewitched, wrote books of praises…

They explain the little scratch
Behind the ear, oh so clever!
“A design trick, not a failure”
I cry with rage, roll with laughter…

I half love and half hate,
It’s charming, peerless and all laud...
Resembles the heartless king,
I served, and I've made a god!

And the scratch silently sings,
How I carved with flesh, blood & bone…
My own innocent name,
On the death note…

~ Manuri Amaya Perera

PS: Side effects of some boring lectures!

response.write(something) doesn't work inside callback function?

I was learning NodeJS as I needed to implement a server for the GSoC project.

I faced a problem. response.write(something) was not working inside a callback function I had. I tried all sorts of things but couldn't figure out the problem. Finally I found the careless mistake I had done, that is I was having a response.end() which gets executed before my callback function. Once it was removed the code worked fine.

NodeJS's execution model has only a single process. Therefore NodeJs is using the concept of event driven asynchronous callbacks. So if you are having a slow operation like a database query even though NodeJS has a single process you will not have to wait until that operation finishes. The callback function can take care of what needs to happen once that slow operation is complete and the rest of the code will be executed without waiting for this slow operation.

In my code I was having such a slow operation and before this operation finishes the "response.end()" which I had mistakenly put, was being executed and because of that, "response.write()" inside the callback wasn't working.

Why would Sencha Touch build fail?

Today I was trying to build and package my app for GSoC but packaging was failing.

Path variables were properly set, every required software were installed and most importantly I built and packaged this same app inside the same directory using the same output folder few days back. I forgot that the only change I did was making the app folder a git repository. The output folder also was inside the repository.

When I changed the output folder in my config file to a folder outside the git repository building and packaging happened successfully.

Always use the process of elimination but before that make sure you have all the facts !

Some other things to check if build is failing

Have you installed Ruby and added it to the path variable?
Have you added the path to jdk as JAVA_HOME environment variable?

Google Summer of Code 2014

I got selected for Google Summer of Code 2014 this year to contribute to Raxa EMR and I'm thankful to Google and Raxa.

I think if you want to get selected for GSoC what matters is not the number of proposals you submit but the quality.

My project is to implement a mobile app that allows users to search for hospitals, doctors, pharmacies and view them on a Google map. And the user can get directions to the location and call if phone number is available. If the user is a registered user he can view his profile after logging in and he can make appointments with doctors. The user can mark his favorite locations. The app will support language translation. The technology used is Sencha Touch and the app will be able to run on both iOS and Android.

Project wiki can be found here.

Code repository can be found here.