Enterprise Architecture & Integration, SOA, ESB, Web Services & Cloud Integration

Enterprise Architecture & Integration, SOA, ESB, Web Services & Cloud Integration

Friday, 27 December 2019

Accessing AWS S3 buckets from Apache Spark throwing Bad Request 400 error

There is no need for any introduction for Apache Spark (http://spark.apache.org) which is very popular for processing large data set in a very quick time. Spark can read data from various sources. S3 is one such popular data source for storing big data sets. So, you would end up very soon to read large data set from S3 and process using Spark for implementing useful business application use cases.

In my journey, I spent a lot of time in troubleshooting one issue "com.amazonaws.services.s3.model.AmazonS3Exception: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request;". I had to go through several blogs, official documentations (of aws, spark, hadoop), stack overflow questions/answers etc to finally make it working. So, I thought of writing a brief write up that can help some one.

Please follow these three steps:

1. You would need to populate the below key and value pairs
sc.hadoopConfiguration.set("fs.s3a.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem")
sc.hadoopConfiguration.set("fs.s3a.aws.credentials.provider", "org.apache.hadoop.fs.s3a.BasicAWSCredentialsProvider")
sc.hadoopConfiguration.set("fs.s3a.access.key", ACCESS_KEY)
sc.hadoopConfiguration.set("fs.s3a.secret.key", SECRET_KEY)
sc.hadoopConfiguration.set("fs.s3a.endpoint",
AWS_REGION)
sc.hadoopConfiguration.set("com.amazonaws.services.s3.enableV4", "true")

Please change AWS_REGION, ACCESS_KEY and SECRET_KEY as appropriate.

For official documentation, you can refer https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html

2. If you are still facing issue with V4 signature issue (https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-authenticating-requests.html), ideally passing "com.amazonaws.services.s3.enableV4" should work. If it doesn't, you can follow another option which did the magic for me FINALLY.

import com.amazonaws.SDKGlobalConfiguration
System.setProperty(SDKGlobalConfiguration.ENABLE_S3_SIGV4_SYSTEM_PROPERTY, "true")

For official documentation, you can refer https://docs.aws.amazon.com/sdk-for-java/


3. Version issue. There are many versions, dependencies between spark, hadoop & aws sdk, I have used the below versions in my "build.sbt":
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "2.4.4" % "provided",
"org.apache.hadoop" % "hadoop-aws" % "2.7.4" % "provided",
"com.amazonaws" % "aws-java-sdk" % "1.7.4" % "provided",
)
Here is the complete program for your reference:

import org.apache.spark.SparkConf
import org.apache.spark.SparkContext
import org.apache.log4j.Logger
import org.apache.log4j.Level
import com.amazonaws.SDKGlobalConfiguration

object S3BigDataAnalysis {
  def main(args: Array[String]) {
    Logger.getLogger("org").setLevel(Level.ERROR);
   
    val conf = new SparkConf()
    conf.setAppName("S3BigDataAnalysis")
    conf.setMaster("local")

    val sc = new SparkContext(conf)
    sc.hadoopConfiguration.set("fs.s3a.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem")
    sc.hadoopConfiguration.set("fs.s3a.aws.credentials.provider", "org.apache.hadoop.fs.s3a.BasicAWSCredentialsProvider")
    sc.hadoopConfiguration.set("fs.s3a.access.key", ACCESS_KEY)
    sc.hadoopConfiguration.set("fs.s3a.secret.key", SECRET_KEY)
    sc.hadoopConfiguration.set("fs.s3a.endpoint", AWS_REGION)
    //sc.hadoopConfiguration.set("com.amazonaws.services.s3.enableV4", "true")
        System.setProperty(SDKGlobalConfiguration.ENABLE_S3_SIGV4_SYSTEM_PROPERTY, "true")


    val iot_devices = sc.textFile("s3a://iot-devices/2019/*.json")
   
iot_devices.foreach(println)
  }
}


For understanding the research done by several people, you can refer below links:

https://stackoverflow.com/questions/34209196/amazon-s3a-returns-400-bad-request-with-spark

https://stackoverflow.com/questions/57477385/read-files-from-s3-bucket-to-spark-dataframe-using-scala-in-datastax-spark-submi

https://stackoverflow.com/questions/30385981/how-to-access-s3a-files-from-apache-spark?noredirect=1&lq=1

https://stackoverflow.com/questions/55119337/read-files-from-s3-pyspark


Hope this tip will be very useful, save several hours of yours. Please leave your comment here if you have any query. I will try to answer at the earliest opportunity!

🙏

Thursday, 28 November 2019

Installing Java 11 in Amazon Linux

Many of the old Java applications have got stuck with JDK 1.8 itself due to several reasons. When possible, you should be migrating to JDK13 or at least JDK11.

Below steps will help you to install JDK 11 and check the java version also.

Step 1: 
Ensure you have internet connectivity from your Amazon EC2 instance. And, then type the below command

sudo amazon-linux-extras install java-openjdk11

The above command will install JDK11.

Step 2:
Type the below command to check the version

java -version

The result will be somthing like this

openjdk version "11.0.5" 2019-10-15 LTS
OpenJDK Runtime Environment 18.9 (build 11.0.5+10-LTS)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.5+10-LTS, mixed mode, sharing)

Please note that from Java 9 onwards, the version numbering system has changed. For JDK1.8, the version number will be something like "1.8.0_192". For latest version, you wont see 1.x.y_zzz. For example, JD11, it will be like this "11.x.y"

Have fun!


Tuesday, 4 July 2017

SSL 3.0 / TLS 1.0 vulnerability issue and solution

Since TLS v1 has vulnerability issues, you are strongly advised to start using TLSv1.1 or TLSv1.2 to secure your corporate applications.

In order to force the application server or standalone application to use TLS v1.2 for example, you can please pass the following JVM argument

-Dhttps.protocols=TLSv1.2

Saturday, 1 July 2017

Maven Dependency Management - Reduce the war file size

A complex web application project might be using a large number of third party libraries in addition to your own application libraries. Together all, these jar files would be increasing the war file size which will be creating issues while transferring and deploying the files in the UAT and production servers.

This can be sorted out by following two steps
1. In your Maven pom.xml, you need to change the scope of the artefact to "provided" instead of "compile".

<dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <scope>provided</scope>
        </dependency>

what Maven does is - it will use the libraries for compiling the source code but will not bundle the dependent libraries along with war file. Now, look at the size of the war file - it would be few KBs, not MBs.

2. Run the following command "mvn dependency:copy-dependencies" in your project home where you have pom.xml. The maven will analyse the pom.xml and copy all required dependant libraries under target/dependency folder. You can copy these dependent jar files under designated server lib of your favorite  application serer, and the job is done!

Thursday, 29 June 2017

Maven Dependency Management - Listing Dependency Jar files

Here is a small tip if you've started using Maven very recently.

How would you know the list of dependencies used in your Maven project?

D:\work\messaging>mvn dependency:list
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building Messaging 0.0.1-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-dependency-plugin:2.8:list (default-cli) @ messaging ---
[INFO]
[INFO] The following files have been resolved:
[INFO]    com.rabbitmq:amqp-client:jar:4.1.1:compile
[INFO]    org.slf4j:slf4j-api:jar:1.7.21:compile

[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 3.265 s
[INFO] Finished at: 2017-06-29T10:48:58+05:30
[INFO] Final Memory: 12M/114M
[INFO] ------------------------------------------------------------------------

Wednesday, 27 April 2016

Using SHA-256 for secure passwords, data integrity protection and digital signatures

Even more seasoned developers mix Encryption with Digest and do not know the difference between them. Both are different and having different purposes. Yes, Digest is not Encryption. Digest is a One-way method and Encryption is two-way method - which essentially means that you can not reconstruct the original message from the message digest whereas you will be able to decrypt the encrypted message to reconstruct the original message.

Digest is used for hashing, checksum, data integrity, digital signature, password etc

Digest is otherwise known as hash, message digest also.

There are many digest algorithms available. Let us see here about SHA-256. Wikipedia says "SHA-2 (Secure Hash Algorithm 2) is a set of cryptographic hash functions designed by the National Security Agency (NSA). SHA stands for Secure Hash Algorithm". You can read more about it here https://en.wikipedia.org/wiki/SHA-2

When you create a digest for a given message using SHA256, it will generate a 32 byte string which is usually represented as 64 digit hexadecimal number. It is not encryption algorithm, so you can not decrypt the digest to reveal the original message.

Here is a Java program to generate digest:

import java.security.MessageDigest;

public class HashingUtil {


public String hash(String message) throws Exception{
System.out.format("Original message is %s\n", message);
MessageDigest messageDigest = MessageDigest.getInstance("SHA-256");
byte[] hash = messageDigest.digest(message.getBytes("UTF-8"));

StringBuffer digest = new StringBuffer(64);
for (int i = 0; i < hash.length; i++) {
String hex = Integer.toHexString(0xff & hash[i]);
if (hex.length() == 1)
digest.append('0');
digest.append(hex);
}
System.out.format("Digest is %s\n", digest.toString());
return digest.toString();
}

public static void main(String[] args) throws Exception{
String hash = new HashingUtil().hash("iKnowWhatYouDidLastSummer");
System.out.println(hash);
}
}

For Java doc, you can refer here https://docs.oracle.com/javase/7/docs/api/java/security/MessageDigest.html 

Hop you like this post. Let me know your comments if any.


Tuesday, 24 November 2015

Read HTTP Header and Pass as CORS headers in WSO2 ESB

Are you facing CORS headers issue? If you want to understand it better, it has been defined well in https://developer.mozilla.org/en-US/docs/Web/HTTP/Access_control_CORS and https://en.wikipedia.org/wiki/Cross-origin_resource_sharing. You can also follow http://enable-cors.org/.

As you read, you would understand that you may have to pass the following response headers
Access-Control-Allow-Origin
Access-Control-Allow-Methods
Access-Control-Allow-Headers


It is straightforward to add the first two headers in the response path (i.e., Out Sequence). For third header i.e., Access-Control-Allow-Headers, you may have to use the value that has come in the request header "Access-Control-Request-Headers". If so, you can achieve this in two steps:

Step 1:
Get the value for header "Access-Control-Request-Headers" and store the value in synapse config scope. It means that the value will be available both in and out sequence.

Snippet that need to be added in "In Sequence":
<property name="Access-Control-Request-Headers" expression="get-property('transport', 'Access-Control-Request-Headers')"/>

Step 2:
Get the value from synapse config, and set it as response header

Snippet that need to be added in "Out Sequence":
<property name="Access-Control-Allow-Headers"
     expression="$ctx:Access-Control-Request-Headers"
     scope="transport"
     type="STRING"/>


Sunday, 15 November 2015

Expose existing SOAP Web Services as REST API using WSO2 ESB

Problem statement

Your enterprise might have invested a lot in exposing software assets as SOAP Web Services to integrate with internal and external applications. If a new "modern" consumer who can (or wants to) consume only REST API, would you rewrite the entire thing to support REST clients?

WSO2 ESB (for example, version 4.8.1 which I have used) supports RESTful integration using API. We can leverage this feature to expose SOAP web services through REST API without changes to SOAP services. So, same time you would be able to support existing SOAP clients and new REST clients.

There are many blogs / web sites available on detailed explanation on REST style. You can follow this https://www.ics.uci.edu/~fielding/pubs/dissertation/top.htm from Roy Fielding. Another blog on REST can be found here in http://rest.elkstein.org/2008/02/what-is-rest.html

Steps involved

The following steps are required to achieve what we said.
  1. Define REST API
  2. Transform request REST data into SOAP data
  3. Call the back end application / URL
  4. Transform response SOAP data into REST data (for example JSON)
  5. Respond with JSON response

Sample use case - Student Exam Result Service


Usecase overview

We will implement a simple use case - student exam results application. The organisation has the SOAP service already running in the premise that will take student id as input parameter and responds with marks for four subjects.

Sample request SOAP message:
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:stud="http://www.example.org/student/">
   <soapenv:Header/>
   <soapenv:Body>
      <stud:getMarks>
         <student>John Smith</student>
      </stud:getMarks>
   </soapenv:Body>
</soapenv:Envelope>

Sample response message:
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:stud="http://www.example.org/student/">
   <soapenv:Header/>
   <soapenv:Body>
      <stud:getMarksResponse>
         <marks>
            <Mathematics>95</Mathematics>
            <Science>97</Science>
            <Lang-I>92</Lang-I>
            <Lang-II>94</Lang-II>
         </marks>
      </stud:getMarksResponse>
   </soapenv:Body>
</soapenv:Envelope>

WSDL:
<wsdl:definitions name="Student" targetNamespace="http://www.example.org/student/" xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/" xmlns:tns="http://www.example.org/student/" xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
   <wsdl:types>
      <xsd:schema targetNamespace="http://www.example.org/student/">
         <xsd:element name="getMarks">
            <xsd:complexType>
               <xsd:sequence>
                  <xsd:element name="student" type="xsd:string"/>
               </xsd:sequence>
            </xsd:complexType>
         </xsd:element>
         <xsd:element name="getMarksResponse">
            <xsd:complexType>
               <xsd:sequence>
                  <xsd:element name="marks" type="tns:Marks"/>
               </xsd:sequence>
            </xsd:complexType>
         </xsd:element>
         <xsd:complexType name="Marks">
            <xsd:sequence>
               <xsd:element name="Mathematics" type="xsd:int"/>
               <xsd:element name="Science" type="xsd:int"/>
               <xsd:element name="Lang-I" type="xsd:int"/>
               <xsd:element name="Lang-II" type="xsd:int"/>
            </xsd:sequence>
         </xsd:complexType>
      </xsd:schema>
   </wsdl:types>
   <wsdl:message name="getMarksRequest">
      <wsdl:part element="tns:getMarks" name="parameters"/>
   </wsdl:message>
   <wsdl:message name="getMarksResponse">
      <wsdl:part element="tns:getMarksResponse" name="parameters"/>
   </wsdl:message>
   <wsdl:portType name="Student">
      <wsdl:operation name="getMarks">
         <wsdl:input message="tns:getMarksRequest"/>
         <wsdl:output message="tns:getMarksResponse"/>
      </wsdl:operation>
   </wsdl:portType>
   <wsdl:binding name="StudentSOAP" type="tns:Student">
      <soap:binding style="document" transport="http://schemas.xmlsoap.org/soap/http"/>
      <wsdl:operation name="getMarks">
         <soap:operation soapAction="http://www.example.org/student/getMarks"/>
         <wsdl:input>
            <soap:body use="literal"/>
         </wsdl:input>
         <wsdl:output>
            <soap:body use="literal"/>
         </wsdl:output>
      </wsdl:operation>
   </wsdl:binding>
   <wsdl:service name="Student">
      <wsdl:port binding="tns:StudentSOAP" name="StudentSOAP">
         <soap:address location="http://www.example.org/"/>
      </wsdl:port>
   </wsdl:service>
</wsdl:definitions>

Now, lets do some hands on in WSO2 ESB.

1. Define REST API

In WSO2 ESB Management Console, goto Main --> Service Bus --> API and create a new API. The steps are very clearly mentioned in the WSO2 documentation site https://docs.wso2.com/display/ESB480/Getting+Started+with+REST+APIs how to create API. 


The API definition will look like this.

<api xmlns="http://ws.apache.org/ns/synapse"
     name="StudentAPI"
     context="/exam-results">
   <resource methods="GET" uri-template="/student/{student}" >

<inSequence> </inSequence>
<outSequence> </outSequence>
</resources>
</api>

This will provide a REST GET url like this http://localhost:8280/exam-results/student/1002 where 1002 is the student id for which we need the marks scored in four subjects. The URL may directly be accessed from a browser for instance.

2. Transform request REST data into SOAP data

Use Payload Mediator to transform REST data into SOAP data

         <payloadFactory media-type="xml">
            <format>
               <stud:getMarks xmlns:stud="http://www.example.org/student/">
                  <student>$1</student>
               </stud:getMarks>
            </format>
            <args>
               <arg evaluator="xml" expression="get-property('uri.var.student')"/>
            </args>
         </payloadFactory>

3. Call the back end application / URL

Set Action property and configure the SOAP end point URL

         <header name="Action" value="http://www.example.org/student/getMarks"/>
     <send>
            <endpoint>
<address uri="http://localhost:8088/services/Student" format="soap11"/>
            </endpoint>
         </send>

4. Transform response SOAP data into REST data (for example JSON)

Now, you will get the response in SOAP format. Use Payload Mediator to transform into REST data i.e., JSON

         <payloadFactory media-type="json">
            <format>{"Language_1": "$1", "Language_2": "$2", "Maths": "$3", "Science": "$4"}</format>
            <args>
               <arg xmlns:stud="http://www.example.org/student/"
                    evaluator="xml"
                    expression="$body/stud:getMarksResponse/marks/Lang-I"/>
               <arg xmlns:stud="http://www.example.org/student/"
                    evaluator="xml"
                    expression="$body/stud:getMarksResponse/marks/Lang-II"/>
               <arg xmlns:stud="http://www.example.org/student/"
                    evaluator="xml"
                    expression="$body/stud:getMarksResponse/marks/Mathematics"/>
               <arg xmlns:stud="http://www.example.org/student/"
                    evaluator="xml"
                    expression="$body/stud:getMarksResponse/marks/Science"/>
            </args>
         </payloadFactory>

5. Respond with JSON response


Change the response type to JSON and send the response to REST client

         <property name="messageType" value="application/json" scope="axis2"/>
         <send/>









When you execute, the output will look like
{
  "Language_1": "92",
  "Language_2": "94",
  "Maths": "95",
  "Science": "97"
}

See below the complete API definition

<?xml version="1.0" encoding="UTF-8"?>
<api xmlns="http://ws.apache.org/ns/synapse"
     name="StudentAPI"
     context="/exam-results">
   <resource methods="GET" uri-template="/student/{student}">
      <inSequence>
         <log level="custom">
            <property name="student_name_##########"
                      expression="get-property('uri.var.student')"/>
         </log>
         <payloadFactory media-type="xml">
            <format>
               <stud:getMarks xmlns:stud="http://www.example.org/student/">
                  <student>$1</student>
               </stud:getMarks>
            </format>
            <args>
               <arg evaluator="xml" expression="get-property('uri.var.student')"/>
            </args>
         </payloadFactory>
         <header name="Action" value="http://www.example.org/student/getMarks"/>
         <property name="Accept-Encoding" scope="transport" action="remove"/>
         <log level="custom">
            <property name="#body" expression="$body/*"/>
         </log>
         <send>
            <endpoint>
               <address uri="http://localhost:8088/services/Student" format="soap11"/>
            </endpoint>
         </send>
      </inSequence>
      <outSequence>
         <log level="custom">
            <property name="#response-body-from-Backend" expression="$body/*"/>
         </log>
         <payloadFactory media-type="json">
            <format>{"Language_1": "$1", "Language_2": "$2", "Maths": "$3", "Science": "$4"}</format>
            <args>
               <arg xmlns:stud="http://www.example.org/student/"
                    evaluator="xml"
                    expression="$body/stud:getMarksResponse/marks/Lang-I"/>
               <arg xmlns:stud="http://www.example.org/student/"
                    evaluator="xml"
                    expression="$body/stud:getMarksResponse/marks/Lang-II"/>
               <arg xmlns:stud="http://www.example.org/student/"
                    evaluator="xml"
                    expression="$body/stud:getMarksResponse/marks/Mathematics"/>
               <arg xmlns:stud="http://www.example.org/student/"
                    evaluator="xml"
                    expression="$body/stud:getMarksResponse/marks/Science"/>
            </args>
         </payloadFactory>
         <property name="messageType" value="application/json" scope="axis2"/>
         <log level="custom">
            <property name="#body" expression="$body/*"/>
         </log>
         <send/>
      </outSequence>
   </resource>
</api>

Hope this is useful to you. Please let me know if any further help.

Wednesday, 14 October 2015

Enable TLS in Standalone Java to connect to WSO2 IS

Some days ago, I wrote a blog on how to disable SSL and enable TLS in WebLogic application server. The link is here if you are interested to read further http://ayyappan-gandhirajan.blogspot.in/2015/09/enable-tls-security-in-weblogic-n-WSO2-IS.html

My colleague has approached me today to know how to do the same thing in a Core Java environment - I mean he is running a standalone program which uses HttpURLConnection to connect to HTTPS URL (which is hosted in WSO2 Identity Server, available at http://wso2.com/products/identity-server/)

After spending some time, I found a way to do this which has been described below:

1. Add JVM argument -Dhttps.protocols=TLSv1 on the client side

2. Or add this line into your the jave program - java.lang.System.setProperty("https.protocols", "TLSv1");

He just added JVM argument, and now he is able to connect the HTTPS URL without any issue.

Hope you like it.

Wednesday, 30 September 2015

Enable TLS security in Weblogic Application server to avoid "Read channel closed" error

Some host servers have been configured "not" to use SSL v1, v2 and v3 protocols for security reasons. Instead, they have been configured to use TLS protocol to ensure more secure HTTPS traffic.

Recently, I faced an issue with using SSL. My WebLogic application server had to connect to WSO2 Identity Server (http://wso2.com/products/identity-server/) for getting access token (https://docs.wso2.com/display/IS500/OpenID+Connect+with+the+WSO2+Identity+Server+and+WSO2+OAuth2+Playground) using an HTTPS URL. However, I was initially getting error "Read channel closed" on the WebLogic side. There was no other useful information. My other colleague, who takes care of WSO2 IS, troubleshooted and found that SSL has been disabled on the WSO2 IS server. This gave me a clue and then finally found the following option to make WebLogic to use TLS rather than SSL for initiating HTTPS traffic.

Pass this JVM argument -Dweblogic.security.SSL.protocolVersion=TLS1 into your WebLogic application server start up script. Restart the server and it is DONE.

With the above, my WebLogic server is now able to connect to WSO2 IS using HTTPS protocol.

Hope this helps.


Wednesday, 15 April 2015

How to find out character set in Oracle database

There is a simple SQL Query which can be used to find out the character set used in Oracle database

select * from nls_database_parameters where parameter='NLS_CHARACTERSET';

When I executed this, I got this below.

NLS_CHARACTERSET    AR8MSWIN1256

Wednesday, 31 December 2014

Configuring proper roles to access ActiveVOS console using standard alone server

ActiveVOS serve that is embedded with designer does not have any login screen and by default you will be able to access the console that shows up dashboards, monitor, catalog and other features. But when you install ActiveVOS server as stand alone, you need to configure the server with
proper user and roles. Otherwise, you would not be able to access your http://localhost:8080/activevos page.

If you are familiar with Tomcat, you can add "admin" as user and role. Now you would be able to login as admin user. But you can't still see any contents as you do not have necessary privilege. You need to add these roles again your "admin" user to see proper contents of ActiveVOS console page.

For your reference, I am pasting contents from ActiveVOS site.

1) tomcat\conf\server.xml (file-based configuration)
<Realm className= "org.apache.catalina.realm.UserDatabaseRealm" resourceName="UserDatabase"/>


2) tomcat\conf\tomcat-users.xml file:

<role rolename="abTaskClient"/>
<role rolename="abServiceConsumer"/>
<role rolename="abAdmin"/>
<user username="admin" password="admin" roles="abAdmin, abTaskClient, abServiceConsumer"/>


 For further read, please follow this http://infocenter.activevos.com/infocenter/ActiveVOS/v92/index.jsp?topic=/doc.server_userguide/html/SvrUG3-4.html


ActiveVOS Error deploying BPR: No catalog mapping found for resource

I created a business process (BPEL) in ActiveVOS Designer and deployed in embedded server sucessfully. When I exported the process and deployed in standalone ActiveVOS server, the deployment failed. The ActiveVOS server spit the following error:

[INFO][CommonErrorHandlerProcess.bpr] Starting BPR archive deployment.
ERROR: [CommonErrorHandlerProcess.bpr] [CommonErrorHandlerProcess.pdd] Error deploying BPR: No catalog mapping found for resource "project:/org.activebpel.rt.email.services/wsdl/email.wsdl".
[INFO][CommonErrorHandlerProcess.bpr] Finishing BPR archive deployment.


After googling several times, I found that it was due to license issue. I have uploaded the license and the issue is resolved.

Before confirming it is license issue, you may have to follow the below steps:
1. Check if the server started successfully
2. Logon to ActiveVOS server console (http://localhost:6060/activevos)
3. Check the server status (Home --> Server Status)
4. Check the contribution (Catalog --> Contributions)
5. Check the resources after unchecking "Hide System" (Catalog --> Resources --> All)

Happy Orchestrating!

Thursday, 6 November 2014

Tomcat authentication using Remote Address Filter / Remote Address Valve



Remote Address Filer or Remote Address Valve lets you to check the remote machine IP address and decide whether to allow or deny access. This is really useful when you want to enforce system to system authentication. Filter is nothing but an interceptor which will be used by Tomcat server to check if remote server can access the application. For more information, you can check the original documentation at http://tomcat.apache.org/tomcat-7.0-doc/config/valve.html#Remote_Address_Filter. In this post, I am trying to explain the power of regular expressions in configuring IP addresses in allow or deny attribute.

 1. A sample valve configuration that allows access only to localhost is:
<Valve className="org.apache.catalina.valves.RemoteAddrValve" allow="127.0.0.1"/>

2. The "allow" attribute can take comma separated values to support configuring more than one remote IP address. This is useful when you have a few IP addresses. If you need to configure a big list of IP addresses, this is going to be tough for you. In this case, you can configure the filter with wild card character to allow (or deny) multiple IP addresses. Sample is as below:

<Valve className="org.apache.catalina.valves.RemoteAddrValve" allow="10.110.156.*"/>
The above will allow from 10.110.156.0 to 10.110.156.255.

3. Alternatively, Tomcat server allows you to use regular expression to have fine control on the way IP addresses are being configured. Look at the below examples:

<Valve className="org.apache.catalina.valves.RemoteAddrValve" allow="10\.110\.156\.\d{1,3}"/>
The above will allow IP addresses from 10.110.156.0 to 10.110.156.999. This is almost similar to output of wildcard example shown above.

4. You may want to still fine tune the values.
<Valve className="org.apache.catalina.valves.RemoteAddrValve" allow="10\.110\.156\.[1-2][0-9]"/>
The above will allow IP addresses from 10.110.156.10 to 10.110.156.29 only.

So it is really up to how you write regular expression to achieve proper filtering of IP addresses. This link http://docs.oracle.com/javase/7/docs/api/java/util/regex/Pattern.html can provide more detailed information on regular expressions.

Happy securing tomcat server!


Friday, 19 September 2014

Apache Camel - Failed to resolve endpoint: smtps://smtp.gmail.com:465

Problem:

While routing incoming JMS message to send mail using "smtps", camel is spitting this error:

Caused by: org.apache.camel.ResolveEndpointFailedException: Failed to resolve endpoint: smtps://smtp.gmail.com:465?debugMode=true&password=bbb&username=aaa%40gmail.com due to: No component found with scheme: smtps
    at org.apache.camel.impl.DefaultCamelContext.getEndpoint(DefaultCamelContext.java:534)
    at org.apache.camel.util.CamelContextHelper.getMandatoryEndpoint(CamelContextHelper.java:63)
    at org.apache.camel.model.RouteDefinition.resolveEndpoint(RouteDefinition.java:192)
    at org.apache.camel.impl.DefaultRouteContext.resolveEndpoint(DefaultRouteContext.java:106)
    at org.apache.camel.impl.DefaultRouteContext.resolveEndpoint(DefaultRouteContext.java:112)
    at org.apache.camel.model.SendDefinition.resolveEndpoint(SendDefinition.java:61)
    at org.apache.camel.model.SendDefinition.createProcessor(SendDefinition.java:55)
    at org.apache.camel.model.ProcessorDefinition.makeProcessor(ProcessorDefinition.java:500)
    at org.apache.camel.model.ProcessorDefinition.addRoutes(ProcessorDefinition.java:213)
    at org.apache.camel.model.RouteDefinition.addRoutes(RouteDefinition.java:909)
    ... 18 more

Camel configuration:
 
<bean id="myNotificationListener" class="MyNotificationListener"/>
  
<camel:camelContext xmlns="http://camel.apache.org/schema/spring">
   <route id="sendmailnotification">
      <from uri="activemq:queue:AuditQueue"/>
      <bean ref="myNotificationListener" method="onMessage"/>
      <setHeader headerName="subject">
        <constant>new incident reported</constant>
      </setHeader>
      <removeHeader headerName="JMSTimestamp">
      </removeHeader>
      <to uri="smtps://smtp.gmail.com:465?username=aaa@gmail.com&amp;password=bbb&amp;debugMode=true"/>
    </route>



Solution:
Please add camel-mail.jar and mail.jar to your classpath

Unable to locate Spring NamespaceHandler for XML schema namespace [http://camel.apache.org/schema/spring]

Problem:
Exception in thread "main" org.springframework.beans.factory.parsing.BeanDefinitionParsingException: Configuration problem: Unable to locate Spring NamespaceHandler for XML schema namespace [http://camel.apache.org/schema/spring]
Offending resource: class path resource [camel-context.xml]


Solution:
Very simple: Add camel-spring.jar to classpath





Tuesday, 29 April 2014

How to access OracleConnection in Tomcat server

If you ever want to access underlying Oracle Connection in an application deployed in Tomcat server, you can use the following simple code.

try{
 java.sql.Connection connection = ConnectionManager.getConnection();
 OracleConnection ocn = connection.unwrap( OracleConnection.class );       

 XMLType xmlReq = new XMLType(ocn, rxml);
 ...
 ...
 ...

}catch(Exception e){
 e.printStackTrace();
}



For more documentation, please read http://docs.oracle.com/javase/6/docs/api/java/sql/Wrapper.html#unwrap(java.lang.Class)

Hope this tip is useful to you.

Tuesday, 25 March 2014

How to access oracle.jdbc.OracleCallableStatement object in JBoss or Tomcat servers

Though it may not be advisable to directly work on the underlying vendor specific API such as oracle.jdbc.OracleCallableStatement, how would do you access it when situation arises.

Use the below snippet for a JBoss deployed application:
public static OracleCallableStatement getOracleCallableStatement(java.sql.CallableStatement callableStatement) throws SQLException {
     OracleCallableStatement ocs = null;

if(callableStatement instanceof org.jboss.resource.adapter.jdbc.WrappedCallableStatement) {
            org.jboss.resource.adapter.jdbc.WrappedCallableStatement wc = (org.jboss.resource.adapter.jdbc.WrappedCallableStatement) callableStatement;
            Statement stmt = wc.getUnderlyingStatement();
            ocs = (OracleCallableStatement) stmt;
        }

       return ocs;
}

For java doc, please read http://docs.jboss.org/jbossas/javadoc/4.0.2/org/jboss/resource/adapter/jdbc/WrappedCallableStatement.html


Use the below snippet for a Tomcat deployed application:
public static OracleCallableStatement getOracleCallableStatement(java.sql.CallableStatement callableStatement) throws SQLException {
     OracleCallableStatement ocs = null;

if( callableStatement instanceof org.apache.tomcat.dbcp.dbcp.DelegatingCallableStatement) {
            org.apache.tomcat.dbcp.dbcp.DelegatingCallableStatement dcs = (org.apache.tomcat.dbcp.dbcp.DelegatingCallableStatement) callableStatement;
            ocs = (OracleCallableStatement)dcs.getInnermostDelegate();
        } 
       return ocs;
}


Hope this is useful. Let me know if comments.