Weblogic Server Not Starting Due to Expired Database Accounts

If you are unable to start your weblogic server and you are getting the following errors, it is being caused by your oracle accounts expiring and the fix is quite easy.

Errors

####<Apr 23, 2013 9:09:16 AM MST> <Error> <Deployer> <localhost.osbsoavm> <AdminServer> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JsrdyCXFw000jzwkno1HTf7V000002> <1366733356615> <BEA-149205> <Failed to initialize the application 'wlsbjmsrpDataSource' due to error weblogic.application.ModuleException: .
weblogic.application.ModuleException:
 .........
Caused By: weblogic.common.ResourceException: weblogic.common.ResourceException: Could not create pool connection. The DBMS driver exception was: ORA-28001: the password has expired

<BEA-280061> <The persistent store "JDBCStore-0" could not be deployed: weblogic.store.PersistentStoreException: Can't find JDBC DataSource wlsbjmsrpDataSource: javax.naming.NameNotFoundException: Unable to resolve 'wlsbjmsrpDataSource'. Resolved ''; remaining name 'wlsbjmsrpDataSource'
weblogic.store.PersistentStoreException: Can't find JDBC DataSource wlsbjmsrpDataSource: javax.naming.NameNotFoundException: Unable to resolve 'wlsbjmsrpDataSource'. Resolved ''; remaining name 'wlsbjmsrpDataSource'

Fix 

            1. Check to see if any of the accounts are expired

                           select username, profile, account_status, expiry_date from dba_users;

2. If no accounts are expired, you can skip to step 7

3.  Dynamically create SQL that will unexpired the expired accounts.  You can unexpired the 

     account by resetting the password.   NOTE: You will need to substitute <password> for the password you are using for your user.

                            select 'ALTER USER ' || username || ' identified by <password>;'

                            from dba_users where account_status like 'EXPIRED%' and username != 'XS$NULL';

                  4. Execute the SQL generated in Step 3

                  5.  Dynamically create SQL that will unlock the locked accounts.

                             select 'ALTER USER ' || username || ' account unlock;' from dba_users where 

                             account_status like 'LOCKED%' and username != 'XS$NULL';

6.  Execute the SQL generated in Step 5

7. Modify the profile assigned to the accounts that you don’t want to expire so the 

    PASSWORD_LIFE_TIME is set to UNLIMITED.  This will keep them from expiring 

    again.  In my case, I needed to update the DEFAULT profile.

alter profile DEFAULT LIMIT PASSWORD_LIFE_TIME UNLIMITED ;

Enabling Weblogic to use SSL

Enable SSL on for server in Weblogic

1) Open Weblogic console

2) Navigate to <Domain> -> Environment -> Servers

3) Select the server you want to configure

4) Goto the Configuration -> General tab.

5) Check the box next to SSL Listen Port Enabled.

6) Specify the port that you want to listen for SSL traffic on.

7) Select Save.   Note: You do not need to restart the server for this change.

Creating XSD from XML document using JDeveloper 11g

JDeveloper 11g gives you the ability to quickly and easily create an XML schema document (XSD) from an XML document.   Below are the steps needed.

1) In JDeveloper, go to File -> New
2) In the New Gallery window be sure the All Technologies tab is select.  Scroll down and select the XML category and then select XML Schema from XML Document.

3) A Create XML Schema from XML Document window will come up.  In this window you will need to fill in information on the schema file you want to create and you will need to select the XML file you want to use to generate the XSD.

Here is some additional information from the help menu on options on the Create XML Schema from XML Document page.

XML Schema
Enter a name for the new file. By default, the filename that JDeveloper creates for new files appears. The extension .XSD is displayed.

Directory
Enter a directory for the file. By default, the directory that JDeveloper creates to store your files appears. To store the files outside of JDeveloper's directory structure, or to redefine what the file directory is called, enter your changes.

Target Namespace
Enter a URI reference of the namespace of this schema.

Prefix
Enter an identifier for a namespace. For example, you don't need to specify an entire namespage in an XML document when referring to its types. You can instread use the the prefix.

XML Document
Choose the XML file to use to generate the schema.

Generate Enumerations
Select to use all current values in the XML data set as enumerated values for each element/attribute in the generated XML schema.

OSB Error Handler Tutorial

Error handling in OSB is fairly straight forward but can seem more complex than needed.  The following exercises will provide hands-on examples of how error handlers function in OSB.  I have also included a sample project which makes it very easy to try different scenarios that can help broaden your understanding of OSB Error Handling.  These examples are meant to give some guidance but please try different scenarios and if there is something you are curious about add it to one of the proxy services and see what OSB does.

Before we start with the hands-on exercises below is a quick overview of the key points in OSB Error Handling. 

OSB Error Handling Overview

Error handling can be configured at 4 different areas in and OSB Proxy Service.

1. Proxy Service
2. Route Node
3. Pipeline
4. Stage Node

If an error is not handled in any of these areas then it will be caught in the System error handler.

An error will be handled by the inner-most encompassing error handler.  In other words, if there is no error handler configured at the level the error occurred then the error will be processed by the next level error handler.  Below is an outline of how the error handlers are nested.

Stage Node -> Pipeline -> Proxy Service -> System Error Handler

Route Node ->  Proxy Service -> System Error Handler

Choosing an error handler action

An Error Handler is not considered completely configured until it has a Resume or Reply Action configured.  If an error handler is missing one of these actions then the other steps in the error handler will be completed but the error will be bubbled up to the next level error handler.

Reply – Will immediately reply back to the calling process with an error response and all further message processing stops. 

Resume – Message flow process will continue as if no error occurred.   The processing will continure after the node or stage that the error handler is configure in.  

Configuring Error Handlers

Error handlers are just another pipeline and can be configured like any other pipeline.  You may use an assign action, publish action…. Etc.

In the error handler a new context variable is available ($fault).  The $fault context variable contains information about any error that occurs during message flow processing and is populated before the error handler is invoked.

For more information on the contents of the $fault variable see  OSB Context Variables

Error Handler Exercises

The following exercises are simple and are meant to demonstrate how error handlers behave in OSB.  Specifically we are going to focus on how error handlers are nested and the behavior of OSB when there are no error handlers. 

To complete these exercises please import the Error_Handler_Demo project.

You can download the project at the following link.

https://docs.google.com/file/d/0B5g0v_BbuvHUV05NaVFOUWxFamM/edit

The Error_Handler_Demo project consists of 3 proxy services.

1)      ValidateID – This proxy service will look at the value of the ID passed in. 

If the ID is <= 5, then it will return a status of “Success” and reply with Success.

If the ID is between 6 and 10, it will return a status of “BusinessFault” and reply with Fault.   

If the ID is between 11 and 20 it will return a status of “TechnicalFault” and reply with Fault.

2)      ErrorHandlerDemoImpl – This proxy service will call the ValidateID proxy service.  We primarily will use this service to demonstrate how OSB handles errors that are returned from a child process.

3)      ErrorHandlerDemo – This proxy service will call the ErrorHandlerDemoImpl proxy service.  This service will be used to show how errors are handled when the error is thrown from a nested service.

ErrorHandlerDemoImpl Service

We will start with calling the ErrorHandlerDemoImpl service directly to demonstrate how it will handle errors returned from the ValidateID service.

For these examples remember that error handler order of execution is as follows when a route node is involved.

Route Node ->  Proxy Service -> System Error Handler

No Error Handler

In the following exercise we will run through several requests to show how the service will behave with no configured error handlers.

1)      Call the ErrorHandlerDemoImpl service and pass in an ID of 2.

§  Based on this ID we would expect a status value of Success to be returned in the response message. Since there were no faults, it will process as expected even though we do not have any error handlers defined.

  

2)      Call the ErrorHandlerDemoImpl service and pass in an ID of 8.

§  Based on this ID we would expect a status of BusinessFault to be returned in the response message.  Unfortunately no status gets returned.  Instead you see a soap fault with a faultstring of “BEA-380000”.   

§  This is because we do not have any error handlers configured.  When the ValidateID service returns a failure it automatically gets processed through the system error handler.  If you look at the Invocation Trace of your test call, you will see that the ValidateID service returns the correct status in the body.  It also returns a response-code of 1.  You can see this in the $outbound context variable.  We are not seeing these in the response because any unhandled faults will be processed through the system error handler and a SOAP fault will be returned.

Proxy Service Error Handler

We will now add a Proxy Service Error Handler to see how it changes the services behavior.

          1)      Add an error handler at the proxy service level.

a.       For detailed instructions on how to do this visit the following link. http://docs.oracle.com/cd/E14571_01/doc.1111/e15867/proxy_errors.htm

b.      In the error handler add a Reply and configure it to Reply With Failure.  Remember the error handler will not be considered completely configured unless it has a Reply or Resume node.

          2)      Call the ErrorHandlerDemoImpl and pass in an ID of 8.

§  Based on this value we would expect to see a response message with a status of BusinessFault returned.   This is because we now have now configured an error handler that will handle the error returned from the ValidateID service and the correct response message should be passed on.

          3)      Try passing in different values for IDs to see how the service behaves.

Route Node Error Handler

Let’s see what happens when we add a route node error handler to our proxy service.

          1)      Add a Route Node Error Handler to the ErrorHandlerDemoImpl proxy service.

a.       For details on how to do this see the following link http://docs.oracle.com/cd/E14571_01/doc.1111/e15867/proxy_errors.htm#autoId4

b.      In your Route Node Error Handler add a Reply which is configured to Reply With Failure.

          2)      Test ErrorHandlerDemoImpl with an ID of 12.

a.       As expected a response message with a  status of TechnicalFault gets returned.  Also note how the error is processed through the Route Node Error Handler instead of the Service Error Handler.

ErrorHandlerDemo Service

Now that we have the error handlers configured in the ErrorHandlerDemoImpl process, let’s look to see how that error information will get propagated to a calling service.  

For these examples remember that error handler order of execution is as follows when a stage node is involved.

Stage Node -> Pipeline -> Proxy Service -> System Error Handler

No Error Handler

The ErrorHandlerDemo proxy service does not have any error handlers defined.

Note:  Remember that the ErrorHandlerDemo service is configure to call the ErrorHandlerDemoImpl service using a Service Callout.

1)      Try calling the ErrorHandlerDemo service with an ID of 10.   When we call the ErrorHandlerDemoImpl process with this value it will return a response message with a status of “Technical Fault”.  This is something that we tested in the previous exercise. The ErrorHandlerDemo service is still going to return a SOAP fault not the correct response message with a status of TechnicalFault.  Since we do not have any error handlers defined the message will be handled by the system error handler which as we saw in our previous exercise will return a SOAP fault.

Proxy Service Error Handler

          1)      In the ErrorHandlerDemo proxy service, add a Proxy Service Error Handler.

          2)      Execute ErrorHandlerDemo service and pass in an ID value of 11.

a.       Based on our previous exercises we would now expect the service to return a response message that contains a status of TechnicalFault and this holds true in this example.  Also if you look in the invocation trace, you will see that the error is getting processed through the service handler.

Pipeline Error Handler

          1)      Add an error handler on the Request pipeline in the ErrorHandlerDemo service.

a.       For detailed steps on how to add this error handler, see the following link

http://docs.oracle.com/cd/E14571_01/doc.1111/e15867/proxy_errors.htm#autoId2

          2)      Execute the ErrorHandlerDemo service and pass in an ID value of 15.

a.       Once again we will see that there is a response message that contains a status of TechnicalFault.  The only thing that is different with this example is that the error is now being handled by the Pipeline Error Handler instead of the System Error Handler.

Stage Node Error Handler

For our last example, we will add a Stage Node Error Handler to demonstrate that it will process the error before the Pipeline or Service Error Handler.

          1)      Add an error handler on the Stage Node in the ErrorHandlerDemo service which calls the    ErrorHandlerDemoImpl service.

a.       For detailed steps on how to add this error handler, see the following link

http://docs.oracle.com/cd/E14571_01/doc.1111/e15867/proxy_errors.htm#autoId3

          2)      Execute the ErrorHandlerDemo service and pass in an ID value of 11.

a.       The response message should contain a status of TechnicalFault and the error will be processed through the Stage Node Error Handler. 

OSB Context Variables

In OSB context variables hold information about the message being processed as well as message content.  Using these variables is crucial in being able to develop OSB services.   I found it useful to have all of the information on all context variables in one place. 

Below is a list of predefined context variables

  

header - For SOAP messages the header will contain the SOAP header.  For any other  type of message the header will contain an empty SOAP header element.

body - Information held in the body will vary based on the type of message.

SOAP Messages - Contains the SOAP Body

Non-SOAP, Non-binary Messages - Will wrap the entire message content in a SOAP:Body element.

Binary Message - Contains a reference to the in memory copy of the binary message wrapped in a SOAP:Body element

Java Objects - Contains a reference to the in memory copy of the Java Object wrapped in a SOAP:Body element

attachments - Contains the MIME attachments

inbound - Contains the inbound transport headers and information on the proxy service that received the message.

service 

providerName - Specifies the name of the service provider(Read-only)

transport

uri - URI by which the message arrived

request - transport specific metadata about the request(Read-only)

response - transport specific metadata about the response

mode - communication style request (one-way) or request-response (two-way)

qualityOfService - specifies the expected quality of service when receiving a message. (Read-only)

Security

transportClient - specifies authenticated transport level user information.  

messageLevelClient - specifies the message level user information.

outbound - Contains the outbound transport headers and information about the target to which the message is being sent.

Service 

providerName - Specifies the name of the service provider (Read-Only)

operation - The name of the operation to be invoked on the target Business Service(Read-Only)

Transport

uri - URI used when sending message

request - transport specific metadata about the request

response - transport specific metadata about the response (read-only)

mode - communication style request (one-way) or request-response (two-way)

qualityOfService - specifies the expected quality of service when receiving a message.

retryCount - Specifies the number of retries to attempt when sending a message from Oracle Service Bus.

Security

doOutboundWss - can be used to disable outboutn WS-Security

fault - Contains information on errors that occurred.    is defined only in error handler pipelines and is not set in request and response pipelines

errrorCode - Specifies the error code as a string value

reason - Contains a text description of the error

details - Contains user-defined XML content related to the error

location -

Identifies the node, pipeline and stage in which the error occurred. Also identifies if the error occurred in an error handler. The sub-elements include:

node—the name of the pipeline, branch, or route node where an error occurred; a string.

pipeline—the name of the Pipeline where an error occurred (if applicable); a string.

stage—the name of the stage where an error occurred (if applicable); a string.

error-handler—indicates if an error occurred from inside an error handler; a boolean.

Choosing Between Route, Service Callout and Publish

When you are first starting with OSB it can be a little tricky to determine when to use a Route, Service Callout or a Publish node.  All three can be used to call either a Business service or a local Proxy service.  You can use the following lists to determine which will best fit your needs.

Route

1. Last node in request processing.  It can be thought of as a bridge between request pipeline processing and the response pipeline processing.
2. You can only execute one route in your Proxy Service.
3. Can only be created in a route node.
4. OSB will wait for the Route call to finish before continuing to process.
   1. If you are calling a Business service and you specify Best Effort for QoS (Quality of Service), then OSB will release the thread it is holding while the business service executes.
   2. If you are calling a Business service and you specify Exactly Once or At Least Once for QoS, then OSB will hold onto the thread while the business service executes.
   3. If you are calling a local Proxy service, then OSB will hold onto the thread until the Proxy service finishes executing.

Service Callout

1. Can have multiple Service Callout nodes in a Proxy service.
2. Pipeline processing will continue after a Service Callout.
3. Can be invoked from the request and/or response pipelines.
4. Used to enrich the incoming request or outgoing response. For example, a call to get a country code.
5. Used for real time request/response calls (Synchronous calls).
6. OSB will hold a thread and not continue until the Service Callout completes.
7. Can tie up resources and degrade performance under heavy loads.

Publish

1. Can be synchronous or asynchronous
   1. If you are calling a business service with a Quality of Service of Best Effort , then it will be an asynchronous call.
   2. If you call a business service with a Quality of Service of Exactly Once or At Least Once, OSB will wait until the processing completes in the business service completes before proceeding and it is effectively a synchronous call.
   3. If you are calling a local proxy service, OSB will wait until the processing in the local proxy service completes and it is effectively a synchronous call.
2. Can be invoked from the request and/or response pipelines.
3. Best to use when you do not need to wait for a response from the process you are calling (Fire and Forget.... Asynchronous Calls)

For more detailed information on the differences between a Route and a Service Callout including how OSB handles threads during each of these calls see OSB, Service Callouts and OQL - Part 1.

Out of Memory Error When Deploying From Eclipse

While trying to deploy OSB changes from Eclipse to Weblogic I was getting the following error:

Runtime exception occurred during publish. The publish is aborted. Please report the bug with the stack trace. The stack can be found on the log and the Error Log view. java.lang.reflect.UndeclaredThrowableException

The below error was reported in the Weblogic Server Log

<Jan 7, 2013 11:34:39 AM MST> <Error> <ALSB Console> <BEA-494002> <Internal error occured in OSBConsole : null
java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at com.bea.alsb.console.support.ConsoleSideMBeanInvocationHandler.__invoke(ConsoleSideMBeanInvocationHandler.java:113)
        Truncated. see log file for complete stacktrace
Caused By: java.lang.OutOfMemoryError: Java heap space
        at java.util.Arrays.copyOf(Arrays.java:2786)

The fix for this problem was very straight forward.  I just had to increase the max java heap space that was available to the Weblogic server.   This was done by setting the USER_MEM_ARGS environmental variable and restarting Weblogic.   Weblogic Service will use the memory settings in this variable when it is starting up. 

export USER_MEM_ARGS="-Xms256m -Xmx1024m -XX:CompileThreshold=8000 -XX:PermSize=256m  -XX:MaxPermSize=1024m"

-Xms -> Minimum Java Heap Size
-Xmx -> Maximum Java Heap Size
-XX:PermSize -> Minimum PermGen Size
-XX:MaxPermSize -> Maximum PermGen Size