Solution for JpsException - The default DB policy store is missing for mixed mode

There are three types of policy store configurations supported by the WebLogic server and it comes with the file based policy store by default. The policy store can be maintained at file level, LDAP level or database level. In production mode, it is not recommended to use file based policy store since it is not as scalable as the other two types and also the LDAP based policy store adds an extra security layer.
The WebLogic server can be configured to use any one of the three different types of policy stores. There are two ways by which you can migrate the file based policy store to either database or LDAP based.

Using reassociateSecurityStore WLST command

Using Security Provider Configuration in Enterprise Manager

jps-config.xml is the file which stores the security configuration. The default jpsContext holds the configuration of the entire domain since this file is domain level. In case if you wish to configure application level security, you can embed another jpsContext in it. Following snippet shows a generic jpsContext of a WebLogic domain which is reassociated with an Oracle Internet Directory server. Note the .ldap extensions at the end of several service instance references.
<jpsContext name="default">
    <serviceInstanceRef ref="credstore.ldap"/>
    <serviceInstanceRef ref="keystore.ldap"/>
    <serviceInstanceRef ref="policystore.ldap"/>
    <serviceInstanceRef ref="audit.ldap"/>
    <serviceInstanceRef ref="trust"/>
    <serviceInstanceRef ref="pdp.service"/>
    <serviceInstanceRef ref="attribute"/>
    <serviceInstanceRef ref="idstore.ldap"/>
</jpsContext>
Now let’s move to the main reason behind this blog post. Recently I configured one of my Oracle Adaptive Access Manager server to use OID based policy store. It is mandatory to restart the WebLogic server once it is reassociated. During startup, I encountered the following error and the server could not be started:
<Critical> <WebLogicServer> <BEA-000386> <Server subsystem failed. Reason: weblogic.security.SecurityInitializationException: The loading of OPSS java security policy provider failed due to exception, see the exception stack trace or the server log file for root cause. If still see no obvious cause, enable the debug flag -Djava.security.debug=jpspolicy to get more information. Error message: JPS-04001: Cannot read the default policy store.
weblogic.security.SecurityInitializationException: The loading of OPSS java security policy provider failed due to exception, see the exception stack trace or the server log file for root cause. If still see no obvious cause, enable the debug flag -Djava.security.debug=jpspolicy to get more information. Error message: JPS-04001: Cannot read the default policy store.
at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.loadOPSSPolicy(CommonSecurityServiceManagerDelegateImpl.java:1402)
at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initialize(CommonSecurityServiceManagerDelegateImpl.java:1022)
at weblogic.security.service.SecurityServiceManager.initialize(SecurityServiceManager.java:873)
at weblogic.security.SecurityService.start(SecurityService.java:141)
at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
Truncated. see log file for complete stacktrace
Caused By: oracle.security.jps.JpsRuntimeException: JPS-04001: Cannot read the default policy store.
at oracle.security.jps.az.internal.runtime.service.PDPServiceImpl.<init>(PDPServiceImpl.java:355)
at oracle.security.jps.az.internal.runtime.provider.PDPServiceProvider.getInstance(PDPServiceProvider.java:89)
at oracle.security.jps.internal.core.runtime.ContextFactoryImpl.findServiceInstance(ContextFactoryImpl.java:139)
at oracle.security.jps.internal.core.runtime.ContextFactoryImpl.getContext(ContextFactoryImpl.java:170)
at oracle.security.jps.internal.core.runtime.ContextFactoryImpl.getContext(ContextFactoryImpl.java:191)
Truncated. see log file for complete stacktrace
Caused By: oracle.security.jps.service.policystore.PolicyStoreException
at oracle.security.jps.az.common.pd.service.PDServiceFinder.getPolicyDistributionService(PDServiceFinder.java:65)
at oracle.security.jps.az.internal.runtime.service.PDPServiceImpl.initializeMixedMode(PDPServiceImpl.java:491)
at oracle.security.jps.az.internal.runtime.service.PDPServiceImpl.initial(PDPServiceImpl.java:467)
at oracle.security.jps.az.internal.runtime.service.PDPServiceImpl.<init>(PDPServiceImpl.java:352)
at oracle.security.jps.az.internal.runtime.provider.PDPServiceProvider.getInstance(PDPServiceProvider.java:89)
Truncated. see log file for complete stacktrace
Caused By: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at oracle.security.jps.az.common.pd.service.PDServiceFinder.getPolicyDistributionService(PDServiceFinder.java:53)
Truncated. see log file for complete stacktrace
Caused By: java.lang.RuntimeException: oracle.security.jps.JpsException: The default DB policy store is missing for mixed mode.
at oracle.security.jps.az.internal.management.pd.PD.getPolicyDistributionService(PD.java:104)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
Truncated. see log file for complete stacktrace
Caused By: oracle.security.jps.JpsException: The default DB policy store is missing for mixed mode.
at oracle.security.jps.az.internal.management.pd.PD.getPolicyDistributionService(PD.java:104)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
Truncated. see log file for complete stacktrace
> 
I have observed that you get this exception (oracle.security.jps.JpsRuntimeException: JPS-04001: Cannot read the default policy store.) whenever there is any problem in the security configurations of the WebLogic server. I think it is a generic exception which will not let you start the server. To solve this problem, you would need to observe the stacktrace more carefully. Most of the times the observation would lead you to verify the jps-config.xml file.
You can see the exception - oracle.security.jps.JpsException: The default DB policy store is missing for mixed mode. And from the stacktrace, you can see that it arose from oracle.security.jps.az.internal.management.pd.PD.getPolicyDistributionService(PD.java:104) which means there is some inconsistency in the policy distribution service. Now you will need to check the PDP instance in the jps-config.xml file. You can see the policy distribution mode whose value is specified as mixed in my case :

<property name="oracle.security.jps.runtime.pd.client.policyDistributionMode" value="mixed"/>.

The parameter oracle.security.jps.runtime.pd.client.policyDistributionMode specifies the mode of policy distribution whose accepted values are controlled-push, non-controlled or controlled-pull. The value mixed indicates that the mode is a combination of controlled-pull and uncontrolled mode and it expects database policy store. Since I am using OID based policy store, I can get away with this exception by just removing this property from the pdp.service service instance which means the default distribution mode would be used during and after the initialization of security configurations of the WebLogic domain.
In this way, to solve a security initialization problem during WebLogic server startup, you would need to analyse the stacktrace, from which you can identify the service instance or service provider in which there are high chances of inconsistencies to be found.
This post is not just about the solution of the issue mentioned in the stacktrace but it gives an approach one should adapt in solving any problem of which you have sufficient stacktrace available.

A Tour of Oracle Privileged Account Manager

Usage management of sensitive accounts of LDAP, Database, Linux or custom applications using Oracle Privileged Account Manager 11gR2PS1

Downloads:

Oracle Identity Management

Overview:

Oracle Privileged Account Manager is a new product added in the Oracle Identity Management 11gR2 release. As its name suggests, it deals with the management of privileged accounts. By the word “privileged account” we mean that these accounts can access the sensitive data in the organization. Following are the examples of the privileged accounts:

• Unix root account
• Database SYSDBA / SYSOPER accounts
• Windows Administrator accounts
• Application accounts which access sensitive information
• System Administrator accounts

Oracle Privileged Account Manager provides an easy to implement and versatile solution for managing high risk accounts. It lets you manage the password policies and the usage policies (e.g. time duration for which you can use the account) for the accounts. It leverages Identity Connector Framework to integrate with third party applications which are supported out of the box. It supports the break the glass facility where one person performs the tasks of another person in case the person is not available due to some emergency. One of the highlighting features is the ability of OPAM to provide comprehensive auditing and reporting of who has acted on a particular account at certain point in time as well as the real time data. It can be combined with Oracle BI Publisher to generate reports. Figure 1 shows the overall architecture of OPAM.

Figure 1

There are two ways in which you can access the Oracle Privileged Account Manager features: Graphical User Interface (GUI) and Command Line Interface (CLI). The latter can be extended to invoke all CLI supported operations through Java API which is given this post. The GUI and CLI interact with the Oracle Privileged Account Manager Server over REST services. The OPAM Server is a component that handles password requests, generates passwords, and protects the password keystore. Figure 2 shows the GUI, CLI and other clients communicate with OPAM using the RESTful interface.

Figure 2

Authentication:

 The authentication framework of the Oracle Privileged Account Manager leverages SAML based token authentication. It can be integrated with Oracle Access Manager Single Sign On. OPAM CLI Supports HTTP Basic Authentication over SSL to connect to OPAM Server.

Authorization:

 There are several Admin Roles supported by OPAM to restrict the sensitive usage of the application:

• Application Configurator – OPAM_APPLICATION_CONFIGURATOR
• Security Administrator – OPAM_SECURITY_ADMIN
• Security Auditor – OPAM_SECURITY_AUDITOR
• User Manager – OPAM_USER_MANAGER

Administration:

1.     Targets:

A target in OPAM is a system which contains users, and relies on users, system, or application accounts. You need to register the target system in order to manage the accounts associated with the target.  Currently as of release 11.1.2.1.0 OPAM supports four types of pre-built targets. You can see the four supported targets in Figure 3.

Figure 3

Generally adding a target involves providing the values of the host, port, domain and other minimum useful information required to retrieve the accounts from the target system. The two targets needs extra attention - The database target is supported for Oracle, MSSQL, MYSQL, DB2 and Sybase. Lockbox allows us to manage a specific target without configuring any connector.

2.     Accounts:

An account represents the privileged account on a target system. OPAM does not create or delete accounts on the target system; it only manages the existing accounts. When a new account is added in OPAM server, only a reference to that account gets created. Deletion of account from OPAM does not delete the account from the target. OPAM enables you to specify whether an account is shared or not shared. Shared accounts enable multiple users to use the account at the same time. Default (unshared) accounts can be used by only one user at a time. Below Figure 4 shows the database accounts containing the word sys in a database target.

Figure 4

3.     Grantee:

Grantees are users or groups that have been granted access to one or more privileged accounts. Users or groups are assigned to a privileged account in the Account tab. As you can see in Figure 5, user.0 is a grantee for the account DEVOAM_OPAM and there is no group specified.

Figure 5

4.     Check out and Check in:

Administrators or end users can check out an account if they have been granted access to that account. When a user checks out an account, by providing a comment, it gets the access to the password of the account. The password is generated based on the Password Policy attached to the privileged account. On successful checking out, unshared accounts become unavailable for other users to check out until the account is checked in again by the user. Checking in of account resets the password of the account. In case of shared accounts, the password is reset only when all the users check in the account. Figure 6 shows that the accounts which are granted to the logged in user are checked in and ready to be checked out.

Figure 6

5.     Password Policies:

The password policies capture the password construction rules enforced by a specific target on an associated privileged account. Every privileged account must have an associated password policy. OPAM provides Default Password Policy. However you can create custom password policies as per your requirements.

6.     Usage Policies:

The usage policies define when and how a privileged account can be used by a grantee. These are applied at the level of a grant. OPAM provides Default Usage Policy. However you can create custom usage policies as per your requirements.

7.     Reports:

The reports provide the real time information about the status of accounts and targets being managed by OPAM. OPAM provides support for 3 types of reports:

• Deployment Reports : View information about how targets and privileged accounts are currently deployed
• Usage Reports : View information about how privileged accounts are currently being used in the deployment
• Failure Reports : View information about current state of target and account failures

8.     ICF Connectors:

Connectors enable OPAM to interact with the third party target systems. OPAM leverages connectors that are compliant with the ICF standard. OPAM can use custom connectors to interact with the target systems other than the traditional LDAP, database and UNIX system. After the connector is developed and ready to be integrated with OPAM, the connector bundle needs to be put in a location where OPAM can access it at runtime. All the connector related configurations are stored in the file ORACLE_HOME/opam/config/opam-config.xml which schematically is governed by ORACLE_HOME/opam/config/opam-config.xsd. If you have made correct <connectorConfig> entries in opam-config.xml, you will see the new target in the Target dropdown after the server is restarted.

Command Line Interface:

Oracle Privileged Account Manager lets you interact with the OPAM server via HTTP Basic Authentication over SSL. You can perform certain tasks via command line which you can do via GUI. CLI can be launched by ORACLE_HOME/opam/bin/opam.sh (UNIX) or ORACLE_HOME/opam/bin/opam.bat (Windows) . Supported commands are - addaccount, addtarget, checkin, checkout, displayallaccounts, displayallgroups, displayalltargets, displayallusers, displaycheckedoutaccounts, displaydomaintree, displaytargettypetree, grantgroupaccess, grantuseraccess, removeaccount, removegroupaccess, removetarget, removeuseraccess, retrieveaccount, retrievegrantees, retrievegroup, retrievetarget, retrieveuser, searchaccount, searchgroup, searchtarget, searchuser, showpassword

Following command shows an example of how to issue addaccount command:

-url https://hostname:sslport/opam -u <username> [-p <password>] [debug] -x addtarget -targetname <targetname> -host <hostname> -port 22 -organization <organization>

Interacting with RESTful Interface:

OPAM Server has exposed certain RESTful interfaces which are consumed by the GUI and also can be incorporated with other applications which provide the ability to consume REST services. Using RESTful interface of OPAM, you can perform several operations on Targets, Accounts, Users, Groups, Password and Usage Policies. Following is the Java code sample showing the basic REST operations that you can perform:

OPAMRestClient.java

package com.mycompany.opam.demo.rest.client;

import java.io.IOException;

import java.io.InputStream;

import java.io.InputStreamReader;

import java.net.MalformedURLException;

import java.net.URL;

import javax.net.ssl.HostnameVerifier;

import javax.net.ssl.HttpsURLConnection;

import javax.net.ssl.SSLSession;

import org.apache.commons.codec.binary.Base64;

/**

 * @author Ganesh

 *

 */

public class OPAMRestClient {

        /**

         * @param args

         */

        public static void main(String[] args) {

                OPAMRestClient client = new OPAMRestClient();

                client.authAndGet();

                client.checkOutAccount();

                client.checkInAccount();

        }

        private void authAndGet() {

                String webPage = "https://192.168.1.31:18102/opam/";

                String result = execute(webPage, "opamadmin", "welcome1", "GET");

                System.out.println("*** RESULT ***");

                System.out.println(result);

        }

        private void checkOutAccount() {

                String webPage = "https://192.168.1.31:18102/opam/account/3a32f71b320645f7b9b594fc8e864089/checkout";

                String result = execute(webPage, "user.0", "welcome1", "PUT");

                System.out.println("*** RESULT ***");

                System.out.println(result);

        }

        private void checkInAccount() {

                String webPage = "https://192.168.1.31:18102/opam/account/3a32f71b320645f7b9b594fc8e864089/checkin";

                String result = execute(webPage, "user.0", "welcome1", "PUT");

                System.out.println("*** RESULT ***");

                System.out.println(result);

        }

        /**

         * @param webPage

         * @return

         * @throws MalformedURLException

         * @throws IOException

         */

        private String execute(String webPage, String name, String password,

                        String requestMethod) {

                String authString = name + ":" + password;

                System.out.println("auth string: " + authString);

                byte[] authEncBytes = Base64.encodeBase64(authString.getBytes());

                String authStringEnc = new String(authEncBytes);

                System.out.println("Base64 encoded auth string: " + authStringEnc);

                URL url;

                HttpsURLConnection httpsCon = null;

                String result = null;

                try {

                        url = new URL(webPage);

                        httpsCon = (HttpsURLConnection) url.openConnection();

                        httpsCon.setRequestMethod(requestMethod);

                        httpsCon.setHostnameVerifier(new HostnameVerifier() {

                                @Override

                                public boolean verify(String hostname, SSLSession session) {

                                        return true;

                                }

                        });

                        httpsCon.setRequestProperty("Authorization", "Basic "

                                        + authStringEnc);

                        httpsCon.connect();

                        InputStream is = httpsCon.getInputStream();

                        InputStreamReader isr = new InputStreamReader(is);

                        int numCharsRead;

                        char[] charArray = new char[1024];

                        StringBuffer sb = new StringBuffer();

                        while ((numCharsRead = isr.read(charArray)) > 0) {

                                sb.append(charArray, 0, numCharsRead);

                        }

                        result = sb.toString();

                } catch (MalformedURLException e) {

                        e.printStackTrace();

                } catch (IOException e) {

                        e.printStackTrace();

                } finally {

                        if (httpsCon != null) {

                                httpsCon.disconnect();

                        }

                }

                return result;

        }

}

Integration with Oracle Identity Management solutions:

Oracle Privileged Account Manager can be integrated with Oracle Identity Governance, Oracle Access Management and Oracle Directory Services – the three building tiers of Oracle Identity Management Suite. OPAM provides the ability to manually request access to accounts and to get approvals for requests in Oracle Identity Manager. OPAM can be protected by enabling Single Sign On using a Webgate agent in Oracle Access Manager.

Backend:

Following is a list of tables from DEV_OPAM schema used by the Oracle Privileged Account Manager:

TARGET, ACCOUNT, GRANTS, USGPOL, PWDPOL, FAILURE, ACTSESSION, FLEXFIELD, FLEXOBJ, FLEXSECFIELD

The Metadata information and passwords managed by OPAM are encrypted and persisted in the Oracle Database. It is recommended to leverage Oracle Database Vault and Transparent Data Encryption (TDE) for additional security.

Conclusion:

Oracle Privileged Account Manager is a complete product which has a comprehensive set of features that are required to manage the password lifecycle of privileged accounts – an  activity for which there was no solution available earlier. OPAM delivers more value when used in conjunction with other Oracle Identity Management Products.

References:

http://www.oracle.com/technetwork/middleware/id-mgmt/overview/opam-wp-11gr2-1697093.pdf

http://www.oracle.com/technetwork/middleware/id-mgmt/overview/opam-ds-11gr2-1697091.pdf

http://docs.oracle.com/cd/E27559_01/admin.1112/e27152/toc.htm