Solution for JpsException - The default DB policy store is missing for mixed mode

There are three types of policy store configurations supported by the WebLogic server and it comes with the file based policy store by default. The policy store can be maintained at file level, LDAP level or database level. In production mode, it is not recommended to use file based policy store since it is not as scalable as the other two types and also the LDAP based policy store adds an extra security layer.
The WebLogic server can be configured to use any one of the three different types of policy stores. There are two ways by which you can migrate the file based policy store to either database or LDAP based.

Using reassociateSecurityStore WLST command

Using Security Provider Configuration in Enterprise Manager

jps-config.xml is the file which stores the security configuration. The default jpsContext holds the configuration of the entire domain since this file is domain level. In case if you wish to configure application level security, you can embed another jpsContext in it. Following snippet shows a generic jpsContext of a WebLogic domain which is reassociated with an Oracle Internet Directory server. Note the .ldap extensions at the end of several service instance references.
<jpsContext name="default">
    <serviceInstanceRef ref="credstore.ldap"/>
    <serviceInstanceRef ref="keystore.ldap"/>
    <serviceInstanceRef ref="policystore.ldap"/>
    <serviceInstanceRef ref="audit.ldap"/>
    <serviceInstanceRef ref="trust"/>
    <serviceInstanceRef ref="pdp.service"/>
    <serviceInstanceRef ref="attribute"/>
    <serviceInstanceRef ref="idstore.ldap"/>
</jpsContext>
Now let’s move to the main reason behind this blog post. Recently I configured one of my Oracle Adaptive Access Manager server to use OID based policy store. It is mandatory to restart the WebLogic server once it is reassociated. During startup, I encountered the following error and the server could not be started:
<Critical> <WebLogicServer> <BEA-000386> <Server subsystem failed. Reason: weblogic.security.SecurityInitializationException: The loading of OPSS java security policy provider failed due to exception, see the exception stack trace or the server log file for root cause. If still see no obvious cause, enable the debug flag -Djava.security.debug=jpspolicy to get more information. Error message: JPS-04001: Cannot read the default policy store.
weblogic.security.SecurityInitializationException: The loading of OPSS java security policy provider failed due to exception, see the exception stack trace or the server log file for root cause. If still see no obvious cause, enable the debug flag -Djava.security.debug=jpspolicy to get more information. Error message: JPS-04001: Cannot read the default policy store.
at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.loadOPSSPolicy(CommonSecurityServiceManagerDelegateImpl.java:1402)
at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initialize(CommonSecurityServiceManagerDelegateImpl.java:1022)
at weblogic.security.service.SecurityServiceManager.initialize(SecurityServiceManager.java:873)
at weblogic.security.SecurityService.start(SecurityService.java:141)
at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
Truncated. see log file for complete stacktrace
Caused By: oracle.security.jps.JpsRuntimeException: JPS-04001: Cannot read the default policy store.
at oracle.security.jps.az.internal.runtime.service.PDPServiceImpl.<init>(PDPServiceImpl.java:355)
at oracle.security.jps.az.internal.runtime.provider.PDPServiceProvider.getInstance(PDPServiceProvider.java:89)
at oracle.security.jps.internal.core.runtime.ContextFactoryImpl.findServiceInstance(ContextFactoryImpl.java:139)
at oracle.security.jps.internal.core.runtime.ContextFactoryImpl.getContext(ContextFactoryImpl.java:170)
at oracle.security.jps.internal.core.runtime.ContextFactoryImpl.getContext(ContextFactoryImpl.java:191)
Truncated. see log file for complete stacktrace
Caused By: oracle.security.jps.service.policystore.PolicyStoreException
at oracle.security.jps.az.common.pd.service.PDServiceFinder.getPolicyDistributionService(PDServiceFinder.java:65)
at oracle.security.jps.az.internal.runtime.service.PDPServiceImpl.initializeMixedMode(PDPServiceImpl.java:491)
at oracle.security.jps.az.internal.runtime.service.PDPServiceImpl.initial(PDPServiceImpl.java:467)
at oracle.security.jps.az.internal.runtime.service.PDPServiceImpl.<init>(PDPServiceImpl.java:352)
at oracle.security.jps.az.internal.runtime.provider.PDPServiceProvider.getInstance(PDPServiceProvider.java:89)
Truncated. see log file for complete stacktrace
Caused By: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at oracle.security.jps.az.common.pd.service.PDServiceFinder.getPolicyDistributionService(PDServiceFinder.java:53)
Truncated. see log file for complete stacktrace
Caused By: java.lang.RuntimeException: oracle.security.jps.JpsException: The default DB policy store is missing for mixed mode.
at oracle.security.jps.az.internal.management.pd.PD.getPolicyDistributionService(PD.java:104)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
Truncated. see log file for complete stacktrace
Caused By: oracle.security.jps.JpsException: The default DB policy store is missing for mixed mode.
at oracle.security.jps.az.internal.management.pd.PD.getPolicyDistributionService(PD.java:104)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
Truncated. see log file for complete stacktrace
> 
I have observed that you get this exception (oracle.security.jps.JpsRuntimeException: JPS-04001: Cannot read the default policy store.) whenever there is any problem in the security configurations of the WebLogic server. I think it is a generic exception which will not let you start the server. To solve this problem, you would need to observe the stacktrace more carefully. Most of the times the observation would lead you to verify the jps-config.xml file.
You can see the exception - oracle.security.jps.JpsException: The default DB policy store is missing for mixed mode. And from the stacktrace, you can see that it arose from oracle.security.jps.az.internal.management.pd.PD.getPolicyDistributionService(PD.java:104) which means there is some inconsistency in the policy distribution service. Now you will need to check the PDP instance in the jps-config.xml file. You can see the policy distribution mode whose value is specified as mixed in my case :

<property name="oracle.security.jps.runtime.pd.client.policyDistributionMode" value="mixed"/>.

The parameter oracle.security.jps.runtime.pd.client.policyDistributionMode specifies the mode of policy distribution whose accepted values are controlled-push, non-controlled or controlled-pull. The value mixed indicates that the mode is a combination of controlled-pull and uncontrolled mode and it expects database policy store. Since I am using OID based policy store, I can get away with this exception by just removing this property from the pdp.service service instance which means the default distribution mode would be used during and after the initialization of security configurations of the WebLogic domain.
In this way, to solve a security initialization problem during WebLogic server startup, you would need to analyse the stacktrace, from which you can identify the service instance or service provider in which there are high chances of inconsistencies to be found.
This post is not just about the solution of the issue mentioned in the stacktrace but it gives an approach one should adapt in solving any problem of which you have sufficient stacktrace available.

OID Performance Tuning

Oracle Internet Directory is highly scalable and manageable in terms of performance tuning as per the hardware resources and high availability configurations.
In this blog I will explain the parameters which can improve the performance of OID.

1. Database Parameters:
                                Recommended values
sga_target,sga_max_size            upto 60-70% of the available
                                   RAM for database machine
db_cache_size                    upto 60-70% of the available 
                                   RAM for database machine
shared_pool_size                  500M
session_cached_cursors            100
processes                        500
pga_aggregate_target              1-4GB
job_queue_processes               1 or more
max_commit_propagation_delay       99 or lower


2. LDAP Server Attributes:
                                  Recommended values
orclmaxcc                        10 - Number of DB Connections 
                                   per Server Processes
orclserverprocs                  4 - Number of OID LDAP Server 
                                   Processes which should be 
                                   equal to the number of cpu 
                                   cores on the system
orclgeneratechangelog             0 - Disables change log 
                                   generation
orclldapconntimeout               60 - LDAP Connection Timeout
orclmatchdenabled                 0 - Enable MatchDN Processing



3. OID Authenticator Parameters:
    If you have configured Oracle Internet Directory Authenticator in myrealm to retrieve users from OID, following parameters can be changed to optimize the performance:
                                            
                                   Recommended values
Group Membership Searching        limited
Connection Pool Size              120
Connect Timeout                  120
Cache Size                       51200
Cache TTL                        300

4. jps-config Parameters

    If the weblogic server is reassociated to an OID and the application policies are stored in it, following parameters should be added in policystore.ldap serviceInstance in jps-config.xml to make the retrieval of policies faster by caching them.

    <property name="oracle.security.jps.policystore.rolemember.cache.type" value="STATIC"/>
    <property name="oracle.security.jps.policystore.rolemember.cache.strategy" value="NONE"/>
    <property name="oracle.security.jps.policystore.rolemember.cache.size" value="100"/>
    <property name="oracle.security.jps.policystore.policy.lazy.load.enable" value="true"/>
    <property name="oracle.security.jps.policystore.policy.cache.strategy" value="NONE"/>
    <property name="oracle.security.jps.policystore.policy.cache.size" value="1000000"/>
    <property name="oracle.security.jps.policystore.refresh.enable" value="true"/>
    <property name="oracle.security.jps.policystore.refresh.purge.timeout" value="43200000"/>
    <property name="oracle.security.jps.ldap.policystore.refresh.interval" value="6000000"/>
    <property name="oracle.security.jps.policystore.rolemember.cache.warmup.enable" value="true"/>
    <property name="connection.pool.min.size" value="120"/>
    <property name="connection.pool.max.size" value="120"/>
    <property name="connection.pool.provider.type" value="IDM"/>
    <property name="connection.pool.timeout" value="300000"/>
    <property name="connection.pool.provider.type" value="5"/>

   OID and weblogic server restarts are required after modifying the above parameters. They can still be optimized depending on the availability of the hardware resources.
   Ref : http://docs.oracle.com/cd/E23943_01/core.1111/e10108/oid.htm

OID Policystore Migration

                 Policystore is basically a node in the hierarchical structure of Oracle Internet Directory where all the application policies are stored. There will be cases where you will want to replicate the policystore structure to some other OID instance in development or in production mode. This is useful in cases where you want to maintain the same application policies across multiple environments. Of course you can use the same OID across different environments but it will be very difficult for troubleshooting
                 OID provides few set of commands using which a policystore can be exported to an LDIF file and then that LDIF file can be imported on another OID.
                 This blog explains the use of ldifwrite and bulkload commands which are used to export the policystore or to be specific any node to an LDIF file and import the LDIF file respectively.
                 Following environment variables must be set before proceeding:

1. WLS_HOME=<path_where_middleware_is_installed>
2. ORACLE_HOME=$WLS_HOME/Oracle_IDM1
3. ORACLE_INSTANCE=$ORACLE_HOME/asinst_1

                 Following directories must be added to the PATH variable:

1. ORACLE_HOME/bin
2. ORACLE_HOME/ldap/bin
3. ORACLE_INSTANCE/bin

                OID instance must be shut down before performing bulkload commands to avoid any inconsistencies in the loading.
                    opmnctl stopall

                Follow the below steps to replicate a node in one OID to another :

• Use ldifwrite to export a node to an ldif file :

        ldifwrite connect="connect_string" basedn="source_dn" file="location.ldif"
     e.g. If you want to export 'cn=mynode,cn=jpsContext,cn=jpsroot' (basedn)  which resides in ODS schema of OIDDB_SOURCE (connect_string as specified in ORACLE_INSTANCE/config/tnsnames.ora) to a file source.ldif which is at location (/u01/export/source.ldif), use the following command :
        ldifwrite connect="OIDDB_SOURCE" basedn="cn=mynode,cn=jpsContext,cn=jpsroot" file="/u01/export/source.ldif"

• Use bulkload to import the ldif file and generate the intermediate SQL*Loader files :

      bulkload connect="connect_string" check="true" generate="true" recover="true" file="location.ldif"
     e.g. Intermediate SQL*Loader files which are to be executed for ODS schema in OIDDB_TARGET (connect_string) can be generated from /u01/export/source.ldif (file) with the following command :
          bulkload connect="OIDDB_TARGET" check="true" generate"true" recover="true" file="/u01/export/source.ldif"

              check flag parses and verifies the input LDIF file to find any corrupt data
              generate flag generates the intermediate files in SQL*Loader format
              restore flag restores the schema in case any problem arises during bulkload operation

• Use bulkload to load the intermediate SQL*Loader files in the target OID schema :

       bulkload connect="connect_string" load="true"
      e.g. Intermediate files generated by 2 option can be loaded in the schema OIDDB_TARGET by the following command :
           bulkload connect="OIDDB_TARGET" load="true"

   After the completion of bulkload execution, start the OID server instance:
                 opmnctl startall

  The logs file generated by the bulkload tool are at location :
        ORACLE_INSTANCE/diagnostics/logs/OID/tools

1. bulkload.log : output log
2. duplicateDN.log : list of duplicate DNs found during loading
3. *.ctl and *.dat : intermediate files

        ORACLE_INSTANCE/OID/load

1. badentry.ldif : list of bad LDIF entries
2. dynGrp.ldif : list of dynamic group entries that can be added using ldapadd command
3. bsl_*.log : intermediate log files generated by SQL*Loader

      If there are any errors during indexing phase of loading, following command can re-create the indices :
           bulkload connect="OIDDB_TARGET" index="true"
      Indices can be verified by using the command :
              bulkload connect="OIDDB_TARGET" check="true" index="true"

     In this way you can migrate the application policystore or any node for that matter from one OID instance to another. The detailed explanation of all the attributes used in the above commands can be found at the location http://docs.oracle.com/cd/E25054_01/oid.1111/e10029/bulktools.htm