Lessons from the Field: A directory transition from DSEE 6.3.1.1 to OUD 11gR2PS1

I was recently involved in a LDAP directory services transition project, from DSEE 6.3.1.1 to OUD 11gR2PS1, for a large manufacturing enterprise. Directory service is medium-sized with a few of million LDAP entries, and is accessed by a wide range of services and applications, ranging from Corporate Directory to Identity Store for Identity Management and user management for intranet and extranet portals.

Here is an overview of the steps we followed and the issues we addressed during this project to successfully transition the infrastructure to OUD.

1. Transition Assessment

Putting in place a sound methodology and design is a key success factor for a directory migration, regardless the final migration options selected.
This assessment was conducted over a period of 3 weeks and included the following:

- Identification and formalization of the main drivers and requirements for transition
- Inventory of the current directory infrastructure, including identification of the application portfolio accessing the directory
- Identification of transition options compatible with requirements
- Estimation of transition effort
- Identification of training requirement for the staff and skills required during the transition, especially people with business knowledge of the data stored in the directory.

For this project, the 4 main drivers to migrate to OUD were:

    - native support of EUS (ability to store Oracle DB accounts used by Enterprise User Security) as deployment of EUS was a Corporate decision
    - smooth integration and official support of OUD with the Oracle Identity stack, especially OIM and OAM
    - superior scalability, able to deliver SLA required by new services to be rolled out in the coming years
    - support for global account lockout

About 10,000 applications word-wide access the directory. Among them, 10 were identified as critical for the transition, mostly provisioning applications.

Figure 1: Existing DSEE topology with 4 masters, 6 read-only replicas in each DC + 2 read-only replicas in remote branches

OUD provides several options to transition from DSEE. For this project, the DSEE and OUD topologies have to cohabit for 6 months in production as transition was planned incrementally on a geo basis. Furthermore, client applications heavily relies on password policy, so strong data consistency is required across the 2 topologies during transition. Based on that, tightly-coupled cohabitation via the replication gateway was selected. In addition to that, this strategy provides smooth and incremental transition without interruption of service.

Transition analysis and design was conducted over a period of 1 week. The goal of this critical phase was to adapt schema, configuration and data to OUD, define and automate procedures to deploy an OUD directory server able to deliver a service equivalent to existing DSEE servers in staging area. Business knowledge of directory data was really important during this phase. Transition of the replication topology was also addressed during this phase.

4 OUD servers have been deployed as a directory backbone. Actual roll-out of other OUD servers (20+ servers all over the world) will be performed incrementally over the next months.

2. Transition Analysis and Design

Transition analysis and design heavily relied on the transitioning tools provided with OUD. Transitioning to OUD implies adapting ODSEE configuration (and sometimes the data) to the OUD format.
The OUD delivery provides tools to automatically adapt ODSEE configuration and data. The few configuration elements that can't be adapted automatically are identified by the OUD diagnostic tools and require manual adaptation.

The work was broken down into the following steps:

1. -Diagnose the DSEE deployment
2. -Migrate the schema, configuration and data to a reference OUD instance
3. -Validate configuration and settings
4. -Deploy additional OUD instances in a replicated topology
5. -Upgrade 2 DSEE masters in place to ODSEE 11gR1 PS2
6. -Deploy Replication Gateway to make the link between the 2 topologies
7. -Use T2P procedure (Test to Production) to roll out server configuration  in production

-

2.1 Diagnosing the DSEE deployment

OUD ships with the ds2oud tool able to diagnose DSEE configuration, schema and existing user data. You can use this tool to identify areas that are likely to require special attention during the transition.
Many of the differences spotted by this tool can be automatically migrated, especially those related to schema and server configuration. Others issues, related to advanced server configuration and user data requires manual intervention as they often require business knowledge or architectural decisions.

In order to run this tool, you must have administrative access to one running DSEE instance. For sake of security, one additional DSEE master server was deployed specifically for that purpose so that production systems are not impacted at all by this diagnostic phase.

2.2 Migrating the schema, configuration and data to a reference OUD instance

LDAP Schema
LDAP schemas were compared first. Custom schema extensions were properly imported to the OUD side. However, we faced a couple of issues with some experimental schemas, e.g. RFC 2307: OUD ships with the latest versions of the RFCs, but in this case, DSEE was using an older and incompatible version of the schema. A huge amount of existing LDAP entries were relying on the older schema, so we decided to use this old schema on OUD too.

Server Configuration
We also used ds2oud to migrate the server configuration. The -F option is used to produce a batch file containing a list of configuration changes to be applied to the OUD directory server. This batch file was reused to setup subsequent OUD servers during server roll-out.

Global parameters, database suffixes, indexes, global password policy were migrated automatically.

Note: Support of global password policy and account lockout during cohabitation of DSEE and OUD via the replication gateway requires that DSEE use the 'DS6' password policy mode. For this deployment, the DSEE topology was already using this mode, so no additional action was required on that side.

User Data
For this project, most of the transition effort was related to user data migration: LDAP was deployed more than 15 years ago. LDAP entries were provisioned over time by a huge variety of applications. The number of provisioning applications is now quite limited. However, it appeared that about 5% of the LDAP entries did not strictly conform to the LDAP schema and/or to the LDAP standard.  By default, OUD strictly enforce the LDAP standard and the LDAP schema, including attribute value syntax check. DSEE does not check attribute value syntax.

Based on that, it was decided to use that transition project to conduct a detailed data assessment and sanitization. This assessment was completed over the 1 week period as originally planned. However, it required involvement of additional stakeholders to figure out whether entries that did not match the LDAP schema were correct or incorrect from a business and application perspective.

Here are a few road blocks hit and addressed:

- Most LDAP entries contained more than one structural LDAP object class
    This non-standard setting is accepted by ODSEE but by default, it is rejected by OUD. That check was relaxed on OUD because applications creating these entries could not be modified.
- Some entries contained attribute type extensions (e.g criteria:criteria;x-custom-extension) that violate LDAP standards because they contain underscores.
    That check was relaxed on OUD as well
- Some DNs and telephoneNumber attributes contain incorrect values
    These values were cleaned up on DSEE.

Directory Metadata
The transition strategy chosen is based on the replication protocol. It provides strong data consistency and all data including directory metadata are replicated back and forth between DSEE and OUD.
For that project, directory metadata, mainly access controls, could be replication w/o any problem, However, we had to adapt directory metadata related to account-based resource limits settings:

Some DSEE entries contain the following resource limit attributes, namely nsSizeLimit,nsTimeLimit, nsLookThroughLimit, nsIdleTimeout. Corresponding attributes on OUD are ds-rlim-size-limit, ds-rlim-time-limit,ds-rlim-lookthrough-limit,ds-rlim-idle-time-limit. In order to replicate the functionality correctly, the OUD schema was modified so that each DSEE attribute name related to resource limits is declared as an alias name for each corresponding OUD attribute.

2.3 Validating OUD Configuration and Settings

A few OUD servers were deployed in staging area and configured as defined above. Then traffic from key customer applications identified during the transition assessment was redirected to the OUD infrastructure.

This phase is very important to validate the changes above and identify behavioral differences between OUD and DSEE that are not always detected automatically. We detected a few problems that required additional configuration changes on the OUD side. These changes were added to the list of configuration to be applied to OUD servers.

Here are a few road blocks hit and how we addressed them:

- Few applications were performing unindexed operations. By default, OUD reject such searches.
     For technical reasons, it was not acceptable to create the appropriate indexes to suppress unindexed searches: For this project, we recommend to disable privileges leading to aci behavioral differences between OUD and ODSEE.
- Attributes present in the rootDSE and in the schema are flagged as operational, so they are not returned to client applications unless they are explicitly specified in the search attribute list. On ODSEE, these attributes are systematically returned.        
    Client applications relied on the DSEE behavior; we had to modify the OUD configuration so that rootDSE entries are returned like user attributes
- By default, in OUD, unauthenticated users are not granted access to cn=schema nor the rootDSE.
    Appropriate access controls was added to OUD

2.4 Deploy additional instances in a replicated topology

- 4 instances having both Directory server and Replication Server roles are deployed, 2 in each data center.
- The number of directory server instances serving search operations could be reduced to 4 due to superior OUD performances. 
- 2 replication groups are defined so that DSs in one data center preferably connect to a RS within the same data center.
- 2 directory servers deployed in remote branches are configured as read-only replicas to conform to corporate rules. These 2 servers can connect to replication server from either data center to receive updates.

Note: Unlike DSEE topology, every directory server running in the main data centers are read-write master. The corresponding servers in DSEE handled a limited write traffic that was redirected to DSEE masters via referrals. The new OUD topology eliminates the need for referrals.

Figure 2: New OUD topology with 4 RS+DS, 4 DS in each DC + 2 read-only DS in remote branches

3 Deploying the OUD topology

The main outcome of the transition analysis and design phase is a collection of commands to be applied to set up an OUD directory server instance.
Additional OUD directory server instances were setup then configured. The Test to Production feature provided by OUD is used to clone configurations to pre-production environment.

Data are exported from DSEE (with the --opends flag) to preserve replication metadata, so that replication can be established between the 2 environments. Data are imported in a single OUD directory server, then replication was enabled between servers and database files are copied to the other servers. In the customer environment, this initialization strategy was preferred over an over-the-network full initialization.

The minimum version required for tightly coupled coexistence is ODSEE 11g Release 1 (11.1.1) for the ODSEE master that communicates directly with the replication gateway. However, the rest of the ODSEE topology does not need to be uniformly based on this version and remain in 6.x, so we upgraded 2 DSEE masters to the latest ODSEE 11gR1 PS2 (11.1.1.7.0). Instances were automatically upgraded in place without having to copy, export or import anything. An alternate solution would have been to deploy 2 new ODSEE 11g instances as replication gateway companion.

At that stage, 2 replication gateways are deployed as described in the OUD administration guide. This is the recommended setting to avoid single point of failure.

Backup strategy was adapted to reflect the new hybrid topology: In a replicated environment involving ODSEE and OUD, you must perform regular backups on the ODSEE side and on the OUD side. A backup must always be restored in the topology it is associated with.

Figure 3: Cohabitation DSEE/OUD

The current plan is to keep the DSEE-OUD cohabitation for 6 months as applications are progressively redirected to OUD.

4 Conclusion

Tightly coupled coexistence of OUD with ODSEE is achieved by deploying OUD and ODSEE in a replicated topology using the “Replication Gateway”. The replication gateway provides out-of-the-box live transition without service interruption.
This enables you to run OUD and DSEE in parallel in a mixed environment so that you can transition to OUD over time, validate your upgrade strategy application by application, and most importantly, without downtime.

Migration tools shipped with OUD addresses most of the transitioning issues. However, data cleaning and/or manual adaptation is sometimes required during this process, so some time should be allocated to address that during the transition analysis and design.