Continuing our old LDAP

So I'm realizing as I'm writing these posts that this is really not quite the straightforward whitepaper style I'd originally envisioned, but is instead somewhat blended with chronological writing. As it turns out, a lot of what I'm talking about is stuff I worked on in rough chronological order.

Anyway, on to the next topic.

Updates to our old OpenLDAP service

So, obviously, my first choice was not to throw out the existing openldap servers wholesale. Instead, I first implemented a number of improvements to the service.

The obvious first improvment is to just do a software update. I upgraded the Redhat Enterprise Linux release from rev 3 to rev 4 on new hardware, and along with that, updated openldap from 2.1 to 2.2. This had no massive impact in and of itself, but proved a great learning experience for me. I had to really work to grok some of the basic LDAP concepts as well as the funky schema mods.

More importantly, now I was ready to start making more substantive changes and fixes.

New management tools

One of the first things I addressed were the management tools and internal procedures and documents (or lack thereof). I started by putting in an installation of phpLDAPAdmin 0.96, and making a few customizations for our locally cracked schema. Fortunately, PhpLDAPAdmin makes it easy to modify it's scripts to assign different userclasses to new accounts and groups, so it was fairly trivial to use our customized objectclasses.

Procedurally, though, I still had to establish a consistant procedure for most of our common issues: mostly creating new accounts and groups and provisioning users to machines. Specifically, new UID allocation was still an issue

Account Creation and UID/GID allocation

I began by cleaning up any UID or GID conflicts I could find. This was a tedious process that involved first some scripting to even pull out the conflicts, then finding any machines those conflicting accounts were on, changing one of the conflicts to a new number, and finally changing ownership on any files. There were less than a dozen instances of this problem, but the legwork still took several weeks. Now, I could hopefully proceed forward.

UID's in this system were still somewhat a mess - most of the accounts had an account specific group, but these were rarely the same GID as the UID. Also, this was not universal, many of the accounts had a shared primary GID. Having a purpose specific GID (e.g. staff, or webspher) works well if that person only ever does one thing, but as soon as they use or develop multiple applications the model falls apart.

Given how much time it took me to fix the UID conflicts of just a dozen or so instances, I was terrified of ever attempting to fix the apx 500 accounts in our old LDAP. There were other messes as well, for instance, often a person would have multiple accounts.

I decided to let that problem sit for now, and concentrate on the procedure for creating new accounts. I started by stating a policy of creating an account specific group for each account, and insuring they woulld have the same names and the same UID and GID numbers. I allocated new UID/GID ranges for accounts owned by real people, accounts used for applications (e.g. websphere, oracle) and non-account specific groups.

I investigated whether phpLDAPAdmin could be easily made to do these things for me as part of a single unified operation, and pretty much determined that it wouldn't be worth the time to make the attempt. Even attempting to just use phpLDAPAdmin's id number allocations would mess things up with several different id number ranges.

Ergo, manual procedures were the answer, at least for now. I made a script to pluck the next available uid and gid number pair from the appropriate range, and wrote a procedure instructing how to use that script combined with two or three different phpLDAPAdmin operations to create new accounts.

Home directory provisioning

Another improvement was to replace the old user home directory creation method. The original method was a perl script which directly used Net::LDAP to query the ldap servers and see what accounts needed home directories. It was scheduled to run nightly via cron. It had several limitations - it needed the Net::LDAP module and all its dependancies installed, and it needed to be configured separately for each host, and also separately from configuring the service search descriptors in /etc/ldap.conf. This opened opportunity for configuration mismatches. It also, of course, also doesn't cover the case of what to do when a user is deprovisioned.

One of the first things I considered was attempting to use pam_mkhomedir. That's a pam module which, if you log into a configured host where you do not have a home directory, it will attempt to create one for you. It's a decent partial solution. It doesn't cover any case where a home directory may be expected but the user hasn't logged in yet, such as mail delivery. It also still doesn't cover deprovisioning. Both of those could be lived with, but the kicker was that it wasn't universally available on all our target platforms. Only the various linuxes supported it, and only recent installations of those.

I ended up writing a much simpler script based on the operating system 'list the user registry' tool, 'getent' in linux and solaris, and 'lsuser' in aix, parsing the output, and creating home dirs if necessary. Like the original script, it was run from cron. This script resolves the perl dependancy issue, and effectivly shares the OS's service search descriptor configuration. It still shares the only run once daily and no deprovisioning limitations, though.

Password management

The last major change I made was to hook this LDAP up to the institution wide password sync service. To prevent passwords getting out of sync as well as some of the password changes problems I mentioned my earlier posts, I set an ACL such that users couldn't change their own passwords. This prevented people from using either the local system's 'passwd' command, or even 'ldappasswd' if they were really savvy. Instead, we began referring everyone to the our institutions central password change application.

This actually was a huge thing for us, and by itself significantly reduced our LDAP related oncall issues.

Going forward

After reaching this point in my improvements, though I stalled. There were still a large number of problems with this system that prevented expanding this to serve our Solaris and AIX hosts, or even continuing to expand it to serve other linux hosts.

• The first and biggest issue is that I wasn't able to get our solaris or AIX to work with this LDAP at all.
• Also, I was never able to update to more recent versions of phpLDAPAdmin. For whatever reason, I never got an upgrade to connect to the old LDAP.
• This was needed because the existing phpLDAPAdmin had some serious performance issues
• The existing UID's and GID's were still a mess.
• I faced a difficult problem of being forced to do future UID leveling - No easy way out of UID conflicts between the existing LDAP and those same UIDs used for local accounts on non-LDAP machines we wanted to bring in.
• Finally, a multi-master replication would be sweet

So, we still had quite a ways to go. More on this in future posts.

-- Pat