Our old LDAP and it's problems

Okay, so I'm also a very irregular blogger. :-) Hopefully I can get this series done, though. I've vaguely promised to give a presentation on this, and should the askers ever get their act together and schedule me, it'd be nice to have it ready.

I promised to talk about several things this time, our old LDAP, a more detailed explanation of what we were trying to do, and why we were trying to do it, and the process we used to select our software. I probably won't finish them all in this post, but let's see how far I get.

Our old LDAP

Our old LDAP service consisted (actually still consists of) three Redhat Enterprise linux servers running OpenLDAP. It's still in place in an upgraded form as I work to migrate us from this old infrastructure to the new one. This is basically a litany of the issues we had with the old service

Replication issues

First, replication. OpenLDAP, at least when this was installed pre my tenure, openldap had no mechanism for multi-master replication schemes. Nowadays there's syncrepl, but not then. This means that our old setup used a single master, and is vulnerable to missing important changes (e.g. password updates) if the master is down.

Custom Schema

Next up, poorly thought out Custom Schema. This LDAP used an additional attribute, userClass, to restrict account visibility to server machines via way of a nss service search descriptor. The method in general turns out to be about the best way to do this (more on this in later posts), but the problem was the uglyness used in shimming this attribute into a posixAccount objectclass object.

An early version of this shim overrode and extended the posixAccount objectclass via a custom schema. A later version (still in place) created a new object class 'ourAccount' as a SUP of pilotPerson, inetOrgPerson, and account , and declared all account objects as 'objectclass ourAccount' and 'objectclass posixAccount' (and a few others).

Both solutions are a bit ugly. First, 'userClass' might not be the best symantic match for the function it was used for - membership in one or more 'groups'. In particular, there's no easy, single stop way to see a list of all userClass's, and there's advantages to having both a reverse and forward map (e.g. posixgroup/groupOfNames *and* memberOf attributes), and userClass has no mechanism for this. Next, the schema hacks above were a bit ugly and worse, not documented anywhere.

My current one, I admit, still uses a schema hack, but it does allow all the critical operations easily: what's a list of all the 'groups' of users, give me a list of everyone in a group, which groups is this user in, and is this user in this group. Further, it's documented. :-)

Management tool(s)

The set of management tools in place for this service were similarily hacked together, not well supported, and undocumented. There were two scripts in an obscure location to reset passwords and create new accounts. The create new account script also attempted to create an email account on an IMAP server that never really existed. There was also X11 LDAP management GUI on a seperate server which didn't manage any of the custom schema elements, nor, because of the custom schema, could be used to create any new objects. Finally there was a half broken perl script intended to be placed on ldap client hosts to create newly provisioned user's home directories.

Another serious issue is that these tools had no provision for doing proper UID / GID allocation for new accounts. In fact, we had several accounts that had the same UID's and groups that had the same GID's. Also there was no provision for automating user addition and, importantly user deactivation and retirement.

There were also a few outright bugs. The worst was the way password updates were (or weren't) managed. The system depended on each client's 'passwd' command being properly configured via pam_ldap such that it would send password updates to the LDAP replica, get a referral to the master, follow that to the master, bind as the password changing user, and update the userPassword attribute. In addition to the obvious possibility for breakage with any of that configuration, there were two other serious problems.

First, not all systems properly followed the referal, and thus might only update the password in the local replica, or hopefully fail to do so so at least the user got an error instead of an inconsistant password depending on replica.

Second, none of the implementations of pam_ldap bothered to update the 'shadowLastChanged' attribute. This meant that if the user was changing their password due to an expired password, which forced a password change, then no matter how often they updated their password it would always remain expired. This caused no end of oncall support issues for us.

All right, I'm about blogged out. More later on our old LDAP service and other stuff, too, like some of the things we did to make our old LDAP more usable, and our attempts to make it more widespread than just our linux boxes.

-- Pat

Late blooming blogger - beginning of our LDAP saga

I don't know if anyone else is like this, but I, at least, don't follow blogs. I search them. That is, when I'm interested in a specific topic, I'll google it. Often, I find an interesting blog post or two which I'll read, but I never bother to go back and scan those blogs again.

Ergo, I'd never bothered to blog before largely because I'd not anything to say worth anyone's time. I'd always thought that, to be worth reading, I should have something informative to say.

Well, I do.

Specifically, I thought it'd be helpful if I were to document a bunch of what I've gone through to create our heterogeneous unix LDAP environment at work. There was a pretty fair amount of patching stuff together in ways that weren't documented anywhere. So, I'm going to start a series of blog posts, titled, roughly:

LDAP in a heterogeneous environment - What they don't tell you.

First up, some background on our situation.

I work as a unix admin for a medium sized company - we have about 600 unix servers split roughtly in thirds between AIX, Linux, and Solaris, with a few oddball HP-UX and VMS boxes. There's also about 3000 Windows servers, some Tandem non-stop, a pair of IBM mainframes, some IBM eSeries, a few racks of OS-X high performance compute cluster boxes, and lots of other odds and drabbles I'm only periferally involved in, if at all.

When I transfered to the Unix admin spot from a DBA team about 4 years ago, we had a partial LDAP infrastructure, consisting of a OpenLDAP master server and two slaves. However this was only used for our Linux boxes, so it had limited penetration, and was in other ways a bit of a mess:

• Uids and gids were pretty much all over the map, including some duplications;
  
• There was no automation of provisioning, users had to be manually created in LDAP, and home directories had to be manually created for users wherever they had a login;
  
• Worse, there was not automation of account de-provisioning for when people transfered or left
  
• Password changes were haphazard, didn't always work, and weren't synchronized with our institutional password management infrastructure
• The schema loaded on the server just plain didn't work with most AIX and Solaris clients.

In short, I could make a few patches to limp this system along, but in the end I was looking at a complete replacement. To start with, I came up with the following set of goals and requirements:

1. Central Authentication and Authorization database/repository


1. All client platforms & os's can use (AIX, Solaris, Linux, HP-UX)
2. Highly available
3. Meets institution security and access policies
4. Uses same username, passwords, and unix specific identifiers (uids, gids) across all systems


2. Repository information is replicated from core sources, including:


1. Username will be the historical Unix and Mainframe format, not Active Directory
2. Real name & information (office, phone#...)
3. Password
4. Account disable and account removal


3. Automation, application access and authorization functions offloaded from unix team


4. Meets auditing requirements for user management


1. Determine for each account what level of access and authorization it has across all systems
2. Maintain access history for each account
• security changes
• access level history


5. Only those accounts valid for the application(s) hosted on a specific machine are presented on that machine.


6. Client machines can be configured without reference to user accounts contained in the repository. E.g. if account 'foo' is to be used on machine 'bar', no configuration file on 'bar' must have a specific entry for 'foo'.


7. Secure on the wire network transactions between client machines and repository server

In later posts, I'll give more detailed information about our old LDAP setup, explain these goals a bit more and talk about the process we used to design and select our new architecture.

Bye for now.
-- Pat