Insyteful

I've discovered that I am now collecting chef's knives.

When I bought my first good knife, it was easily defensible: every cook needs a high-quality chef's knife. When my wife bought me a gorgeous santoku, I would tell people that I prefered it for working with vegetables. Then I bought the new Ken Onion blade. I can't defend three knives.

So I've accepted that I am a collector. Which means that I can now justify buying any other cool knives that catch my eye!

A few months back we moved our entire campus to using network authentication with OpenLDAP for the backend. The first step was to use it as the backend for the mail server. That worked fine. Then we turned it loose on the computer labs. This initially appeared to work fine, but occasionally a computer would take several minutes to successfully log in. Once logged in, computers would randomly exhibit extremely slow AFP access. This is the story of troubleshooting that problem.

The initial computer used for network authentication was, in retrospect, underpowered. In an attempt to compensate for the perceived configuration problems, I made a number of tweaks, including some timeout settings.

Over the weekend, I built a new server to be the directory server, hoping it would do better. It appeared to handle the queries without any noticeable load or slowdown, but was still dropping a large number of authentication attempts. Matt and I began an analysis of the actual network traffic, looking at the individual network packets. We noticed an anomaly in the way LDAP sessions were being closed down after a period of inactivity. The LDAP server was sending a "FIN,ACK" packet, the normal way to request the shutdown of a TCP session. The client responded, appropriately, with an "ACK" packet.

This effectively closed the connection from the server to the client, but not the connection in the other direction. The way the client should proceed from here is to respond with its own "FIN,ACK" packet, to which the server would've responded "ACK" and the session would be cleanly shut down.

Normal shutdown of a session:

 server                client

   (normal data exchange)
  <------- SEARCH ----------
  -------- RESULT --------->
    (starting shutdown)
  ---------FIN ACK--------->
  <----------ACK------------
     (halfway closed)
  <--------FIN ACK----------
  -----------ACK----------->
    (completely closed)

Instead, the client would, frequently, attempt to send a new search request through the half-closed connection. The server, righteously and appropriately indignant, responded with a "RST" (or "reset") packet, signalling that it was done and wasn't interested in hearing any more. The client, rebuffed and confused, would deem the server unreachable and go into a funk for 30 seconds or so (depending on its config).

Broken shutdown of a session:

server                client

    [normal data exchange]
  <------- SEARCH ----------
  -------- RESULT --------->
  [server starting shutdown]
  ---------FIN ACK--------->
  <----------ACK------------
      [halfway closed]
  <------- SEARCH ----------
  ---------- RST ---------->
     [client panics and
      stops talking]

Once we understood the problem, the solution was clear: Make sure the client always attempted to shut down idle sessions first, so its buggy half-closed SEARCH requests would not have opportunity to occur. So we upped the idle timeout on the LDAP server high enough to guarantee it would only come into play if the client machine was completely hung already. As soon as we made this change, all the logins became speedy, sessions stopped hanging, and all was right with the world.

I have now eliminated the original (slow) LDAP server completely, shifting all the load to the new ultra-beefy server. The next step is to upgrade the old server to a bit more modern standards and rebuild it to serve as a backup.