The Nokia N900is a great device for *nix sytems admins. It features, among other things, a native port of both xterm and OpenSSH. Being a good sysadmin, the first thing I did was generate a new (encrypted) SSH key. The next was discover that ssh-agentwas not managed by the OS so that all processes have access to the $SSH_AUTH_SOCKvariable. As I was not feeling quite adventurous enough to hack up the somewhat non-standard boot process / X startup, I needed to find a different way.

With a little trial and error I was able to come up with a brief snippet of code to include in ~user/.profileto ensure that I always have access to ssh-agentand can share that agent between numerous xterm sessions.

This is what I did:

# cat ~user/.profile

unset SSH_AUTH_SOCK
unset SSH_AGENT_PID

if [[ -f ~/.ssh-agent ]]
then
        source ~/.ssh-agent >> /dev/null
fi

ssh-add -l 2>&1 > /dev/null
r=$?
if [[ $r == 2 ]]
then
        echo No agent.
        pkill ssh-agent
        ssh-agent -t 5400 > ~/.ssh-agent
        source ~/.ssh-agent > /dev/null
fi

What does it do?

First I check to see if ~/.ssh-agentexists. If it does, it should contain the details for the last ssh-agentinstance launched. By sourcing that file, the appropriate environment variables are set allowing sshand ssh-addto find the (possibly) running agent.

Next, I attempt to execute ssh-add -l, which if an agent is running and has a key loaded, would list the loaded keys. If ssh-add -lexited with 2, that means either there isn't an ssh-agentrunning or that we can't find it. In either case, next I attempt to kill off any running instances of ssh-agentand launch a new one, caching the details in ~/.ssh-agent.

Note the -t 5400option on ssh-agent. That dumps any loaded keys after 90 minutes. I don't want to leave unencrypted ssh keys lying around in such an easily stolen package...

I came across two fascinating articles today about why SATA RAID can be scary and dangerous:

• SATA vs. SCSI reliability
• Are Fibre Channel and SCSI Drives More Reliable?

Basically, drive size has increased dramatically. The "bit error rate"[1] has remained essentially constant. As you approach something around 12TB, it's virtually certain that you have unreadable blocks on your array.

Who cares if a couple of bytes in the middle of some old archived log files are munged? I sure as hell don't. Unfortunately, most failed blocks are only discovered during the one process that requires reading and checksumming every block of data on the array: Rebuilding the array after a drive failure. What happens then? Depends on the RAID controller. Apparently it's pretty common for it to give up and leave you with no useable data at all.

It wasn't until I read this article that I understood why the über-smart folks at Coraidimplemented a technology called RAIDShield that basically continuously reads every sector of every disk in the background. If a bad block is found, the correct data is calculated from the parity data and written out to the "bad" sector. This triggers the drives internal sector remapping, guaranteeing that the data is written to a good portion of the disk and flagging the sector as unusable.

I really like knowing our vendors have solved problems we didn't even know we had. Makes me feel all warm and fuzzy.

[1] The ratio of the number of unreadable blocks to the total number of blocks on the drive/array.

I've been a fan of Max Barrysince I read Jennifer Governmenta few years back, although I think I liked Syrupbetter. So when I read that Companywas coming out soon, I pre-ordered it from Amazon.

Capsule review:

Fascinating and highly entertaining corporate satire featuring rambling, multiple-forwarded voicemail, a sociopathic but brilliant receptionist, meaningless mission statements, and a donut.

Read this. Then read Syrupand Jennifer Government.

Extract from the first chapter of Company: READ

I've been doing some work on the NetEqualizerand needed to find a quick-n-dirty way to generate a ton of network traffic between two linux devices. Netcatto the rescue!

On the client side:

yes BBBBBBBBBBBBBBBBBBBBBB | nc 10.0.8.2 2223 > /dev/null

And on the server:

yes AAAAAAAAAAAAAAAAAAAAAA | nc -v -v -l -p 2223 > /dev/null

Is there anything netcat can't do?

I've discovered that I am now collecting chef's knives.

When I bought my first good knife, it was easily defensible: every cook needs a high-quality chef's knife. When my wife bought me a gorgeous santoku, I would tell people that I prefered it for working with vegetables. Then I bought the new Ken Onionblade. I can't defend three knives.

So I've accepted that I am a collector. Which means that I can now justify buying any other cool knives that catch my eye!

A few months back we moved our entire campus to using network authentication with OpenLDAP for the backend. The first step was to use it as the backend for the mail server. That worked fine. Then we turned it loose on the computer labs. This initially appearedto work fine, but occasionally a computer would take several minutes to successfully log in. Once logged in, computers would randomly exhibit extremely slow AFP access. This is the story of troubleshooting that problem.

The initial computer used for network authentication was, in retrospect, underpowered. In an attempt to compensate for the perceived configuration problems, I made a number of tweaks, including some timeout settings.

Over the weekend, I built a new server to be the directory server, hoping it would do better. It appeared to handle the queries without any noticeable load or slowdown, but was still dropping a large number of authentication attempts. Matt and I began an analysis of the actual network traffic, looking at the individual network packets. We noticed an anomaly in the way LDAP sessions were being closed down after a period of inactivity. The LDAP server was sending a "FIN,ACK" packet, the normal way to request the shutdown of a TCP session. The client responded, appropriately, with an "ACK" packet.

This effectively closed the connection from the server to the client, but not the connection in the other direction. The way the client shouldproceed from here is to respond with its own "FIN,ACK" packet, to which the server would've responded "ACK" and the session would be cleanly shut down.

Normal shutdown of a session:

 server                client

   (normal data exchange)
  <------- SEARCH ----------
  -------- RESULT --------->
    (starting shutdown)
  ---------FIN ACK--------->
  <----------ACK------------
     (halfway closed)
  <--------FIN ACK----------
  -----------ACK----------->
    (completely closed)

Instead, the client would, frequently, attempt to send a new search request through the half-closed connection. The server, righteously and appropriately indignant, responded with a "RST" (or "reset") packet, signalling that it was done and wasn't interested in hearing any more. The client, rebuffed and confused, would deem the server unreachable and go into a funk for 30 seconds or so (depending on its config).

Broken shutdown of a session:

server                client

    [normal data exchange]
  <------- SEARCH ----------
  -------- RESULT --------->
  [server starting shutdown]
  ---------FIN ACK--------->
  <----------ACK------------
      [halfway closed]
  <------- SEARCH ----------
  ---------- RST ---------->
     [client panics and
      stops talking]

Once we understood the problem, the solution was clear: Make sure the client always attempted to shut down idle sessions first, so its buggy half-closed SEARCH requests would not have opportunity to occur. So we upped the idle timeout on the LDAP server high enough to guarantee it would only come into play if the client machine was completely hung already. As soon as we made this change, all the logins became speedy, sessions stopped hanging, and all was right with the world.

I have now eliminated the original (slow) LDAP server completely, shifting all the load to the new ultra-beefy server. The next step is to upgrade the old server to a bit more modern standards and rebuild it to serve as a backup.