Insyteful

The Nokia N900 is a great device for *nix sytems admins. It features, among other things, a native port of both xterm and OpenSSH. Being a good sysadmin, the first thing I did was generate a new (encrypted) SSH key. The next was discover that ssh-agent was not managed by the OS so that all processes have access to the $SSH_AUTH_SOCK variable. As I was not feeling quite adventurous enough to hack up the somewhat non-standard boot process / X startup, I needed to find a different way.

With a little trial and error I was able to come up with a brief snippet of code to include in ~user/.profile to ensure that I always have access to ssh-agent and can share that agent between numerous xterm sessions.

This is what I did:

# cat ~user/.profile

unset SSH_AUTH_SOCK
unset SSH_AGENT_PID

if [[ -f ~/.ssh-agent ]]
then
        source ~/.ssh-agent >> /dev/null
fi

ssh-add -l 2>&1 > /dev/null
r=$?
if [[ $r == 2 ]]
then
        echo No agent.
        pkill ssh-agent
        ssh-agent -t 5400 > ~/.ssh-agent
        source ~/.ssh-agent > /dev/null
fi

What does it do?

First I check to see if ~/.ssh-agent exists. If it does, it should contain the details for the last ssh-agent instance launched. By sourcing that file, the appropriate environment variables are set allowing ssh and ssh-add to find the (possibly) running agent.

Next, I attempt to execute ssh-add -l, which if an agent is running and has a key loaded, would list the loaded keys. If ssh-add -l exited with 2, that means either there isn't an ssh-agent running or that we can't find it. In either case, next I attempt to kill off any running instances of ssh-agent and launch a new one, caching the details in ~/.ssh-agent.

Note the -t 5400 option on ssh-agent. That dumps any loaded keys after 90 minutes. I don't want to leave unencrypted ssh keys lying around in such an easily stolen package...

More fun and games building stuff on OS X.

I like Snow Leopard. Really, I do. However there is pain involved in getting ipython to work properly. After the upgrade (on a fresh install of Leopard) I used the system python to install ipython:

# easy_install ipython

Initially everything looked good. Aside from the rather ominous "Leopard libedit detected." message on startup. Unfortunately once I started using it, problems quickly became apparent:

• Cursor not returning to the correct location on the line when using up/down arrow to browse history.
• No ctrl-r to search through history.
• Other random weirdness.

Fortunately Patrick van Kouteren wrote an excellent article that got me headed in the right direction. Unfortunately, building the ipython-packaged readline failed miserably during the initial configure process. As always, config.log tells the tale:

configure:1883: gcc -isysroot /Developer/SDKs/MacOSX10.5.sdk 
  -arch i386 -arch ppc -arch x86_64 -arch ppc64 -DNEED_EXTERN_PC
  -isysroot /Developer/SDKs/MacOSX10.5.sdk -arch i386 -arch ppc
  -arch x86_64 -arch ppc64 conftest.c  >&5
ld: library not found for -lcrt1.10.6.o
collect2: ld returned 1 exit status
ld: library not found for -lcrt1.10.6.o
collect2: ld returned 1 exit status

Fortunately for my sanity, it looks like distutils no longer tries to force building of Universal binaries on the Mac but still this looked awfully similar.

Solution:

1. Edit setup.py and comment out these lines:

   if distutils.util.get_platform().find('macosx-10.5') == 0:
       UNIVERSAL = '-isysroot /Developer/SDKs/MacOSX10.5.sdk ' + \
                   '-arch i386 -arch ppc -arch x86_64 -arch ppc64'
   
   

2. Edit rl/build.sh and comment out these lines:

   export CFLAGS='-isysroot /Developer/SDKs/MacOSX10.5.sdk 
     -arch i386 -arch ppc -arch x86_64 -arch ppc64'
   export LDFLAGS='-isysroot /Developer/SDKs/MacOSX10.5.sdk
     -arch i386 -arch ppc -arch x86_64 -arch ppc64'
   
   

I came across two fascinating articles today about why SATA RAID can be scary and dangerous:

• SATA vs. SCSI reliability
• Are Fibre Channel and SCSI Drives More Reliable?

Basically, drive size has increased dramatically. The "bit error rate"[1] has remained essentially constant. As you approach something around 12TB, it's virtually certain that you have unreadable blocks on your array.

Who cares if a couple of bytes in the middle of some old archived log files are munged? I sure as hell don't. Unfortunately, most failed blocks are only discovered during the one process that requires reading and checksumming every block of data on the array: Rebuilding the array after a drive failure. What happens then? Depends on the RAID controller. Apparently it's pretty common for it to give up and leave you with no useable data at all.

It wasn't until I read this article that I understood why the über-smart folks at Coraid implemented a technology called RAIDShield that basically continuously reads every sector of every disk in the background. If a bad block is found, the correct data is calculated from the parity data and written out to the "bad" sector. This triggers the drives internal sector remapping, guaranteeing that the data is written to a good portion of the disk and flagging the sector as unusable.

I really like knowing our vendors have solved problems we didn't even know we had. Makes me feel all warm and fuzzy.

[1] The ratio of the number of unreadable blocks to the total number of blocks on the drive/array.

If forced to use an MS-SQL database, I find it much easier and more convenient to use the excellent sqsh shell. On our Ubuntu boxes, installing it was as simple as apt-get install sqsh. Unfortunately, DarwinPorts doesn't include it. Building from source was somewhat problematic due to OS X's none-GNU-readline readline.h (it's actually part of the BSD editline library).

Here's what I needed to do:

1. Install FreeTDS:

   port install freetds
   
   

2. Download and untar sqsh.

3. Configure as you would normally, setting $SYBASE to point to the top of the FreeTDS install:

   SYBASE=/opt/local ./configure --with-readline
   
   

4. Edit src/Makefile and append freetds to line #18 so it looks like this:

   SYBASE_INCDIR = -I$(SYBASE_OCOS)/include/freetds
   
   

5. Modify the header/library paths for libreadline to point to the version from DarwinPorts:

   READLINE_LIBDIR =  -L/opt/local/lib
   READLINE_INCDIR =  -I/opt/local/include/
   
   

6. make and make install

I've moved away from using the Python from MacPorts (DarwinPorts) and am, instead, using the Official Mac Python build from python.org. Overall, it seems to have been a good move. However, the first time I tried to build a C module using distutils, ld popped up this error when trying to link against libcrypto.dylib:

cputype (18, architecture ppc) does not match cputype (7) for specified -arch flag: i386 (file not loaded)

Scrolling back to look at the actual gcc command line revealed a clue. Present among the flags were: -arch i386 -arch ppc.

Another clue:

tarja:~ ben$ file /opt/local/lib/libcrypto.0.9.8.dylib /opt/local/lib/libcrypto.0.9.8.dylib: Mach-O dynamically linked shared library ppc

So Python's distutils was determined to build universal binaries, but the OpenSSL install created by MacPorts was PPC only. That's not going to work.

A grep through the source for distutils didn't reveal any instances of "-arch i386", so I resorted to Google. This post provided the necessary clue. After editing /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/config/Makefile and removing all references to -arch i386, the linking succeeded.

Almost.

Then I started seeing errors similar to this one: /usr/bin/ld: warning can't open dynamic library: /Developer/SDKs/MacOSX10.4u.sdk/opt/local/lib/lib3rdparty.dylib referenced from: libMyLibrary.dylib (checking for undefined symbols may be affected) (No such file or directory, errno = 2)

Note that in this example, lib3rdparty.dylib actually does exist in /opt/local/lib. As I eventually learned from this thread, using the -isysroot option prepends the SDK directory to all paths. As you'll note from the above-linked thread, whether or not this is a bug depends on who you ask. The kludgy workaround:

Make a symlink under /Developer/SDKs/MacOSX10.4u.sdk/ to /opt/local/lib.

Ugly.

But it works.

Thanks, Apple.

I've been a fan of Max Barry since I read Jennifer Government a few years back, although I think I liked Syrup better. So when I read that Company was coming out soon, I pre-ordered it from Amazon.

Capsule review:

Fascinating and highly entertaining corporate satire featuring rambling, multiple-forwarded voicemail, a sociopathic but brilliant receptionist, meaningless mission statements, and a donut.

Read this. Then read Syrup and Jennifer Government.

Extract from the first chapter of Company: READ

I've been doing some work on the NetEqualizer and needed to find a quick-n-dirty way to generate a ton of network traffic between two linux devices. Netcat to the rescue!

On the client side:

yes BBBBBBBBBBBBBBBBBBBBBB | nc 10.0.8.2 2223 > /dev/null

And on the server:

yes AAAAAAAAAAAAAAAAAAAAAA | nc -v -v -l -p 2223 > /dev/null

Is there anything netcat can't do?

I've discovered that I am now collecting chef's knives.

When I bought my first good knife, it was easily defensible: every cook needs a high-quality chef's knife. When my wife bought me a gorgeous santoku, I would tell people that I prefered it for working with vegetables. Then I bought the new Ken Onion blade. I can't defend three knives.

So I've accepted that I am a collector. Which means that I can now justify buying any other cool knives that catch my eye!

A few months back we moved our entire campus to using network authentication with OpenLDAP for the backend. The first step was to use it as the backend for the mail server. That worked fine. Then we turned it loose on the computer labs. This initially appeared to work fine, but occasionally a computer would take several minutes to successfully log in. Once logged in, computers would randomly exhibit extremely slow AFP access. This is the story of troubleshooting that problem.

The initial computer used for network authentication was, in retrospect, underpowered. In an attempt to compensate for the perceived configuration problems, I made a number of tweaks, including some timeout settings.

Over the weekend, I built a new server to be the directory server, hoping it would do better. It appeared to handle the queries without any noticeable load or slowdown, but was still dropping a large number of authentication attempts. Matt and I began an analysis of the actual network traffic, looking at the individual network packets. We noticed an anomaly in the way LDAP sessions were being closed down after a period of inactivity. The LDAP server was sending a "FIN,ACK" packet, the normal way to request the shutdown of a TCP session. The client responded, appropriately, with an "ACK" packet.

This effectively closed the connection from the server to the client, but not the connection in the other direction. The way the client should proceed from here is to respond with its own "FIN,ACK" packet, to which the server would've responded "ACK" and the session would be cleanly shut down.

Normal shutdown of a session:

 server                client

   (normal data exchange)
  <------- SEARCH ----------
  -------- RESULT --------->
    (starting shutdown)
  ---------FIN ACK--------->
  <----------ACK------------
     (halfway closed)
  <--------FIN ACK----------
  -----------ACK----------->
    (completely closed)

Instead, the client would, frequently, attempt to send a new search request through the half-closed connection. The server, righteously and appropriately indignant, responded with a "RST" (or "reset") packet, signalling that it was done and wasn't interested in hearing any more. The client, rebuffed and confused, would deem the server unreachable and go into a funk for 30 seconds or so (depending on its config).

Broken shutdown of a session:

server                client

    [normal data exchange]
  <------- SEARCH ----------
  -------- RESULT --------->
  [server starting shutdown]
  ---------FIN ACK--------->
  <----------ACK------------
      [halfway closed]
  <------- SEARCH ----------
  ---------- RST ---------->
     [client panics and
      stops talking]

Once we understood the problem, the solution was clear: Make sure the client always attempted to shut down idle sessions first, so its buggy half-closed SEARCH requests would not have opportunity to occur. So we upped the idle timeout on the LDAP server high enough to guarantee it would only come into play if the client machine was completely hung already. As soon as we made this change, all the logins became speedy, sessions stopped hanging, and all was right with the world.

I have now eliminated the original (slow) LDAP server completely, shifting all the load to the new ultra-beefy server. The next step is to upgrade the old server to a bit more modern standards and rebuild it to serve as a backup.