With a little trial and error I was able to come up with a brief snippet of code to include in ~user/.profile to ensure that I always have access to ssh-agent and can share that agent between numerous xterm sessions.

This is what I did:

# cat ~user/.profile

unset SSH_AUTH_SOCK
unset SSH_AGENT_PID

if [[ -f ~/.ssh-agent ]]
then
        source ~/.ssh-agent >> /dev/null
fi

ssh-add -l 2>&1 > /dev/null
r=$?
if [[ $r == 2 ]]
then
        echo No agent.
        pkill ssh-agent
        ssh-agent -t 5400 > ~/.ssh-agent
        source ~/.ssh-agent > /dev/null
fi

What does it do?

First I check to see if ~/.ssh-agent exists. If it does, it should contain the details for the last ssh-agent instance launched. By sourcing that file, the appropriate environment variables are set allowing ssh and ssh-add to find the (possibly) running agent.

Next, I attempt to execute ssh-add -l, which if an agent is running and has a key loaded, would list the loaded keys. If ssh-add -l exited with 2, that means either there isn't an ssh-agent running or that we can't find it. In either case, next I attempt to kill off any running instances of ssh-agent and launch a new one, caching the details in ~/.ssh-agent.

Note the -t 5400 option on ssh-agent. That dumps any loaded keys after 90 minutes. I don't want to leave unencrypted ssh keys lying around in such an easily stolen package...

I like Snow Leopard. Really, I do. However there is pain involved in getting ipython to work properly. After the upgrade (on a fresh install of Leopard) I used the system python to install ipython:

# easy_install ipython

Initially everything looked good. Aside from the rather ominous "Leopard libedit detected." message on startup. Unfortunately once I started using it, problems quickly became apparent:

• Cursor not returning to the correct location on the line when using up/down arrow to browse history.
• No ctrl-r to search through history.
• Other random weirdness.

Fortunately Patrick van Kouteren wrote an excellent article that got me headed in the right direction. Unfortunately, building the ipython-packaged readline failed miserably during the initial configure process. As always, config.log tells the tale:

configure:1883: gcc -isysroot /Developer/SDKs/MacOSX10.5.sdk 
  -arch i386 -arch ppc -arch x86_64 -arch ppc64 -DNEED_EXTERN_PC
  -isysroot /Developer/SDKs/MacOSX10.5.sdk -arch i386 -arch ppc
  -arch x86_64 -arch ppc64 conftest.c  >&5
ld: library not found for -lcrt1.10.6.o
collect2: ld returned 1 exit status
ld: library not found for -lcrt1.10.6.o
collect2: ld returned 1 exit status

Fortunately for my sanity, it looks like distutils no longer tries to force building of Universal binaries on the Mac but still this looked awfully similar.

Solution:

1. Edit setup.py and comment out these lines:

   if distutils.util.get_platform().find('macosx-10.5') == 0:
       UNIVERSAL = '-isysroot /Developer/SDKs/MacOSX10.5.sdk ' + \
                   '-arch i386 -arch ppc -arch x86_64 -arch ppc64'
   

2. Edit rl/build.sh and comment out these lines:

   export CFLAGS='-isysroot /Developer/SDKs/MacOSX10.5.sdk 
     -arch i386 -arch ppc -arch x86_64 -arch ppc64'
   export LDFLAGS='-isysroot /Developer/SDKs/MacOSX10.5.sdk
     -arch i386 -arch ppc -arch x86_64 -arch ppc64'
   

• SATA vs. SCSI reliability
• Are Fibre Channel and SCSI Drives More Reliable?

Basically, drive size has increased dramatically. The "bit error rate"[1] has remained essentially constant. As you approach something around 12TB, it's virtually certain that you have unreadable blocks on your array.

Who cares if a couple of bytes in the middle of some old archived log files are munged? I sure as hell don't. Unfortunately, most failed blocks are only discovered during the one process that requires reading and checksumming every block of data on the array: Rebuilding the array after a drive failure. What happens then? Depends on the RAID controller. Apparently it's pretty common for it to give up and leave you with no useable data at all.

It wasn't until I read this article that I understood why the über-smart folks at Coraid implemented a technology called RAIDShield that basically continuously reads every sector of every disk in the background. If a bad block is found, the correct data is calculated from the parity data and written out to the "bad" sector. This triggers the drives internal sector remapping, guaranteeing that the data is written to a good portion of the disk and flagging the sector as unusable.

I really like knowing our vendors have solved problems we didn't even know we had. Makes me feel all warm and fuzzy.

[1] The ratio of the number of unreadable blocks to the total number of blocks on the drive/array.

Here's what I needed to do:

1. Install FreeTDS:

   port install freetds
   

2. Download and untar sqsh.

3. Configure as you would normally, setting $SYBASE to point to the top of the FreeTDS install:

   SYBASE=/opt/local ./configure --with-readline
   

4. Edit src/Makefile and append freetds to line #18 so it looks like this:

   SYBASE_INCDIR = -I$(SYBASE_OCOS)/include/freetds
   

5. Modify the header/library paths for libreadline to point to the version from DarwinPorts:

   READLINE_LIBDIR =  -L/opt/local/lib
   READLINE_INCDIR =  -I/opt/local/include/
   

6. make and make install

cputype (18, architecture ppc) does not match cputype (7) for specified -arch flag: i386 (file not loaded)

Scrolling back to look at the actual gcc command line revealed a clue. Present among the flags were: -arch i386 -arch ppc.

Another clue:

tarja:~ ben$ file /opt/local/lib/libcrypto.0.9.8.dylib /opt/local/lib/libcrypto.0.9.8.dylib: Mach-O dynamically linked shared library ppc

So Python's distutils was determined to build universal binaries, but the OpenSSL install created by MacPorts was PPC only. That's not going to work.

A grep through the source for distutils didn't reveal any instances of "-arch i386", so I resorted to Google. This post provided the necessary clue. After editing /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/config/Makefile and removing all references to -arch i386, the linking succeeded.

Almost.

Then I started seeing errors similar to this one: /usr/bin/ld: warning can't open dynamic library: /Developer/SDKs/MacOSX10.4u.sdk/opt/local/lib/lib3rdparty.dylib referenced from: libMyLibrary.dylib (checking for undefined symbols may be affected) (No such file or directory, errno = 2)

Note that in this example, lib3rdparty.dylib actually does exist in /opt/local/lib. As I eventually learned from this thread, using the -isysroot option prepends the SDK directory to all paths. As you'll note from the above-linked thread, whether or not this is a bug depends on who you ask. The kludgy workaround:

Make a symlink under /Developer/SDKs/MacOSX10.4u.sdk/ to /opt/local/lib.

Ugly.

But it works.

Thanks, Apple.

Capsule review:

Fascinating and highly entertaining corporate satire featuring rambling, multiple-forwarded voicemail, a sociopathic but brilliant receptionist, meaningless mission statements, and a donut.

Read this. Then read Syrup and Jennifer Government.

Extract from the first chapter of Company: READ

On the client side:

yes BBBBBBBBBBBBBBBBBBBBBB | nc 10.0.8.2 2223 > /dev/null

And on the server:

yes AAAAAAAAAAAAAAAAAAAAAA | nc -v -v -l -p 2223 > /dev/null

Is there anything netcat can't do?

When I bought my first good knife, it was easily defensible: every cook needs a high-quality chef's knife. When my wife bought me a gorgeous santoku, I would tell people that I prefered it for working with vegetables. Then I bought the new Ken Onion blade. I can't defend three knives.

So I've accepted that I am a collector. Which means that I can now justify buying any other cool knives that catch my eye!

The initial computer used for network authentication was, in retrospect, underpowered. In an attempt to compensate for the perceived configuration problems, I made a number of tweaks, including some timeout settings.

Over the weekend, I built a new server to be the directory server, hoping it would do better. It appeared to handle the queries without any noticeable load or slowdown, but was still dropping a large number of authentication attempts. Matt and I began an analysis of the actual network traffic, looking at the individual network packets. We noticed an anomaly in the way LDAP sessions were being closed down after a period of inactivity. The LDAP server was sending a "FIN,ACK" packet, the normal way to request the shutdown of a TCP session. The client responded, appropriately, with an "ACK" packet.

This effectively closed the connection from the server to the client, but not the connection in the other direction. The way the client should proceed from here is to respond with its own "FIN,ACK" packet, to which the server would've responded "ACK" and the session would be cleanly shut down.

Normal shutdown of a session:

 server                client

   (normal data exchange)
  <------- SEARCH ----------
  -------- RESULT --------->
    (starting shutdown)
  ---------FIN ACK--------->
  <----------ACK------------
     (halfway closed)
  <--------FIN ACK----------
  -----------ACK----------->
    (completely closed)

Instead, the client would, frequently, attempt to send a new search request through the half-closed connection. The server, righteously and appropriately indignant, responded with a "RST" (or "reset") packet, signalling that it was done and wasn't interested in hearing any more. The client, rebuffed and confused, would deem the server unreachable and go into a funk for 30 seconds or so (depending on its config).

Broken shutdown of a session:

server                client

    [normal data exchange]
  <------- SEARCH ----------
  -------- RESULT --------->
  [server starting shutdown]
  ---------FIN ACK--------->
  <----------ACK------------
      [halfway closed]
  <------- SEARCH ----------
  ---------- RST ---------->
     [client panics and
      stops talking]

Once we understood the problem, the solution was clear: Make sure the client always attempted to shut down idle sessions first, so its buggy half-closed SEARCH requests would not have opportunity to occur. So we upped the idle timeout on the LDAP server high enough to guarantee it would only come into play if the client machine was completely hung already. As soon as we made this change, all the logins became speedy, sessions stopped hanging, and all was right with the world.

I have now eliminated the original (slow) LDAP server completely, shifting all the load to the new ultra-beefy server. The next step is to upgrade the old server to a bit more modern standards and rebuild it to serve as a backup.