Perhaps the most rewarding work I’ve ever done. The Dundee Satelite Receiving Station had a PDP11 system to capture data from circumpolar satelites, and had recently upgraded to a 4/330 with a DR11W VME board. Unfortunately, this interface had no buffering, and we needed to capture 22K of data at 6Hz. I devised a multiprocess, non-blocking implementation which did the job admirably. My reward was seeing stunning live pictures of the earth. Utterly beautiful!
This is probably the biggest and longest running single project I’ve ever undertaken at Sun, and it was enough to convince me that I would prefer shorter “flog-it and leg-it” projects in future. DEC (remember them?) gone a long way with the customer, and a fully functional VMS/Oracle proof of concept implementation was in place, complete with VT100 screen dumps and hundreds of DCL scripts. It also made extensive use of VMS batch queues and print spooling. Somehow we won the business, and my job was to architect the final implementation on SunOS/Oracle. My solution was to convert all the DCL to Bourne Shell, and to write my own batch/print queueing system (which was network extensible with NFS … Sun Grid Engine eat your heart out!) plus tape management system. The system went live in over 50 sheriffs’ courts (some with just a single SPARCserver 1, but larger sites having a Sun 4/330 server and multiple SPARCserver 1 clients). This was my first exposure to Oracle and SQL*NET. I have no idea if or when the system was replaced (except that I was asked to advise on the Solaris 2 port a few year later).
This was an early instance of holistic system performance work. Automated Pit Trading was a world first. Instead of chaps in blazers waving at eachother and throwing orders on the floor (to be collected by sweepers), APT provided a SunView GUI to traders across London. Fairness was guaranteed by providing everyone with exactly the same hardware platform. The customer was not at all happy when we introduced the SPARCstation 1+, because it meant everyone had to be upgraded. RAM was always at a premium, and I had been asked to go in to pare down the kernel and system services to a minimum. While I was there, I was asked to investigate an application hang. It turned out that the printer (which was not spooled) had run out of paper, and that the application was blocking on a write to the printer port!
Probably the first serious code I wrote at Sun was my solution to the “rtelnet” problem (but it was much better than the utility of that name that came with certain TELNET terminal servers). The pty.telnet service allowed system administrators to setup robust, fixed mappings from named pseudo terminal devices to specific ports on specific terminal servers. It became a “UK Consulting Special”, which meant people paid real money for it. I alter ported it to Solaris 2, where the dynamically allocated, clone psuedo terminal devices required a dance with symlinks to provide the same functionality. I’ve got the code still somewhere, if anyone is interested!
On Tuesday (September 1st, 2009) I had the long-awaited phone call from my manager, and am now able to report that I will be leaving Sun on September 30th, after 20 years and 7 months. Yes, this differs from my previous post, and it’s not the first change of plan, but it’s hard to soar like eagles…
Although I’ve been waiting for this for some time, I’m still quite numb. Most of the last 20 years has been a lot of fun, but I’m not leaving the company I joined. However, there are still some very special people at Sun, and I will miss being their colleague.
I hope to post a few more things here before taking up “garden leave”, and to rehost the blog elsewhere so that I can continue to share a few morsels from time to time. I’m not sure what the future holds, but I know who holds the future, and that He has a plan.
The atmosphere was charged with emotion for Scott’s farewell at JavaOne, and I was there to witness and take part in the prolonged standing ovation. You could really feel the love (which is great because love covers a multitude of sins). In February I clocked up 20 years at Sun, so I remember the golden years with Scooter at the helm. But the one thing I have learned from his successor is to have an exit strategy. My current plan is for my fun at Sun to end on August 31st and to use the intervening time to blog some personal highlights from the past 20 years (the kind of stuff you’d love to put into a CV, but which would bore the socks off most potential employers). You have been warned.
This afternoon at CEC 2008, Jon Haslam took our experience of DTrace to a new frontier by stimulating our visual cortexes with some jaw dropping 3D visualisations. As we flew between the nodes of Solaris call graphs graph to the strains of Star Trek theme tunes, those heady early days of truss(1) seemed positively prehistoric.
I was there when they launched the Sun Storage 7000 Unified Storage Systems (“there” being the CEC conference in Las Vegas). Within an hour of the launch I had downloaded the Sun Unified Storage Simulator for VMware Fusion and started playing with this ultra-cool software stack on my MacBook Pro.
Later in the evening, Bryan Cantril and Mike Shapiro, the geniuses behind these amazing new products, issued a challenge for someone to be the first to set up a pair of laptops with a fully replicated virtualised service.
Not being one to pass up such challenges lightly, I nonetheless thought I would take it stage further with an attempt at setting up replicated servers on just one laptop (because that’s all I had with me).
I had hoped to write a long and detailed tale of how I laboured hard and long (against the all odds, etc) to bring this amazing feat to pass, but in the event it was just all too easy. In fact it is taking longer to write this blog than it took to do the work!
I used Fusion’s host-only networking to set up a private 10.0.0.0/24 network between the two virtual machines. Everything I tried “just worked” and was so intuitive, that there was no need to read any documentation.
Having demonstrated my solution to Mike, I took a series of three screen shots of my Mac desktop to illustrate the replication in progress for this blog. Each includes VMware Fusion consoles for both servers, the web management interface of each, and a local Terminal window showing the Mac being an NFS client of the primary server.
The first screen shot shows the replica in sync with the primary server, with each seeing 20.2MB in the default share.
The second shows a dd(1) command having just created a 10MB file, with 5.1MB (i.e. 25.3MB – 20.2MB) having made it to disk already on the primary server (but no change on the replica).
The final shot, taken less than a minute later, shows 30.2MB on both servers, the replication of the new 10MB file having just completed.
How cool is that?!
Over a year ago I blogged about building a Low power Solaris home server. I was very pleased with the 77W achieved, but have since added a couple of 1TB SATA drives. Sadly, my cheap Trust PW-5252 420W PSU didn’t see out its first year, so I swapped in a generic Antec unit from an unused PC that was kicking around.
I’ve just found a new home for the unloved PC, so I decided to swap the PSU back and to take the opportunity to upgrade my Solaris server’s PSU to something a little more environmentally friendly. I selected the Antec TruePower Trio 430 for its green credentials, especially its claimed “up to 85%” efficiency. I was amazed at the difference!
With the addition of the two 1TB SATA drives, and with the swapped-in generic Antec PSU, my Solaris server was pulling about 85W (i.e. up 10% from the 77W in original posting). With the Antec TruePower Trio 430 in place, it is now drawing a miserly 37 Watts! Some of this is doubltess due to smarter cooling (e.g. the internal 12cm fan hardly ever kicks in, and the PSU also takes over control of the case fans), but the majority is probably attributable to smarter circuit design.
Smarter, greener power supplies don’t come cheap. I paid £47 for mine. However, “doing the math” I find that this upgrade will have paid for itself in less than a year (mine is a 24×7 server). It also comes with a five year warranty! In all, buying the cheaper (£19) power supply has proven to be a false economy. Think on.
I’ve now been at Sun for the best part of two decades. It was Solaris and SPARC which first attracted me to Sun, and it’s exciting to see both very much alive and kicking all these years on. But this didn’t happen by chance. Sun currently spends $2B per year on R&D, making Sun one of the top 50 technology investors worldwide.
Having myself spent the last five years doing R&D, why have I decided to move back into Sun UK’s field organisation? Simply, it’s because I think Sun has a very compelling story to tell, and a very exciting portfolio of products and services to offer. In short, I think I’m going to have a lot of fun!
In my new role I’m finding that the thinking behind my Killer Combination,
Wicked Bible and Brief History postings is resonating well with a lot of people. Quite frankly, I’m astonished by the number of downloads of my extremely minimalist presentation given to the Sun HPC Consortium in Dresden.
Such has been the interest of late that I thought it would be worth sharing my latest minimalist slide deck: Solaris: greater than the sum of its parts. A lot of the material may be familiar, although I have been experimenting with tag clouds as an alternative to boring bullet points. My basic thesis is: Sun is where the innovation is, so why flirt with a imitations?
I’ve been a big Lego fan for as long as I can remember. Whilst some kids are content to follow the supplied instruction guides, the real thrill for me has always been that Lego allows me to design and build my very own creations, limited only by my imagination. I feel the same way about UNIX.
UNIX has always been about innovation. UNIX has always provided a rich set of simple, consistent, elegant and well defined interfaces which enable to developer to “go create”. This “innovation elsewhere” often takes the seed innovation to places unforeseen by its inventors, and this in turn leads to new innovations.
Lego has undergone a similar evolution. At first I only had chunky 2×2 and 2×4 bricks in red and white to work with. Then came 1xN bricks and 1/3 height plates and more colours. Next came the specialised pieces (e.g. window frames, door frames, wheels, flat plates, sloping roof tiles bricks, fences and so on). But all the time these innovations extended the original, well-designed interfaces, with a big commitment to compatibility, thus preserving investment in much the same way as the Solaris ABI (application binary interface).
Obviously, there are places where these parallels break down, but I think we can push the analogue a little further yet [note to self: a USENIX paper?]. In my mind, Solaris is somewhat akin to Lego Technics, and Solaris 10 to Lego Mindstorms. And in this vein, I see Linux rather in the Betta Builder mould (i.e. innovative interfaces copied from elsewhere, actually cheaper and lighter, but not quite the same build quality as the original implementation). And this is where I’m going to get a little more provocative.
In my new presentation I experiment with tag clouds to enumerate some of Sun’s more important UNIX innovations over time. The first cloud lists prehistoric stuff from SunOS 3 and 4 days. The second focussed mostly of Solaris 2. The third focusses on Solaris 10. And while Sun may not be able to take sole credit for ideas such as /proc and mmap, it can claim to have the first substantive implementations.
The fourth tag cloud is included to demonstrate that Sun does not suffer from terminal NIH (not invented here) syndrome. Indeed, I think it recognises that Sun is a pretty good judge of excellence elsewhere (most of the time).
Whatever you think of the detail (and I concede some of it could do with a little more research) I do think it is helpful to ask “where does the innovation happen?”. At the very least, I think I’ve shown that there is heaps of innovation in Solaris which we simply take for granted.
To put it another way: as a Solaris enthusiast I can’t help feeling at ease in a Linux environment because I find so many familiar objects from home (I guess a GNU/BSD devotee might say something similar). That’s not to deny the achievements of the Linux community in implementing interfaces invented elsewhere, but when I look at the flow of innovation between Solaris and Linux it does feel rather like a one-way street.
We live in interesting times! My own particular area of interest is multithreading. With the dawning of the brave new world of large scale chip multithreading Solaris seems uniquely placed to ride the next wave. This is not by accident. Sun has made a huge investment in thread scalability over the past 15 years.
One of my slides asks “What is the essential difference between single and multithreaded processes?” For some this is not a trivial question. For some it depends on which thread library is being used. But with Solaris, where application threads have been first class citizens ever since Solaris 10 first shipped, the answer is simply “the number of threads”.
Enough of this! The weekend is upon us. Where’s my Lego?