Obscene Art

So, over at the Virtualmin forums I get a lot of questions about which operating system to use. Partly, they’re asking, “What does Virtualmin support?”, but they’re also looking for guidance on what will be most productive and least problematic for the tasks they want to perform with the system. Given that I’m building server management software, and the majority of the systems I deal with are web servers, the following advice pretty much applies exclusively to servers. Based on my past ten or so years of heavy Linux usage on servers, I’m comfortable calling myself an expert on OS selection and deployment, so disregard the advice at your own peril.

The good news is that I’m not going to tell you that any particular Linux distribution sucks for servers. Some of them do, but I won’t spend a lot of time naming names, except where there is a clear and objective measure that clearly makes some distros better than others. I’d much rather cover what I’ve found makes an OS great for server usage and what makes one painful.

Which brings us to the very first and single most important factor in your OS decision.

Support Life Cycle

Servers remain in service far longer than desktop machines. Migrating from one to another is painful and time-consuming (even with tools like Virtualmin providing a really clean migration path, you still have the DNS and temporary loss of service issues to contend with), so you want to do it as rarely as possible. But, you cannot safely run a server without regular security updates. You just can’t. Even if you carefully choose historically secure packages for your services (Postfix or qmail for MTA, OpenSSH for file transfer and shell access, Webmin for web-based administration, Dovecot for POP/IMAP, etc.) and carefully shut down every service you won’t be needing or firewall it down to just the hosts that need to talk to them, some security problem will pop up in the future. If you can’t easily upgrade that service within hours of the vulnerability being discovered, your system will be compromised, and you’ll be forced into a rapid migration to a new system, or trying to find a security consultant who can clean up the problem and paying the very high price that said consultants charge.

So, pick an OS with a long life cycle, and choose the most recent version of that OS. Debian has a very long life cycle. RHEL and CentOS have a very long life cycle. Mandriva and SUSE have a long life cycle (but I believe you have to get the paid versions to get the long-term support). Ubuntu LTS also has a long life cycle, but non-LTS releases do not, so you’d want to avoid any version other than LTS. Fedora, now that Fedora Legacy is gone, has a very short life cycle, and should be avoided for servers. I’m unable to find the life cycle of Gentoo.

I’m not kidding about life cycle. If you pay no attention to anything else in this article, pay attention to this advice. I’ve managed hundreds of servers in my life, and the single biggest pain point is updates and upgrade management and migration when updates and upgrades are no longer feasible. Avoid that pain as much as possible!

Package Management

Packages are great. They take away several major pain points in managing servers. So, choose an OS that offers good package management. The definition of “good” in this context is that it, roughly in order of importance: handles dependencies automatically, allows alternate software repositories so you can manage local packages or make use of third party repositories for specialized software, offers access to a wide array of “official” or at least nicely built third-party packaged software, allows you to make your own packages or customize existing ones, allows unattended operation and operation from scripts.

apt-get+dpkg and yum+RPM and urpmi+RPM are the best package managers available today, meeting all of those qualifications. After spending a couple of weeks porting our application and all of its dependencies to Debian 4.0 and CentOS/RHEL 5, I can say that Debian wins on the package management front, due to the sheer number of packages available in the universe repository. I like yum and RPM better from a technical perspective, and Fedora Extras makes it comparable to Debian on the package management front, but we’ve already disqualified Fedora due to stupidly short life cycle.

Ease of Management

Every Linux distribution has quirks that make it harder to use than it should be. In some cases, better ways are known and well understood, but historical behavior sticks around due to inertia and remaining compatible with existing users expectations. That said, there is value in predictability and good documentation. I find CentOS and RHEL to be the most capable out-of-the-box, and with quite sane management methods. Debian is also strong, but has a few historical stupidities, such as lingering reliance on START=yes crap in configuration files for some services. In a world where chkconfig has been around for years, this is foolishness. This is merely an example, and is a minor nuisance in any single instance, but by the time you’ve had to edit a dozen files just to get your services running (even if no configuration is required), it becomes painful.

SUSE has made a habit of changing major components of the system in every single revision. Package management on SUSE has changed in major ways three times in the past three revisions, and it’s still weaker than yum or apt-get. Configuration files move from one version to the next, and many can only be sanely managed with YaST. In fact, SUSE has a horrible habit of designing the underlying system for the convenience of the YaST developers. A specific example, so folks won’t think I’m just being bitchy because we build a management tool that overlaps with YaST in some areas: SUSE uses a firewall save format different from every other distro (which uses the standard iptables save file format). The SUSE format is less powerful, less flexible, more complicated to edit, and incompatible with all other firewall management tools. There’s no reason for this, except that it’s easier for YaST to modify a few variables in a shell script than to actually parse the iptables save file format and deal with it cleanly.

Webmin and Virtualmin nearly nullify the management problem, by abstracting it out to a nearly identical interface across all Operating Systems. But even with Webmin and Virtualmin, there are still differences that can increase the friction of going from zero to “in service”. I also like to know that when I hit the command line, the configuration files and mechanisms are sane and well-documented.

Documentation

Having good documentation is pretty darned important. And by documentation, I refer to every scrap of information that Google can dig up for you on a particular problem, not just the official documentation for the distribution. In this regard, the more popular an operating system for the tasks you’re performing, the more likely you are to find good documentation on the web.

The Debian official documentation is particularly verbose and particularly useless, and Ubuntu only marginally corrects that. The Debian documentation suffers from the same disease as man and info pages in the GNU system: No examples. Both are quite popular on servers, though, so one can often find good examples via Google.

The Red Hat, CentOS and Fedora documentation is much stronger, despite being less voluminous. The Red Hat based systems, are also the most popular, and so examples and discussions on the web abound. I’ve found them to be the easiest to find solutions for. If you’re new to Linux going with the most popular distro might be wise.

Gentoo has shockingly good official documentation with fantastic examples. The distro itself falls down in numerous other categories for server usage (pretty much all of the above, as far as I can tell), but the documentation is simply awesome. Whatever process they’ve used for their documentation ought to be replicated by the other distributions, because despite the youth of the Gentoo platform, it has far better official documentation than any other distribution. Hell, I need to spend some time studying how the Gentoo documentation project is run, so I can replicate it for Virtualmin and Webmin.

SUSE is far more popular on desktops than servers, and so server-related documentation is slimmer than we’d like. It is also less popular than Debian/Ubuntu and the Red Hat based systems, and so finding stuff on the web is less likely. The official documentation is of good quality, but again, there is simply less topic coverage.

Mandriva also has reasonably good official documentation, though it also is more popular on the desktop, and so much of the official and unofficial documentation is focused on things that we, as server administrators, don’t care about.

Your Use Case

This one is more difficult to give advice on, because it depends on what you’re doing with your system. If you’re a .net developer, you might look to SUSE, because Mono is a core part of that system. If you like Python, you might consider Red Hat or CentOS, because Python is an important part of their system and thus the packages are well-cared for in the official repository. Perl is well-supported by all of the distributions (though Mandriva and Debian have the most official packages for various CPAN modules), and Ruby is equally poorly supported by all of them (though Debian 4.0 seems to have improved remarkably on this front). Debian and Ubuntu are particularly strong on PHP, as they provide official packages for both PHP4 and PHP5, and they can co-exist on the same system (it took me weeks to replicate this capability on the Red Hat based systems).

So, what are you doing with your server? Search the web before making your OS decision to see if others are using one particular OS more often for tasks like the ones you’ll be tackling. This is a variation of the old “it’s the applications, stupid” rule. If all of your necessary applications are being used heavily on a particular OS, like maybe by the developers themselves, you’d be well-advised to pay attention to that.

In our case, we are heavily invested in Perl development, so our options are pretty wide open. We also use Joomla for our new in-development website, so we need reliable PHP support. Another, particularly difficult, piece of our puzzle is that we need a virtualized server to run our demo on, so native support for virtualization would be nice. We currently reside on a Red Hat Enterprise Linux 4 server, but if I were choosing today, I’d probably select Debian 4.0 on the strength of the universe repository. But, that’s just me.

Y’all pick your favorites, and let me know what criteria you use, but try to avoid “I like it because it’s easier to X”, because that intangible quality is almost always a function of what you’re familiar with…I’ve tried to steer clear of familiarity as a basis for selection, since there are far more folks who don’t have familiarity with any particular Linux distribution than there are that do.

I’ve been in the planning stages of a new product for Virtualmin users (and eventually for anyone who has a web server), which would predominantly run on on our servers. It would be our first application that allows us to choose the implementation language (our current codebase is over nine years old and includes hundreds of thousands of lines of Perl…we won’t be changing languages for our installable products, though we do look forward to Perl 6).

This got me to thinking about language popularity. PHP is, by far, the most popular language for the web. It’s also the worst among the popular choices. At least it’s the worst by every metric I consider important…except one: Availability.

PHP syntax is horrible. The vast majority of PHP code in the wild is horrible. The PHP standard library is inconsistent from top to bottom. It encourages bad practices, and dangerous behavior is the default for many PHP functions. Only in the past few years has PHP acquired the ability to safely and sanely use databases. The database problem is the cause of a huge percentage of the horrible PHP code in the wild…and folks are still writing code using the ugly way to access databases, despite better ways existing, because it’s all they’ve ever seen. PHP code, in general, is hard to maintain, hard to read, and often poorly documented (again, a lot of the documentation covers historic techniques that are simply dangerous). The object system is bolted on. It has no functional programming constructs to speak of.

So, why the heck is it so pervasive? Why on earth would smart people choose to use a language that is less effective for almost every non-trivial task in web development? And, I should be clear, smart people do choose to use it, and some fantastic applications have been written in it. This isn’t a situation where a bunch of idiots are using it because they don’t know better. Some of the coolest web applications are written in PHP, by some of the coolest developers working in the field.

The answer, and an answer that the Ruby On Rails folks, Pylons folks, Django folks, Perl Catalyst folks, etc. should all pay really close attention to (assuming they want their apps to be widely used), is that PHP is pervasive because it is pervasive. Tautology, I know, but what I’m trying to get at is that people code in PHP because they know it will work wherever they go or whoever they hand it off to. They know every web developer will be familiar with it, and will have access to a webserver that can run it. They know it will run on the cheapest DreamHost account, the midrange TextDrive account, or the most expensive RackSpace managed server, without modification and without jumping through too many hoops. They also know that if they have to hire someone to maintain it, or round up a community of users to help add features, there are no shortage of knowledgeable developers (it’s hard to find the good ones, as in any language, but some things don’t require good developers). Further, it’ll be reasonably fast, if they don’t do anything stupid or pathological with databases or files.

In short, if you want your installable application to be used, probably the single most useful thing you can do is write it in PHP. You also need to make it do something good, of course, and without huge glaring bugs. I’m guessing, but based on UI studies that show that for every field you add to a form, the completion rate drops by a huge amount (an order of magnitude in some extreme cases), I believe that an installable PHP application will initially get ten times the users of a Ruby On Rails application that does the same thing. I don’t have numbers to back up this bold assertion, of course, but I’m pretty sure that’s the way it will play out, unless and until Ruby On Rails is as pervasively available as PHP (which is unlikely to ever happen).

I don’t like this fact. I’m obviously not a fan of PHP as a language. I can’t think of a commonly used web development language that I would choose PHP over from a purely technical standpoint, except Java or C# (and even those have certain areas where they are particularly strong and well-suited for the task). But if I were building a small application (by small, I mean anything under 50,000 lines of code) that I wanted thousands of users to install, I would probably write it in PHP.

Of course, I’m not building an installable application this time…so I’ll be picking among the languages I like (Perl, Haskell, Python, Erlang). But my moments of blinding insight are few and far between, so I thought I’d share this one with the 12 people that visit my blog.

Virtualmin.com runs on a mildly customized version of an Open Source CMS called OpenACS 5.1.5, but it won’t be running on it for much longer. OpenACS is one of the oldest content management systems available. It is written in TCL and runs on top of AOLServer and PostgreSQL.

Some folks are already saying, “Ah! It’s written in TCL and you said the word AOL. No wonder you’re leaving OpenACS!” But that’s not why. I’m predominantly a perl developer working on a large existing codebase…I know that dropping a mountain of working code just to work in a more fashionable language is foolish. I also know that, despite its bad reputation, modern TCL is actually a pretty good scripting language. It’s not as pretty as Python or Ruby (but what is?) and not as powerful as Perl+CPAN (but what is?), but it’s a pretty good mix of features and clean syntax. TCL even has a few cool Lisp-y bits thrown in; which, as we all know, means that it is closer to god or something. More importantly, OpenACS seemingly provides a mountain of working code.

No, the reason I’m leaving OpenACS is because large portions of that existing mountain of code, despite claims to the contrary, don’t work. Worse, the aspects of the code that don’t work are not becoming fewer with each new revision, they are becoming more common and more difficult to fix. Working applications are being deprecated in favor of incomplete, bug-ridden, and poorly documented rewrites. What was once a pragmatic workhorse is now an academic playground. Unfortunately, I have only experienced the academic playground era of the project–but I’ll take everyone’s word for it that ACS was once a helluva stable and cool set of web applications.

So far, I’m just being bitchy, and not really doing much to point out the real and serious flaws in OpenACS.

An example is in order. The OpenACS Bug Tracker. It’s a nice bug tracker. You can see it in action at OpenACS Bugs. Have a look around. It’s slick. Very powerful, entirely useable, fast, searchable, all sorts of good things. I approve. So what’s the problem? What you see is not the code you get when you download and install OpenACS. In fact, it shares almost no code with the new bug tracker. How can that be, you wonder? Me, too. The OACS bug tracker was rewritten a year or two ago by Lars Pind, a smart fellow who thinks very deeply about web application problems. The problem is that Lars didn’t set out to work on the bug tracker. He set out to make a cool workflow package, and somewhere along the way decided that a bug tracker would be a great example application. Apparently, the rest of the OpenACS folks agreed, because when the bug tracker was rewritten to use Lars’ new workflow package, it replaced the original bug tracker.

The new bug tracker looks the same on the surface. It has no new features or bugfixes. As one might expect of all new code, the new bug tracker has some new bugs of its own. The most egregious being that it doesn’t allow discussion of bugs except between the site administrator and the creator of the bug. Think about that for a moment. How many times have you googled about a bug, found an entry in bugzilla or roundup or trac, and dropped in to leave your two cents. Maybe you’ve got a solution or a workaround to contribute. Maybe you just want to get on the CC list so you know what’s happening. Or you have an extra data point that might help the developer reproduce or isolate the problem. It’s an absolutely vital and core part of the modern bug tracker. And the new OACS bug tracker can’t do it.

“Big deal!” I hear you cry. I said so, too, when I was excited enough about OpenACS to ignore all of the problems and push forward with our deployment. “I’ll fix that this weekend. How tough can it be? Really, we’ll just drop a comment button on that sucker for other users and all is well. I bet it’s three lines of code and a template change.”

A year later the bug still exists. I spent a couple of whole weekends trying to fix it, and then after a couple of months of periodically trying to fix it myself, I filed a bug. One of the OpenACS developers has spent quite a bit of time trying to fix it. It’s been discussed in the OACS forums. It’s been discussed ad nauseum in a bug about the problem. I’m not a software development guru, and I’ll be the first to admit it, but this makes me very itchy. How can such a seemingly simple problem be seemingly impossible to solve? Probably, Lars Pind could drop in and fix it in an hour, but he’s not around on the OACS project any more…so it’s gotta be someone else, and no one else is available. In fact, no one else on the project even seems to realize that the bug tracker they’ve been using at OACS.org is completely different code from the one that’s being distributed with the OACS downloadable packages. It does seem that a few folks are using Lars’ workflow in other projects, so the expertise is probably there somewhere…they’re just not working on existing OACS code.

Which brings me to an interesting and confusing aspect of the OpenACS project. The developers on the project are universally very smart. They’re well-educated in software development, many of the them have years of of web application development experience, they keep on top of modern development techniques, and are aware of the competition. The codebase they started from (ACS) was, by all accounts, among the best in the industry. But somehow, the current state of the project can only be described as “a mess”.

The scary part is that I can’t really point to any obvious thing that the OACS folks are doing wrong. They are aware of many of the problems, and have excellent sounding plans for resolving them. In the beginning of my time with OACS, I thought I was seeing a rebirth of a once-great CMS. I accepted all of the minor quirks, because I could see the incredible potential of the system. It has a lot of great applications, already built. I’m a big fan of “already built”, when it comes to software development. I like to put together components, and I don’t like starting from scratch. What could be better than a project that has dozens or hundreds of supposedly well-written, and time-tested, applications?

But, it’s over a year later. The ecommerce application is still just a bunch of incomplete components that have to be heavily modified to be useful. Well documented applications like “Edit This Page” have been deprecated with no documented replacement…the documentation still covers using ETP, as though it works, when it hasn’t for several revisions. The tutorials for creating your own OACS applications result in non-functional code.

In short, OACS is a mess and has been for the entire time I’ve been using it, and it hasn’t improved remarkably in the areas that I use. So, I’m jumping ship. Rather than upgrade to 5.2.x on my production server, I’m going to use that time to write a migration script to move our users, forum posts, and bugs, from OpenACS to some other CMS. I don’ t know what that CMS will be, yet, but Joomla is certainly looking like a nice package from the hour or two I’ve spent playing with it (I’m going to reserve judgement until I’ve spent more time with it, since early enthusiasm is what got me into the OACS mess to start with).

I’ve spoken about the problems with OpenACS several times in the past year in the OACS forums, so if you’re interested in more of my bitch-n-moan sessions, this’ll keep you busy for a good half an hour or so:

http://openacs.org/forums/message-view?message_id=351756
http://openacs.org/forums/message-view?message_id=352267
http://openacs.org/forums/message-view?message_id=352288
http://openacs.org/forums/message-view?message_id=352505

Note that all of these posts are several months old, some nearly a year. All issues addressed still apply, as far as I can tell from a glance at my devel server (which is running the latest version of OACS). Which is why I’m moving off of OACS rather than spend more time/money/frustration on it. I’d use PHP before I’d touch another line of OpenACS code, and if that aint a damning proclamation, I don’t know what is.