Obscene Art

Being the undisputed master of all things IT, I get a lot of questions about security. It seems that a lot of people are worried about various nefarious hacker types (hereinafter referred to as “bad monkeys”) breaking into their system and doing dastardly deeds, like sending spam or deleting all of those funny pictures of cats fixing servers.

Because my knowledge of IT and system security is so vast, and my understanding of the dark and twisted minds of bad monkeys is so deep, I could write a book on the subject of computer security. I won’t. But I sure could, you bet yer sweet bippy, I could.

Instead, I’m going to provide for you, absolutely free of charge, a timeless and unbeatable security plan in the form of a bulleted list:

• Update your system and all software that runs on it, every day or so. This is easy on a good OS that has good package management, like apt-get or yum. You can even set it to happen automatically. If you don’t have a well-supported OS that is easy to update, replacing it with one that is would be step one.
• Turn off unneeded services. If you aren’t sure what it does, find out. Google knows. If you don’t use it, turn it off.
• Use strong passwords. Your birthday is not a strong password. Your name is not a strong password. Your dog’s name is not a strong password. Any one of the passwords on this list, or this one, or this one, are not strong passwords. A strong password is one that is eight or more characters in length, has letters and either numbers or symbols or both, and is not based on a dictionary word.

That’s it. You’re done. Your system is secure, forever and ever, amen. Now you don’t need to ask me about securing your server anymore. As long as you write secure code to run on it, you can rest easy knowing that the bad monkeys will be flinging their poop around on some other poor slobs server.

So, over at the Virtualmin forums I get a lot of questions about which operating system to use. Partly, they’re asking, “What does Virtualmin support?”, but they’re also looking for guidance on what will be most productive and least problematic for the tasks they want to perform with the system. Given that I’m building server management software, and the majority of the systems I deal with are web servers, the following advice pretty much applies exclusively to servers. Based on my past ten or so years of heavy Linux usage on servers, I’m comfortable calling myself an expert on OS selection and deployment, so disregard the advice at your own peril.

The good news is that I’m not going to tell you that any particular Linux distribution sucks for servers. Some of them do, but I won’t spend a lot of time naming names, except where there is a clear and objective measure that clearly makes some distros better than others. I’d much rather cover what I’ve found makes an OS great for server usage and what makes one painful.

Which brings us to the very first and single most important factor in your OS decision.

Support Life Cycle

Servers remain in service far longer than desktop machines. Migrating from one to another is painful and time-consuming (even with tools like Virtualmin providing a really clean migration path, you still have the DNS and temporary loss of service issues to contend with), so you want to do it as rarely as possible. But, you cannot safely run a server without regular security updates. You just can’t. Even if you carefully choose historically secure packages for your services (Postfix or qmail for MTA, OpenSSH for file transfer and shell access, Webmin for web-based administration, Dovecot for POP/IMAP, etc.) and carefully shut down every service you won’t be needing or firewall it down to just the hosts that need to talk to them, some security problem will pop up in the future. If you can’t easily upgrade that service within hours of the vulnerability being discovered, your system will be compromised, and you’ll be forced into a rapid migration to a new system, or trying to find a security consultant who can clean up the problem and paying the very high price that said consultants charge.

So, pick an OS with a long life cycle, and choose the most recent version of that OS. Debian has a very long life cycle. RHEL and CentOS have a very long life cycle. Mandriva and SUSE have a long life cycle (but I believe you have to get the paid versions to get the long-term support). Ubuntu LTS also has a long life cycle, but non-LTS releases do not, so you’d want to avoid any version other than LTS. Fedora, now that Fedora Legacy is gone, has a very short life cycle, and should be avoided for servers. I’m unable to find the life cycle of Gentoo.

I’m not kidding about life cycle. If you pay no attention to anything else in this article, pay attention to this advice. I’ve managed hundreds of servers in my life, and the single biggest pain point is updates and upgrade management and migration when updates and upgrades are no longer feasible. Avoid that pain as much as possible!

Package Management

Packages are great. They take away several major pain points in managing servers. So, choose an OS that offers good package management. The definition of “good” in this context is that it, roughly in order of importance: handles dependencies automatically, allows alternate software repositories so you can manage local packages or make use of third party repositories for specialized software, offers access to a wide array of “official” or at least nicely built third-party packaged software, allows you to make your own packages or customize existing ones, allows unattended operation and operation from scripts.

apt-get+dpkg and yum+RPM and urpmi+RPM are the best package managers available today, meeting all of those qualifications. After spending a couple of weeks porting our application and all of its dependencies to Debian 4.0 and CentOS/RHEL 5, I can say that Debian wins on the package management front, due to the sheer number of packages available in the universe repository. I like yum and RPM better from a technical perspective, and Fedora Extras makes it comparable to Debian on the package management front, but we’ve already disqualified Fedora due to stupidly short life cycle.

Ease of Management

Every Linux distribution has quirks that make it harder to use than it should be. In some cases, better ways are known and well understood, but historical behavior sticks around due to inertia and remaining compatible with existing users expectations. That said, there is value in predictability and good documentation. I find CentOS and RHEL to be the most capable out-of-the-box, and with quite sane management methods. Debian is also strong, but has a few historical stupidities, such as lingering reliance on START=yes crap in configuration files for some services. In a world where chkconfig has been around for years, this is foolishness. This is merely an example, and is a minor nuisance in any single instance, but by the time you’ve had to edit a dozen files just to get your services running (even if no configuration is required), it becomes painful.

SUSE has made a habit of changing major components of the system in every single revision. Package management on SUSE has changed in major ways three times in the past three revisions, and it’s still weaker than yum or apt-get. Configuration files move from one version to the next, and many can only be sanely managed with YaST. In fact, SUSE has a horrible habit of designing the underlying system for the convenience of the YaST developers. A specific example, so folks won’t think I’m just being bitchy because we build a management tool that overlaps with YaST in some areas: SUSE uses a firewall save format different from every other distro (which uses the standard iptables save file format). The SUSE format is less powerful, less flexible, more complicated to edit, and incompatible with all other firewall management tools. There’s no reason for this, except that it’s easier for YaST to modify a few variables in a shell script than to actually parse the iptables save file format and deal with it cleanly.

Webmin and Virtualmin nearly nullify the management problem, by abstracting it out to a nearly identical interface across all Operating Systems. But even with Webmin and Virtualmin, there are still differences that can increase the friction of going from zero to “in service”. I also like to know that when I hit the command line, the configuration files and mechanisms are sane and well-documented.

Documentation

Having good documentation is pretty darned important. And by documentation, I refer to every scrap of information that Google can dig up for you on a particular problem, not just the official documentation for the distribution. In this regard, the more popular an operating system for the tasks you’re performing, the more likely you are to find good documentation on the web.

The Debian official documentation is particularly verbose and particularly useless, and Ubuntu only marginally corrects that. The Debian documentation suffers from the same disease as man and info pages in the GNU system: No examples. Both are quite popular on servers, though, so one can often find good examples via Google.

The Red Hat, CentOS and Fedora documentation is much stronger, despite being less voluminous. The Red Hat based systems, are also the most popular, and so examples and discussions on the web abound. I’ve found them to be the easiest to find solutions for. If you’re new to Linux going with the most popular distro might be wise.

Gentoo has shockingly good official documentation with fantastic examples. The distro itself falls down in numerous other categories for server usage (pretty much all of the above, as far as I can tell), but the documentation is simply awesome. Whatever process they’ve used for their documentation ought to be replicated by the other distributions, because despite the youth of the Gentoo platform, it has far better official documentation than any other distribution. Hell, I need to spend some time studying how the Gentoo documentation project is run, so I can replicate it for Virtualmin and Webmin.

SUSE is far more popular on desktops than servers, and so server-related documentation is slimmer than we’d like. It is also less popular than Debian/Ubuntu and the Red Hat based systems, and so finding stuff on the web is less likely. The official documentation is of good quality, but again, there is simply less topic coverage.

Mandriva also has reasonably good official documentation, though it also is more popular on the desktop, and so much of the official and unofficial documentation is focused on things that we, as server administrators, don’t care about.

Your Use Case

This one is more difficult to give advice on, because it depends on what you’re doing with your system. If you’re a .net developer, you might look to SUSE, because Mono is a core part of that system. If you like Python, you might consider Red Hat or CentOS, because Python is an important part of their system and thus the packages are well-cared for in the official repository. Perl is well-supported by all of the distributions (though Mandriva and Debian have the most official packages for various CPAN modules), and Ruby is equally poorly supported by all of them (though Debian 4.0 seems to have improved remarkably on this front). Debian and Ubuntu are particularly strong on PHP, as they provide official packages for both PHP4 and PHP5, and they can co-exist on the same system (it took me weeks to replicate this capability on the Red Hat based systems).

So, what are you doing with your server? Search the web before making your OS decision to see if others are using one particular OS more often for tasks like the ones you’ll be tackling. This is a variation of the old “it’s the applications, stupid” rule. If all of your necessary applications are being used heavily on a particular OS, like maybe by the developers themselves, you’d be well-advised to pay attention to that.

In our case, we are heavily invested in Perl development, so our options are pretty wide open. We also use Joomla for our new in-development website, so we need reliable PHP support. Another, particularly difficult, piece of our puzzle is that we need a virtualized server to run our demo on, so native support for virtualization would be nice. We currently reside on a Red Hat Enterprise Linux 4 server, but if I were choosing today, I’d probably select Debian 4.0 on the strength of the universe repository. But, that’s just me.

Y’all pick your favorites, and let me know what criteria you use, but try to avoid “I like it because it’s easier to X”, because that intangible quality is almost always a function of what you’re familiar with…I’ve tried to steer clear of familiarity as a basis for selection, since there are far more folks who don’t have familiarity with any particular Linux distribution than there are that do.

While I enjoy seeing Mentos and Diet Coke symphonies as much as the next guy, I find myself spending a lot more time watching technical videos on Gootube and Yahoo! Video…Some of them are simply fantastic, so I thought I’d share a list of my favorites.

Guy Kawasaki’s Art of the Start presentation – Holy crap. You don’t get much better than this. Best 40 minutes you can spend if you’re starting a company or thinking of doing so.

YUI Theater – This one is awesome. Douglas Crockford has a three part workshop series that’s a soup to nuts JavaScript and DOM introduction. I learned JavaScript (by some definition of “learned”) in one weekend from this series of videos. There’s also a nice video in which Joe Hewitt introduces his FireBug, which is the greatest thing to happen to Firefox since…ummm…ever. Nothing else comes close to FireBug, and it’s been absolutely vital to my successes with JavaScript, so far. There’s a lot of other cool stuff in FireBug that this video doesn’t cover, but it’ll get you started.

Anatomy of a Debian Package – Unfortunately, I find Debian and Ubuntu documentation about packaging to be horribly opaque, due to lack of examples of the process of packaging. This video runs a little slow, but it’s at least a usable source of information. I really just want to see a “create this file, put this into it, now run some-command to build the software and bundle it into a package…and here’s the command you use to sign it”, similar to several existing examples for RPM. That part of the video starts at about 20 minutes in. I wish some of the docs would have pointed out dh_make and how to use it…would have saved me hours of frustration when I first started packaging debs. And, I also wish some of the too many options for building debs would be deprecated. There’s just too many, “or you could do it this way…or you could to it that way…or you could do it typing only with your toes”. Let me explain something to the Debian folks: I never want to build the damned thing manually. Get mentions of that stuff the hell out of the introductory packaging documentation. Same goes for signing packages and repositories. I don’t care how the hell you implemented package signing…show me what command to run and tell me where the resulting files (if any) go in my apt-get repositories. (Actually, this video doesn’t do much good with package signing, either.)

Competing on the Basis of Speed – This one isn’t particularly technical, but it’s interesting. I believe running really fast is absolutely key to success in the current tech industry. The key to this is stripping away things that are pointless or stupid about your business, product, or process. The trouble is figuring out what is pointless and/or stupid. Being a young and tiny company, it seems easier…but I suspect that even our two-man shop has some stupid and pointless pieces, which I hope we’ll shake out pretty quick once we have some infrastructure in place for measuring how our customers are really using the product.

As I mentioned in a previous post, I’ve been hanging out with a lot of mobile application developers lately. Now, I’ve never made a secret of my disdain for telephones. But seeing the things they’re doing with mobile technology got me to thinking. What’s a cool use for mobile technology that applies to what I’m doing every day with Webmin and Virtualmin? Well, it turns out it’s the same thing that gets me to spend $72/month on a Sidekick. Being able to administer my server, fix customer problems, and deal with email, while traveling…even if there’s no laptop in sight or WiFi in range. So, on our way home from dinner last Tuesday night, Jamie and I chatted about what keeps Webmin and Virtualmin from being useful on our phones. Technically we can login to Webmin via our phones (Jamie has a Treo and I have a Sidekick), the problem is that it’s so slow to load all of the graphics and deal with the frames and menus (not to mention clicking “OK” when any site with more than a few lines of JavaScript eats up all of the memory on my Sidekick and it can’t keep running them), that it’s faster to drop in via SSH (both of our phones also have SSH clients) and do it all on the command line. Despite the horrid little keyboard, this is faster than using the web interface.

So, we came to the conclusion that we needed a new theme just for mobile phones. A theme targeted to the most common operations, and stripped of everything unnecessary, so it would be fast despite the client being slow and limited. But, we’d also have to address the problem of showing that theme only when a mobile client connected, since we can’t reasonably make an AJAX-filled fully graphical theme degrade all the way down to tiny. So, Jamie added some code to detect mobile browsers:

# is_mobile_useragent(agent)
# Returns 1 if some user agent looks like a cellphone or other mobile device,
# such as a treo.
sub is_mobile_useragent
{
local ($agent) = @_;
local @prefixes = (
    "UP.Link",    # Openwave
    "Nokia",      # All Nokias start with Nokia
    "MOT-",       # All Motorola phones start with MOT-
    "SAMSUNG",    # Samsung browsers
    "Samsung",    # Samsung browsers
    "SEC-",       # Samsung browsers
    "AU-MIC",     # Samsung browsers
    "AUDIOVOX",   # Audiovox
    "BlackBerry", # BlackBerry
    "hiptop",     # Danger hiptop Sidekick
    "SonyEricsson", # Sony Ericsson
    "Ericsson",     # Old Ericsson browsers , mostly WAP
    "Mitsu/1.1.A",  # Mitsubishi phones
    "Panasonic WAP", # Panasonic old WAP phones
    "DoCoMo",     # DoCoMo phones
    );
local @substrings = (
    "UP.Browser",         # Openwave
    "MobilePhone",        # NetFront
    "AU-MIC-A700",        # Samsung A700 Obigo browsers
    "Danger hiptop",      # Danger Sidekick hiptop
    "Windows CE",         # Windows CE Pocket PC
    "Blazer",             # Palm Treo Blazer
    "BlackBerry",         # BlackBerries can emulate other browsers, but
                          # they still keep this string in the UserAgent
    "SymbianOS",          # New Series60 browser has safari in it and
                          # SymbianOS is the only distinguishing string
    );
foreach my $p (@prefixes) {
        return 1 if ($agent =~ /^\Q$p\E/);
        }
foreach my $s (@substrings, @mobile_agents) {
        return 1 if ($agent =~ /\Q$s\E/);
        }
return 0;
}

So, there’s a quick and dirty perl function (BSD-licensed, like all of Webmin core) that makes detecting a mobile client easy. Of course, there’s more granularity out there than “mobile” or “big” client. That’s where something like WURFL will come in (that’s not gonna happen in Webmin in the short term, but it’ll come later, as our themes get smarter and more dynamic). Anyway, so now that Webmin can detect what kind of client it’s talking to, it can choose a theme to suit the target. We’ve made a new dead simple, zero-graphics, menu-based theme for Virtualmin, that allows really quick administration on even the most limited web client.

Now, as of version 1.328 of Webmin and next weeks version of our themes, you’ll get a different interface based on how advanced your client is. Small clients will see something like this:

While big clients will see one of the all-singing, all-dancing, maybe AJAX-filled, themes with images and such. Everybody wins, and now I don’t really need an SSH client on my next phone–this interface is faster, and if I really, really, need a shell, I can wing it with the non-interactive CGI-based Command Shell module in Webmin. The neat thing is that this will also work in Usermin, so we’ll have a kick ass webmail client that is fast and clean in our mobile devices. I’m glad I’ve been thrown in with a bunch of mobile guys lately. It’s an eye-opening experience.

As I mentioned in yesterday’s post Webmin gets even cooler when you have more than one system running it. The BIND module can automatically create slave zones. Virtualmin can automatically setup a hold and relay backup mail server. There are a dozen or so cluster category modules that allow management of users, groups, packages, etc. across any number of servers. And, to continue our theme from yesterday, you can have your servers keep an eye on each other to make sure things are running smoothly.

Because one of the things that could cause your web server to go off-line is an actual system crash, hardware failure, or network outage, it makes sense to run status checks of important services remotely.

So, let’s setup a couple of servers in Webmin and create a couple of monitors.

First up, install Webmin on both of the systems you want monitor (and perform monitoring on). Then login to Webmin and browse to the Webmin category, and click on “Webmin Servers Index”. Click “Register a new server”, and fill in the form with the hostname, port, and OS type. Then select “Login via Webmin with username” for the link type and fill in the username and password for a Webmin user on the destination server (root will work, but you can also make use of Webmin’s users, groups and access control list capabilities to create a special user just for service monitor purposes). Save it.

Now, in the System and Server Status module, you can add any kind of monitor that you can add locally for execution on the remote machine, and you can make use of some of the remote service check types (which doesn’t actually require you to have the system to be tested setup with Webmin). Webmin offers a simple “Alive System” check, that confirms that the system (and Webmin) are running, so it’s a nice one to have setup for each of your servers.

I’ve been hanging around with a lot of mobile application developers lately, and so I’ve been thinking about the value folks get from services being available all the time. And, over the years, I’ve been asked many times how to get a page or an email sent to a cell phone when something happens on a server (where “something” can be “the web server crashed”, “disk space is running low”, “I got an email from my boss”, or “we just started getting referals from reddit.com in the access_log”).

The great thing about this question, unlike some questions (like “how do I migrate my application to a new server with zero downtime”), is that I already have a good answer and it’s pretty simple to explain (the “how do I migrate” question also has a good answer, but it’s not simple to explain, even if you’re using Virtualmin and everything can be bundled into a Virtualmin virtual server backup). The simple answer, in it’s shortest form, is “Use Webmin’s System and Server Status module.”

But, people never seem to believe that Webmin has a lock on best of breed solutions to so many problems…so they’ve been trying to use a bunch of standalone status and monitoring tools and gotten frustrated by how difficult they are to install, configure, or customize. While there are standalone products out there, and some of them are even pretty good, the Webmin option is dead simple to install, crazy simple to configure, and does probably all you’ll ever need.

So, install Webmin, if you don’t already have it on all of your servers. Go ahead and do that now. I’ll wait. If you have Virtualmin Professional, you can skip this step. It includes all of Webmin (hiding under the “Webmin” tab in the left menu).

Ok, so you’ve got it on all of your servers now, right? Wrong? Seriously, go ahead, install it on all of them. You don’t need to “try it” on one. You’re gonna love it and it’s free, and it gets more powerful the more servers you run it on. We’ll get to the “more servers, more power” thing in another post.

Now, login to Webmin (port 10000 in your web browser…something like http://mydomain.tld:10000, or https://mydomain.tld:10000, if you have the Perl Net:SSLeay module) and head on over the Others category. Click on “System and Server Status module.

There’s a bunch of monitors already setup, by default:

Yours will look different depending on the theme you’re using. All of the existing monitors are popular services type status monitors–the green check means the test resulted in a “true” result (which, depending on the test could be all kinds of things..in this screenshot, they’re mostly “is this service running?”). The red X means the test returned “false”, and the red box with an arrow pointing up means the test is “not available” (which could occur if the service is not even installed, or Webmin doesn’t know where to find the configuration and PID for the service).

Creating service status monitors is so easy, I think we’ll skip right ahead to something interesting. reddit is the coolest user-generated news site around, so if you starting finding reddit in the referer string on your server, your business might be picking up (and who knows, maybe the media will start calling to find out all about your new Web 2.0 AJAX-enabled frobnoxilator). So let’s setup a monitor to let us know about it the moment it happens.

So select “Execute Command” from the dropdown list. Enter a description (this will be in the title of all status reports via email). I called this monitor “Reddited!” to convey the necessary excitement. In the check on schedule field, select “Yes, and report on status changes”, so it’ll tell us when we start getting links from Reddit and after that we’ll only hear about it when we go a whole day without getting linked from Reddit. Finally, in the “Command to check exit status of” field, enter something like:

grep reddit /var/log/httpd/access_log

Where the log path should be replaced with the correct path to the relevant Apache access_log.

Save it. Now, unless you’re already getting hits from Reddit, you should see a little red X beside the Reddited entry. Now, get back to work to do something or write something cool enough to get your site on Reddit, so you can turn it into a green check!

Ok, now that you’ve done something cool to test your Reddited! status check, let’s set it up to send you a message. Click on the “Scheduled Monitoring” button. Set “Scheduled checking enabled?” to yes. Decide whether the default “Check every” periodicity is frequent enough for you (five minutes is the default, and that makes me happy, but maybe you wanna know within two minutes). Set the “Email status report to” to your email address, preferably the one on your phone so you’ll always be on top of things, even when skiing in the Swiss Alps with a couple of beautiful blond Scandinavians. Save it.

It can also use a pager to notify you of changes, if that’s the sort of thing you enjoy…but everybody these days has email-enabled phones, so let’s stick with the high tech route.

All done! Now have a look through the rest of the types of monitors that can be setup–there’s dozens of them. And if there isn’t already a perfect match, you can do what we’ve done here, and write your own command to do the job. Just write it such that it returns true when things are going right, so you get a green check when good things happen. This could also be used to call an arbitrarily complex shell script, perl/python/php/ruby script, or any other command on the system. (Don’t forget that grep can return false except when something isn’t found by adding “-v” to the command line.)

For more on this module, see the brief coverage in chapter 15 of the Book of Webmin (my book), and the much more comprehensive chapter 28 of Managing Linux Systems with Webmin (Jamie’s book). The latter is available in PDF format only at this time.

Edit (March 13): Webmin 1.332 and above has added an “email address” converter for folks who don’t know their mobile SMS email address. It knows about most of the major providers, so you just punch in your phone number and select the provider, and Webmin will be able to send you instant messages.

For the past couple of weeks, I’ve been focusing on Webmin/Virtualmin, and how much nicer they could be with an AJAX face-lift. Well, we gave a demo of the product before a bunch of really smart folks last night (many of whom had already seen the ongoing UI work on Virtualmin), so I wanted to have something new to show. So Sunday night I hunkered down and began the port of the first AJAX-enabled Webmin theme to Usermin. By the time of the demo the next evening it was ready to roll (or, at least, ready for a demo in which I could carefully navigate around all of the new bugs introduced by all of the new JavaScript code). The funny thing is that I believe the theme in Usermin is somehow more immediately “cool!” than the much harder project of getting it working for all of Webmin and Virtualmin (which are still under way, and won’t wrap up for another week or two).

Webmail is one of those things that has become a fact of life for nearly everybody. Yahoo and Google raised the bar with their next generation AJAX webmail products, and we’re not quite there with Usermin. But the new theme makes it more obvious how great the core mail reader in Usermin really is. With merely a pretty and AJAX-ified face, it has suddenly become the best Open Source webmail available (possibly excepting RoundCube, which is admittedly awesome–but give me another week or two…I only have to write the front-end to an already robust and complete mail client, while they’re writing everything from scratch). Sometimes, even I’m surprised by how much amazingly useful code is in Webmin and Usermin. Jamie’s like a damned machine, just churning out thousands of lines of great software every year.

And, sure enough, at our demo, the mail client drew quite a bit of the attention. Everyone understands mail clients. It’s like the mythical bike shed. Maybe when all of this is done, we’ll roll up a special version of Usermin that is focused on webmail. I think that’s probably why it isn’t used more. Nobody knows to look at Usermin for webmail…so they end up with SquirrelMail, which is fine, but there are several better products out their to satisfy the need. (Horde also has an image problem…it has a great mail client, too, but it’s far less popular than Squirrel.)

I hate yum. I hate apt-get. I hate urpmi. I really hate up2date and yast.

It’s blasphemous to say, but sometimes it’s the only possible emotion. Ordinarily, these tools are a miracle of modern software. It’d be nice if all of the distros could agree on the same miracle…yum for RPM-based systems, please, and apt-get will have to do for Debs, but that’s not my complaint today. My complaint today is with the fact that it is damned near impossible to convince any of these tools to install all dependencies with a single command and without asking any questions.

I know what you’re thinking. You’re thinking, “That’s their job. Of course they can do it, and I use it for just that purpose all the time.” But you’re wrong. And I know you’re wrong because I use them for that purpose and none of them work for some selections of packages, even when an acceptable resolution is available.

Here’s an example that applies to Fedora Core:

yum -y install caching-nameserver

When your BIND is not up to date. yum fetches the caching-nameserver package, sees that it needs a newer version of BIND, and bails with a fatal error. Did it bail because the newer version of BIND wasn’t available? No, it bailed because it wanted to bail. How about that “–install-all-of-the-damned-deps-I-really-effing-mean-it” option? It’d come in real handy right about now.

Just so you Debian and Ubuntu users don’t get all uppity and think you’re better than the RPM-based systems, try to non-interactively install a package that depends on postfix. It fails, despite postfix being available in the repository. The reason, I assume, is because exim is installed and exim is the super-exalted MTA on Debian and removing it non-interactively to resolve a dependency is not acceptable. Speaking of installing packages with apt-get non-interactively, do you know what kind of hoops you have to jump through to have it not stop and ask questions? apt-get not only needs an “–install-all-of-the-damned-deps-I-really-effing-mean-it” option, it also needs a “–you-better-not-ask-me-any-damned-question-about-it-either” option.

Just in case you think I’m overstating the problem, here’s how you install using apt-get without user interaction:

apt-get –config-file apt.conf.noninteractive -y –force-yes install

Holy crap! We need a “-y” and a “–force-yes”? Yep. Now, what’s in apt.conf.noninteractive?

APT::Get::Assume-Yes “true”;
APT::Get::Show-Upgraded “true”;
APT::Quiet “true”;
DPkg::Options {“–force-confmiss”;”–force-confold”};
DPkg::Pre-Install-Pkgs {“/usr/sbin/dpkg-preconfigure –apt”;};
Dir::Etc::SourceList “/etc/apt/sources.list”;

Holy crap, redux! Are you serious? There isn’t a one-letter option to get a quiet and non-interactive install with apt-get? I don’t know about you, but where I come from noninteractive is the default behavior of UNIX command line applications, and with good reason. Output is for errors. If you did what I wanted you to, I don’t need to hear anything about it.

Of course, apt-get isn’t the only guilty package manager, it’s just the most egregious example. yum has become more chatty, by default, in every major revision…it seems the technical users who speak up on mailing lists always want to see more output flying by, and developers always want to give it to them. It just confuses my users to see so much crap, so I’m filtering down the output as aggressively as I can in my scripts, as we speak. My users still get confused a lot, and so do I, particularly by the various dependency problems that pop up. The worst part is that these problems are extremely unpredictable; our install script might work one day and fail the next on the exact same system, even if no changes have been made to the system, because a new package rolled into the OS repository has different dependencies than the currently installed version. Actually, the worst part is that users either over-react to all of the output, believing it all to be errors, or they under-react, and watch serious errors go by without giving them a second thought. Oh, yeah, it also makes scripting with these tools very difficult.

I should make clear that we’re not dealing with broken or unavailable packages here (even though apt-get wrongly says that the postfix package is broken when you try to install a package that depends on postfix when exim is installed). We’re talking about OS-provided packages. They install without error when installed separately or via an update. But, without fail, every new OS and package manager that I work with has some quirk that prevents it from installing all of the packages we need with a single command. So, I have to have a two-stage process for all of them: install the deps, and then install the stuff that depends on them. yast is the worst, while yum is probably the best. But they all have issues that would probably be solved if the simply started thinking in terms of “what can we do to accomplish what the user asked of us without having to ask any questions?” It doesn’t even have to be the default, but if the user went to the trouble to stick a “-y” in there, take it as a given that they really wanted the dependencies installed without question or error if they are available.

And another thing, while I’m bashing package manager developers as a single entity. They’re kinda ornery about bugs. Sure, I’m all-around ornery most of the time, and you might assume that they’re just responding in kind (as would be their right), but when I file bugs I’m as sweet as sugar. I’m a developer myself, and I know that bugs are a difficult part of the process. I try to keep my bug reports friendlym concise, accurate, and I try to include all relevant details. The result has been less than stellar in almost every single case. Jeff Johnson, the RPM maintainer (which is not, strictly, in the same class as the other high level tools I’m speaking of, but I figured I’d pick on him anyway as it is trendy to do so lately), is renowned for surly responses to bugs, and I’ve tasted it more than a few times over the seven years I’ve been packaging RPMs. But I actually prefer Jeff’s surly but usually prompt response to what I’ve gotten from SUSE and Red Hat on rug and up2date bugs, which is effectively no response. SUSE closed a bug twice as a duplicate of a wholly unrelated bug, while Red Hat has had my up2date bugs open for over a year, one of which has no response at all in that time. Seth Vidal is a notable exception, as he has been extremely responsive and effective in dealing with yum bug reports. It’s no coincidence that yum is, by far, my favorite package management tool. When the system default package manager is just too broken to use (up2date and rug, at this time) I install yum to manage our packages. I don’t like the extra work, but they give me no choice.

And that’s all I have to say about that. I reckon I’m all bitched out for the day. I’ll feel much better tomorrow and I’ll be singing the praises of yum and apt-get from the highest rooftops. But you’ll never get me to say nice things about the package management features in yast–they’ve rightly taken those out and shot them. Unfortunately, they’ve replaced them with rug instead of yum, and introduced a whole new slew of problems.