Obscene Art

So, I like Ron Paul. His no-nonsense, uncompromising dedication to the Constitutional principles our nation was founded on is exactly what America needs today. I don’t agree with him on abortion or immigration, but we can fight that one out after he’s elected. In the meantime, I’ve just dropped another $500 into the campaign, which is 5x what I’ve been contributing each week since the first money bomb. I’ve quintupled down on Operation Live Free Or Die, as well, giving $25 instead of the “Five for Freedom” the organizers have asked for. I’m not quite able to quintuple down on the blimp, but I’m going to drop another $50 there to keep it flying through the New Hampshire primaries. Good thing my business is doing OK this month.

Just a short post today to celebrate Webmin’s 10th birthday! I managed to corner Jamie and force him to answer a few questions for this interview at In the Box (In the Box is the official Webmin blog), and we’re running a logo contest over on SitePoint which is going swimmingly…it’s the most popular contest running right now.

I think it’s pretty cool that Webmin is going to cross the 9 million download (from SourceForge…hard to say how many downloads from other sources) mark during Webmin’s birthday month.

I started using Webmin over 8 years ago, in an early pre-1.0 version, and it’s saved me so much time and effort over the years…it’s a real staple of the system administrator diet, along with: OpenSSH (which turned 9 last month!), vim, lsof, strace, and lots more. Open Source has really had an impact on the systems management market–perhaps more in that field than any other–and Webmin is an important part of that, and I’m excited to be a part of it.

So I bought a wireless web cam so I can keep an eye on my dog when I’m out of the house. It’s a cute little Panasonic BL-C20A. Reasonably good reviews and a reasonably good price. Problem is, it has some stupid Windows only automatic setup program. Argh. The camera itself works fine with Linux (reportedly) or Mac OS X, but you need Windows and ActiveX to install it in the only documented way. I usually don’t buy any hardware that doesn’t work out of the box with Linux (limits my hardware choices, but saves me lots of hassle). But for reasons I won’t go into, this was the only good option, so I bought it against my better judgment.

Anyway, on to the fun. The camera is completely configurable without Windows, if you’re even slightly computer savvy.

Here’s what you do:

Plug it into your DHCP enabled router. Or a switch if you have a server providing DHCP. I think you need DHCP for this, and I don’t think it knows how to fall back to a static IP address in the event DHCP fails. But maybe I’m wrong.

Turn it on.

Give it a few seconds to find an IP.

Now, run a ping scan using nmap:

$ nmap -sP 192.168.1.0/24

Where 192.168.1.0 is replaced with your local network address. Find the one that isn’t one of your existing machines (hopefully you don’t have too many). Hit that address with any browser. It’ll ask you to set an admin password, and from there you can configure it just like our Windows brethren. Done! I can’t imagine why they had to write a big cumbersome ActiveX encumbered Windows application just for that, but that’s the way it is. The web-based UI seems to be browser agnostic once you’ve got it running.

It’s worth noting that the IP might also show up in your DHCP leases list…but it didn’t in my Ruckus Metroflex, so it seems to not handle DHCP correctly (or the Ruckus doesn’t…but all other devices show up in leases).

Being the undisputed master of all things IT, I get a lot of questions about security. It seems that a lot of people are worried about various nefarious hacker types (hereinafter referred to as “bad monkeys”) breaking into their system and doing dastardly deeds, like sending spam or deleting all of those funny pictures of cats fixing servers.

Because my knowledge of IT and system security is so vast, and my understanding of the dark and twisted minds of bad monkeys is so deep, I could write a book on the subject of computer security. I won’t. But I sure could, you bet yer sweet bippy, I could.

Instead, I’m going to provide for you, absolutely free of charge, a timeless and unbeatable security plan in the form of a bulleted list:

• Update your system and all software that runs on it, every day or so. This is easy on a good OS that has good package management, like apt-get or yum. You can even set it to happen automatically. If you don’t have a well-supported OS that is easy to update, replacing it with one that is would be step one.
• Turn off unneeded services. If you aren’t sure what it does, find out. Google knows. If you don’t use it, turn it off.
• Use strong passwords. Your birthday is not a strong password. Your name is not a strong password. Your dog’s name is not a strong password. Any one of the passwords on this list, or this one, or this one, are not strong passwords. A strong password is one that is eight or more characters in length, has letters and either numbers or symbols or both, and is not based on a dictionary word.

That’s it. You’re done. Your system is secure, forever and ever, amen. Now you don’t need to ask me about securing your server anymore. As long as you write secure code to run on it, you can rest easy knowing that the bad monkeys will be flinging their poop around on some other poor slobs server.

We went grocery shopping last night. When we got in the car, there was the usual brief discussion of which grocery to go to. There are five groceries in short driving distance of the house, but only three of them get regular business from us: the Milk Pail Market, Whole Foods Market, and Albertsons. Last night we went to Albertsons for the staples of milk, loaf bread, a few frozen foods, peanut butter, etc. and then drove back across town for the Milk Pail. This got me thinking about customer loyalty, and what takes us to one side of Mountain View for Albertsons, despite there being a Safeway a block away from the Milk Pail.

The Milk Pail is pretty much a regular weekly visit for us. The fresh cheeses, breads, and crazy cheap and fantastically fresh produce is a killer feature. Even the cramped and crowded quarters can’t deter me from this place. So, the Milk Pail earns our loyalty through great prices, fast service, and really high quality. They don’t carry even half of the items that we buy each week, so it is definitely not convenience that takes us to the Milk Pail, but we go every week.

Albertsons and Safeway are roughly identical in almost every way. They carry roughly the same mass-market goods. Produce is pricey and of marginal quality. Both have a wider selection and better prices than the Whole Foods. Both have a “customer loyalty” program that involves annoying applications and little plastic cards that one has to carry around. But, and this is the distinction that brings us back to Albertsons and causes us to shun Safeway, Albertsons doesn’t make you have a “loyalty” card in order to get the “loyal customer” prices. If you don’t have a card, the cashier will swipe their own. These customer loyalty cards must be good for somebody somewhere, but I never apply for them, and every time I happen to stop off at the Safeway, and I see a special price for cardholders I feel a small measure of hostility towards the entirety of the Safeway organization. Since I don’t enjoy feeling hostile, I don’t shop there unless it’s on my way somewhere and I must have something in a hurry. Instead I’ll drive an extra couple of miles and go to Albertsons.

Seth Godin, in his book Permission Marketing seems to come down on the side of these kinds of programs, though he seems to think they aren’t being used effectively. And it’s hard for me to argue with a well-known expert on marketing, but I know how I feel about it, and I can say with absolute confidence that there is almost no “value” that a grocery store could offer that would make me want to fill out their applications and fill up my wallet with their stupid little plastic cards. I believe they’re a bad idea on multiple levels.

It’s Inefficient

Since I generally pay for groceries with a credit or debit card, they already know what I’m buying and can track my usage patterns as deeply as they like. So, it’s inefficient to have a whole other data gathering mechanism…they already have almost all of the data, and they shouldn’t inconvenience their customer to gather it a second time. Of course, they’re also gathering my home address in this process, but that’s for their benefit, not mine. Sure, they could send me coupons for the things I buy, as Seth suggests, or for things related to those I buy. But they can already do that, and many groceries do, at the register as I’m checking out. It’s also an inefficient use of the cashiers time and mine in the checkout line. Sure, it’s only thirty seconds, but it’s thirty seconds of at least two peoples time that didn’t have to be wasted. One of those people is me, and I hate it when people waste my time.

It’s hostile to new customers

When we first moved to Mountain View, our first grocery expedition was to Safeway. We were harried from the move, but needed a few things to help us get settled in, and we’d seen the Safeway when coming in. We picked up about $50 worth of items, and checked out. We were exhausted, and definitely did not feel like filling out an application for a Safeway card. For the privilege of shopping at Safeway without a Safeway card, we paid over $12 more for those groceries! Five months later, I still hold a grudge about that measly 12 bucks. Safeway thus earned our disloyalty through being actively discouraging to new customers–those who might not know yet if they want to be a Safeway customer enough to fill out an application. In our case, they insured that we would never be loyal Safeway customers. Perhaps in other circumstances, the grudge would not have developed, but I doubt in any circumstance would it encourage me to shop at Safeway.

Another way

There’s another way to encourage loyalty in your customers. I’ve hinted at it above, in reference to the Milk Pail, but I believe even in the case of groceries that are going toe to toe with Safeway, there are better ways to turn an occasional customer into a regular customer. Which brings me to HEB. HEB is a Texas-based chain of groceries, probably the largest independent chain in the US. HEB doesn’t have a loyalty card program. In its place it has practices that engender real loyalty. Genuine loyalty for a grocery store is a strange phenomenon to witness, but I’ve seen it (and felt it) for HEB. When the Wufoos were here in town for Under the Radar or some other Web 2.0 gathering Kevin and I talked about grocery stores (now you know what Y Combinator company founders talk about when they get together). In the course of our conversation it was revealed that both our moms have a strong preference for HEB. My mom, having recently moved from Texas to Atlanta, frequently laments the loss of HEB in her life. I thought it was silly until I moved out of Texas and found myself in a world without HEB. I now know what it is to miss a grocery store.

So what makes HEB so damned fancy? While the quality of their produce and selection of both mass-market and organic foods is good, they aren’t a premium grocery like Whole Foods. They don’t have fresh cheeses and breads from local bakers like the Milk Pail. And they don’t have any kind of “loyalty” program at all, like Safeway.

But, they’ve got good prices on everything in the store. Literally everything in the store is competitively priced…even the mighty price-cutting Wal-Mart Supercenters can’t topple an HEB, despite concerted efforts to do so in Texas. I feel like I’m always getting a fair price at HEB.

HEB is fast. When there’s a line of more than one or two people at an HEB, they open another register. Always. Even at 7 PM, when every register was active and the aisles were packed with people, I’ve never had to wait more than five minutes to check out at an HEB. It’s an amazing thing to see. They respect my time, and that’s a powerful thing.

HEB is generous. They offer to carry your groceries out for you, if you have more than a couple of items. Really. Have you had anyone offer to carry out your groceries in the past two decades at any other store? HEB does it every time, and they offer it even if you’re not older or in a wheelchair or otherwise seem like you might need help. Of course, the vast majority of people don’t take them up on the offer, so it costs them almost nothing to offer. But, when my mom had an extra large load, or wasn’t feeling well, or whatever, she knew she would be best served by HEB, because no other grocery would offer to help her out to the car. Of course, any grocery will probably find someone to help you with your groceries if you ask them to, or if you’re obviously not able to do so on your own. But, my mom, like most moms, wouldn’t want to trouble the busy folks at the grocery if there wasn’t the assurance that it was part of their job and they expected to help you out to your car if you needed it. And, of course, no one likes to be identified as uniquely incapable of doing something for themselves…at HEB, it’s not special treatment, it’s part of the package, and when you need it you simply say, “Yes”, to one question. In short, I believe this one standard practice at HEB is better for loyalty than a million coupons or plastic cards. I appreciate it, and I’ve never needed help. Who knows when I might.

Finally, HEB is friendly and personal. I don’t know what their hiring practices are that leads to such high caliber people, but I’ve never been dissatisfied with an interaction with an employee at HEB. They’re universally friendly and helpful. It may simply be that they keep enough people on the floor to keep things running smoothly which helps employees not feel stressed or overworked, or it may be that the people are better trained (which may explain how HEB seems to magically open a new register when more are needed, without any intercom noise begging someone to come to the front), or it may be that they pay their employees more than most groceries, or have better in-store management, or some combination of those things. HEB is also the only grocery that hasn’t drunk the Kool-aid on self-checkout (though I believe some of the HEB stores offer it, they don’t do so at the expense of not having enough human cashiers, unlike Wal-Mart). Whatever it is, shopping at HEB is pleasant.

Of course, they do all of the other stuff right, too. Aisles are wide and comfortable. The stores are clean and well-lit. The shelves are generally well-stocked and neat. Etc. But many groceries meet those qualifications, including Safeway and Wal-Mart…and I’ve never heard anyone exclaim their dedication to a Safeway or Wal-Mart, and I can name off half a dozen people who deeply appreciate HEB.

Applying the HEB model to other industries

Now, I’m not in the grocery business. In fact, my business doesn’t have a physical presence, at all. But when a business as boring as groceries can instill the kind of loyalty that HEB customers have, it’s worth paying attention to.

So, what are the important elements? Good value, don’t make your customer wait, no artificial barriers to becoming loyal (e.g. don’t make someone fill out an application and sign a loyalty oath before you treat them like a good customer…give them the benefit of the doubt), always offer to go above the call of duty (especially when you know most customers won’t take you up on it).

We’re already doing a lot of these things with our product and website, but I think we can do better.

A “fair price” for any non-commodity item is whatever price customers will pay and feel like they’ve come out ahead. This means “ahead of where I’d be without any product of this kind” as well as “ahead of where I’d be if I’d chosen a similar competing product”. So, in our case, we have to provide better value or a better product than cPanel and Plesk, or do both. We’re solid on both counts, though I believe we can do better at explaining to people why we’re building a better product.

Not making your customer wait has a lot of implications in the software business. The software can’t be slow. The website where ordering and support take place can’t be slow. Delivery of purchased digital goods should be immediate. Support query response time should be fast. These are hard problems, and almost every business could do better. In our case, we have some sluggish components in our product (not many left, but still a few remain), our website is pretty sluggish (to be mostly fixed by our upcoming migration to a new CMS), we already make purchased products (except human services) available immediately, and finally, support queries are sometimes handled within a few hours and usually not more than 24. We’d like to improve on that last number, but I’m not sure it’s possible with only two people. I think the new website will help us on that count, as well…a wiki for documentation will allow us to better translate support interactions into better coverage in the documentation, and a better issue tracker will help us involve some of our more educated users in the support process (in exchange for fabulous prizes, like T-shirts and free software and such).

Avoiding artificial barriers to becoming a customer is a hard problem, not made easier by most of the pre-written software out there to handle selling and communicating with customers online. Our current website is running on OpenACS, and it’s actually pretty good about staying out of the way, but it still has some stupidity that is inexcusable. One of the major projects in our Joomla migration has been to make it less demanding of the end user. By default, it’s got a lot of features that really get in the way of simply using the applications, and it’s been a long hard slog to get rid of most of those nuisances. User registration should ask for only two things: Email address and a password. A username my be a valid extra third field, if it’ll be displayed publicly in forums or wikis or whatever. In our case we’ve opted to have usernames. But we don’t ask for anything else to sign up. If the user wants to give us more data, or give us billing information in order to buy the product, we get to that later. Posting to the forums, using the bug tracker, wiki, etc. can all be done without giving up more than those bits of data.

Being friendly and personal. Well, that’s just how you interact with people. I’d suggest not hiding behind generic email aliases in order to seem bigger. We send out support emails with our name attached from our personal mailbox. When we talk about things in the bug tracker and forums, we use first names, and treat it all like a friendly conversation…both for ourselves and our customers. Sure, people now know that we’re a two man shop, and there’s not a suit in sight. So, what? If we’re kicking cPanel and Plesk’s ass with two guys imagine what we’ll do when there’s five of us this time next year? I think being genuine is a net win.

Going above the call of duty is the hardest to define. What is expected, and what is cost-effective to offer above that? A full refund policy is a given. If you don’t have a good refund policy, you’re losing sales and reducing customer loyalty. You’re also probably hurting employee morale. Nobody likes to have to tell a customer there’s nothing they can do, so don’t put anyone in that position. With software, support is the expensive thing, and so it’s hard to make a practice of offering it on top of the product itself at no extra cost. In our case, we’re doing it anyway. I’m not sure it is a sustainable practice, but during our open beta we’re viewing it as a development expense. The customer is our second tier of QC. When we find that the vast majority of support queries are, in fact, not problems with the software or the documentation and instead are a failure on the part of the customer to make a modest effort with the documentation first, that’s when we’ll start rethinking the policy. We’re getting closer to the point: we have a few problem customers that take far more than their fair share of support resources with questions that are well-served by the documentation and online help. The primary way we can tell is when they have questions that the other 99.8% (1 out of roughly 500 commercial users) of our users have managed to figure out on their own. If it’s a rarely used or new feature, we’ll instead assume it is poorly documented, has a bad UI, or simply isn’t working right. In all cases, as long as we’re calling the software beta, we’re giving unlimited support at no extra cost.

What else could we do to go above the call? We’ve got an Open Source version. Free stuff is a pretty good thing…but it doesn’t directly help our paying customers, except to build a wider pool of people who have expertise in our product (so if they need custom development, third party plugins, etc., the odds of finding it are better). We’ve got an open bug tracker so anyone can file a bug. That’s pretty valuable, though not a lot of non-technical users are familiar with the process. We’re launching a wiki for our documentation, so users can tell us what’s wrong with it and help fix it.

What are you doing to go above the call and engender real loyalty?

The California Gold Rush holds many parallels to our own current second Internet boom. More technology businesses are being started now than at any time in history, including during the first boom. Far fewer of them will go public, and far fewer of them will blow tens or hundreds of millions in VC (at least, I hope so). But, there are some similarities between this new boom and the previous boom and the Gold Rush of 1849.

A lot of people in all three instances struck it rich by tapping a rich vein…gold in the 1800’s, and huge user base and network effects in 2000 and today. But, and this is a big but, even more people barely broke even. If you don’t strike just the right balance of luck, hard work, and talent, you may end up holding an empty bag. Likewise, some walked a different path, and chose to serve those looking to find gold. Levi Strauss and Leland Stanford both began building their fortunes during the gold rush, but they did it not by searching for gold, but by selling tools, clothing, and supplies to the folks looking for gold. Yes, they had to work longer than the folks who actually found gold. But their odds of success were significantly higher. If you’re surrounded by people who need what you’re selling, and the alternative for your customers is to give up and go home or delay and miss out on a strike, it’s a safe bet you’re going to move a lot of product.

During the first boom, Sun and Oracle made more money than just about anyone by providing tools to the folks building businesses. They weren’t doing anything interesting on the Internet. They weren’t building websites that benefited from network effects, and it didn’t matter to them if millions of users chose their product or service every day. They only needed thousands of users…but they needed those thousands to pay a lot for their goods and services. Paul Graham talked about this in his An Alternative Theory of Unions, and it’s true that when time is of the essence, and staking a claim to your territory is a vital element of success, you’ll pay more to avoid delay.

Of course, Sun and Oracle weren’t the only businesses to make their fortunes providing tools for the first boom, and they aren’t going to be the ones making the money on this boom. The landscape is different, and the success stories are going to be in different areas. Hardware is cheaper and faster, entirely a commodity. Dell will make plenty on servers, but nothing like Sun’s astronomical profits. Databases, too, are a commodity. Every Web 2.0 service is using MySQL or PostgreSQL for their database needs.

We’re placing our bets on systems management. It’s boring, just like servers and databases. It’s necessary, just like servers and databases. And it’s not a commodity (except where we’ve made it a commodity with our own Open Source software). In fact, it’s surprising how weak server management is for web servers. We’re not the only folks placing our bets this way, of course, and it’s a rather big field. Analytics have been hot, because it’s easy to build technically and there’s been big upside for a few of the analytics providers (Urchin sold to Google, NetIQ bought WebPosition from FirstPlace, Instadia sold to Omniture, etc.), so it’s not surprising to see Mint, and dozens of others. Likewise for web service monitoring, with lots of new companies approaching the problem in new ways (we even have some plans in that direction, but only as an ancillary to our core product line). These are both products that tap into a desire that I believe is deeply ingrained into almost everyone, but managers tend to have almost no other desire: Measurement of success. I think of it as the “Ooh, look at the pretty graphs!” factor.

So, while our Open Source project Webmin is a wide-ranging general purpose web-based system administration tool (the most popular in the world with ~2 million downloads per year), our commercial products are tightly focused on web servers and the tools people building the next generation of web services need. So, we don’t get to enjoy the excitement of our first “million page view” day…we do get to live vicariously through our customers successes. It’s exciting to talk to Evan Williams about how he uses our software at Twitter. Nobody knows about it, since it’s on the back end. But it’s cool, nonetheless.

So, maybe if your consumer web business hasn’t taken off as well as you’d hoped, maybe you can refocus on making it a success as a tool for enabling others who are building businesses. Or, maybe not, and it’s time to get back to panning for gold. But it’s worth considering the alternatives to building a business whose only chance of success is convincing millions of people to use it every single day.

So, over at the Virtualmin forums I get a lot of questions about which operating system to use. Partly, they’re asking, “What does Virtualmin support?”, but they’re also looking for guidance on what will be most productive and least problematic for the tasks they want to perform with the system. Given that I’m building server management software, and the majority of the systems I deal with are web servers, the following advice pretty much applies exclusively to servers. Based on my past ten or so years of heavy Linux usage on servers, I’m comfortable calling myself an expert on OS selection and deployment, so disregard the advice at your own peril.

The good news is that I’m not going to tell you that any particular Linux distribution sucks for servers. Some of them do, but I won’t spend a lot of time naming names, except where there is a clear and objective measure that clearly makes some distros better than others. I’d much rather cover what I’ve found makes an OS great for server usage and what makes one painful.

Which brings us to the very first and single most important factor in your OS decision.

Support Life Cycle

Servers remain in service far longer than desktop machines. Migrating from one to another is painful and time-consuming (even with tools like Virtualmin providing a really clean migration path, you still have the DNS and temporary loss of service issues to contend with), so you want to do it as rarely as possible. But, you cannot safely run a server without regular security updates. You just can’t. Even if you carefully choose historically secure packages for your services (Postfix or qmail for MTA, OpenSSH for file transfer and shell access, Webmin for web-based administration, Dovecot for POP/IMAP, etc.) and carefully shut down every service you won’t be needing or firewall it down to just the hosts that need to talk to them, some security problem will pop up in the future. If you can’t easily upgrade that service within hours of the vulnerability being discovered, your system will be compromised, and you’ll be forced into a rapid migration to a new system, or trying to find a security consultant who can clean up the problem and paying the very high price that said consultants charge.

So, pick an OS with a long life cycle, and choose the most recent version of that OS. Debian has a very long life cycle. RHEL and CentOS have a very long life cycle. Mandriva and SUSE have a long life cycle (but I believe you have to get the paid versions to get the long-term support). Ubuntu LTS also has a long life cycle, but non-LTS releases do not, so you’d want to avoid any version other than LTS. Fedora, now that Fedora Legacy is gone, has a very short life cycle, and should be avoided for servers. I’m unable to find the life cycle of Gentoo.

I’m not kidding about life cycle. If you pay no attention to anything else in this article, pay attention to this advice. I’ve managed hundreds of servers in my life, and the single biggest pain point is updates and upgrade management and migration when updates and upgrades are no longer feasible. Avoid that pain as much as possible!

Package Management

Packages are great. They take away several major pain points in managing servers. So, choose an OS that offers good package management. The definition of “good” in this context is that it, roughly in order of importance: handles dependencies automatically, allows alternate software repositories so you can manage local packages or make use of third party repositories for specialized software, offers access to a wide array of “official” or at least nicely built third-party packaged software, allows you to make your own packages or customize existing ones, allows unattended operation and operation from scripts.

apt-get+dpkg and yum+RPM and urpmi+RPM are the best package managers available today, meeting all of those qualifications. After spending a couple of weeks porting our application and all of its dependencies to Debian 4.0 and CentOS/RHEL 5, I can say that Debian wins on the package management front, due to the sheer number of packages available in the universe repository. I like yum and RPM better from a technical perspective, and Fedora Extras makes it comparable to Debian on the package management front, but we’ve already disqualified Fedora due to stupidly short life cycle.

Ease of Management

Every Linux distribution has quirks that make it harder to use than it should be. In some cases, better ways are known and well understood, but historical behavior sticks around due to inertia and remaining compatible with existing users expectations. That said, there is value in predictability and good documentation. I find CentOS and RHEL to be the most capable out-of-the-box, and with quite sane management methods. Debian is also strong, but has a few historical stupidities, such as lingering reliance on START=yes crap in configuration files for some services. In a world where chkconfig has been around for years, this is foolishness. This is merely an example, and is a minor nuisance in any single instance, but by the time you’ve had to edit a dozen files just to get your services running (even if no configuration is required), it becomes painful.

SUSE has made a habit of changing major components of the system in every single revision. Package management on SUSE has changed in major ways three times in the past three revisions, and it’s still weaker than yum or apt-get. Configuration files move from one version to the next, and many can only be sanely managed with YaST. In fact, SUSE has a horrible habit of designing the underlying system for the convenience of the YaST developers. A specific example, so folks won’t think I’m just being bitchy because we build a management tool that overlaps with YaST in some areas: SUSE uses a firewall save format different from every other distro (which uses the standard iptables save file format). The SUSE format is less powerful, less flexible, more complicated to edit, and incompatible with all other firewall management tools. There’s no reason for this, except that it’s easier for YaST to modify a few variables in a shell script than to actually parse the iptables save file format and deal with it cleanly.

Webmin and Virtualmin nearly nullify the management problem, by abstracting it out to a nearly identical interface across all Operating Systems. But even with Webmin and Virtualmin, there are still differences that can increase the friction of going from zero to “in service”. I also like to know that when I hit the command line, the configuration files and mechanisms are sane and well-documented.

Documentation

Having good documentation is pretty darned important. And by documentation, I refer to every scrap of information that Google can dig up for you on a particular problem, not just the official documentation for the distribution. In this regard, the more popular an operating system for the tasks you’re performing, the more likely you are to find good documentation on the web.

The Debian official documentation is particularly verbose and particularly useless, and Ubuntu only marginally corrects that. The Debian documentation suffers from the same disease as man and info pages in the GNU system: No examples. Both are quite popular on servers, though, so one can often find good examples via Google.

The Red Hat, CentOS and Fedora documentation is much stronger, despite being less voluminous. The Red Hat based systems, are also the most popular, and so examples and discussions on the web abound. I’ve found them to be the easiest to find solutions for. If you’re new to Linux going with the most popular distro might be wise.

Gentoo has shockingly good official documentation with fantastic examples. The distro itself falls down in numerous other categories for server usage (pretty much all of the above, as far as I can tell), but the documentation is simply awesome. Whatever process they’ve used for their documentation ought to be replicated by the other distributions, because despite the youth of the Gentoo platform, it has far better official documentation than any other distribution. Hell, I need to spend some time studying how the Gentoo documentation project is run, so I can replicate it for Virtualmin and Webmin.

SUSE is far more popular on desktops than servers, and so server-related documentation is slimmer than we’d like. It is also less popular than Debian/Ubuntu and the Red Hat based systems, and so finding stuff on the web is less likely. The official documentation is of good quality, but again, there is simply less topic coverage.

Mandriva also has reasonably good official documentation, though it also is more popular on the desktop, and so much of the official and unofficial documentation is focused on things that we, as server administrators, don’t care about.

Your Use Case

This one is more difficult to give advice on, because it depends on what you’re doing with your system. If you’re a .net developer, you might look to SUSE, because Mono is a core part of that system. If you like Python, you might consider Red Hat or CentOS, because Python is an important part of their system and thus the packages are well-cared for in the official repository. Perl is well-supported by all of the distributions (though Mandriva and Debian have the most official packages for various CPAN modules), and Ruby is equally poorly supported by all of them (though Debian 4.0 seems to have improved remarkably on this front). Debian and Ubuntu are particularly strong on PHP, as they provide official packages for both PHP4 and PHP5, and they can co-exist on the same system (it took me weeks to replicate this capability on the Red Hat based systems).

So, what are you doing with your server? Search the web before making your OS decision to see if others are using one particular OS more often for tasks like the ones you’ll be tackling. This is a variation of the old “it’s the applications, stupid” rule. If all of your necessary applications are being used heavily on a particular OS, like maybe by the developers themselves, you’d be well-advised to pay attention to that.

In our case, we are heavily invested in Perl development, so our options are pretty wide open. We also use Joomla for our new in-development website, so we need reliable PHP support. Another, particularly difficult, piece of our puzzle is that we need a virtualized server to run our demo on, so native support for virtualization would be nice. We currently reside on a Red Hat Enterprise Linux 4 server, but if I were choosing today, I’d probably select Debian 4.0 on the strength of the universe repository. But, that’s just me.

Y’all pick your favorites, and let me know what criteria you use, but try to avoid “I like it because it’s easier to X”, because that intangible quality is almost always a function of what you’re familiar with…I’ve tried to steer clear of familiarity as a basis for selection, since there are far more folks who don’t have familiarity with any particular Linux distribution than there are that do.

I’ve noticed an interesting trend among our customers…the developers start really small, but buy more over time. Sometimes a lot more. This makes sense, of course, since freelance developers move from project to project and each new project has a new budget and a new server. Our product makes managing a server and the websites on it really, really, incredibly easy and fast, so folks who’ve used it end up using it on every web server they deploy. We’ve had dozens of web developers buy our cheapest offering one month and the next buy two or three of the more expensive versions.

Developers are also more demanding than most customers. They recognize when difficulties they are having are failings of the software rather than failure of their knowledge of the subject matter (generally), and so they file more bugs and ask harder questions. So far, you’re probably with me that developers are awesome users.

They also have some flaws, as users. They want things that are only useful to other developers. And I’m not just talking about APIs here. APIs are awesome, of course, and we’ve got APIs all over the place, and you should, too. But sometimes there are opportunities for making a product more useful to developers just by the features that you include.

An example in our product is support for private IP addresses, dynamic DNS, and the ability to operate correctly behind a NAT firewall. This is actually quite a bit of code to support, and I was reluctant to add it. Our product already has too many features and too many options (flexibility can be a curse…but it’s also nice to always be able to answer “yes” to the question “does you product support X?”, so it’s a wash), so adding a bunch of stuff that’s only useful to web developers on their development servers in their home office seemed maybe a little crazy.

But, we did it anyway (OK, Jamie did it without really consulting with me on it, because his thought process on just about any new feature is, “yeah, I can do that” and then he does it that evening and launches it the next day). And, it turns out I was wrong. It’s worth the effort, the additional support expense, and the added options, because developers and IT are the key to customers for any technology product. If the nerd working on a project likes you and finds what you do valuable, they will recommend it at every opportunity within their company, or on every project they work on if they are contractors. If you save the IT department, or contractor, time and repetitive effort, you will have the whole company using your product in short order.

From here on out, when one of our developer customers has a request I’m going to think long and hard before dismissing it as too much of a niche feature. If I can visualize a large number of web developers across the world breathing a sigh of relief when we roll out such a feature, then we will roll it out. Our primary customer will always be web hosting providers, the bigger the better, and their usually non-technical end users. But, if we piss off the developers who actually build the websites, I’m certain we’d regret it.

So, what’s the punchline? What can you do about it? Well, if you’re building a web service that’s useful to a large segment of people, you may have built something that can save the IT departments in major companies time and repetitive effort. An obvious example of a fellow Y Combinator company is Wufoo. They make a really nice form builder service. Seems simple, until you realize how many forms could be used in day to day operations of most businesses…got a trip to plan and need to know who’s going? got a software audit coming up and need to know who’s using what products? got T shirts to distribute to customers and need to know where they are and what sizes to send out? Wufoo’s got it covered and you don’t need to bother your IT department about building another stupid form!

Online office productivity software is another obvious example. Saves IT departments tons of time and effort. Getting PowerPoint or Office or whatever for a system that doesn’t have it is a PITA in most companies. File a requisition, wait for an IT to get approval from your manager, wait for IT to license it and install it, find out that it needs more memory or disk space, requisition more memory and disk or new computer, etc. Three weeks later, the project has come and gone and now you’ve got PowerPoint or Office and don’t need it anymore. If you’ve got IT on your side, they will instead say to their user, “Try Zenter, it’ll let you do a nice presentation right away.” Or Google Apps. Or whatever…lots of possibilities there for making the life of IT guys better.

As I mentioned in a previous post, I believe the day is coming in which there will be an upset in the roles traditional media and polling play in elections, where they will be replaced by the Internet. I don’t know if we’ve reached that day yet, but there is a striking disparity between the two sources of data regarding Ron Paul.

Ron Paul’s website is trouncing the other Republican candidates in visitors, despite being a crappy website:

He’s also beating the leading Democratic offerings:

He’s also winning, or coming in second, on many online or SMS polls regarding the Republican debates.

MSNBC’s California post-debate poll

Fox’ Post-Debate Text Poll (following Rudy Giuliani’s 9/11 outburst)

Paul is the top search on Technorati’s blog search:

But, in traditional polls, Paul is consistently fourth or fifth, after Giuliani, Romney, McCain, and sometimes Thompson (I can’t keep up with which Thompson), with single digit percentages. So, where’s the disparity? How can there be such a big difference? Traditional polling uses land line telephone calls to likely voters. I don’t have a land line. I don’t know anyone, other than my parents and my older sister, with a land line. So, a good percentage of people under about 35 are under-represented in these polls. That’s probably OK for accuracy on the lower end of the age scale, since the people least likely to have a land line (those under about 24) don’t vote…so the skew might be good for overall accuracy. But, telephone polls have become less likely to predict outcomes in elections over the past few election cycles. Internet polls have never been accurate, but maybe they’ll become more accurate as more people begin to get all of their news online and identity becomes more solid on the Internet. Interesting that the mainstream sites are far less likely to know how accurate their voting is than sites like Reddit or Digg or Slashdot, simply because readers aren’t involved in the process and so are unknown anonymous readers rather than editors and participants.

Anyway, despite all that we’d like to think about Democracy in America, the fact is that the candidate with the most money to spend generally wins. Paul isn’t bringing the big bucks, since he isn’t in any corporate pockets like the other candidates, so probably couldn’t win against a fat-pocketed corporate shill. His nomination could save the GOP from itself, however, and would certainly go a long way to convince me that it’s worth saving. Hell, I’d be proud to call myself a Republican if Ron Paul were the nominee. Otherwise, I guess I’ll have to vote Democrat…at least they haven’t been in power long enough to gerrymander, commit voter fraud on a wholesale basis, and otherwise use every despicable trick available to them to win elections at any cost.

I’ve been in the planning stages of a new product for Virtualmin users (and eventually for anyone who has a web server), which would predominantly run on on our servers. It would be our first application that allows us to choose the implementation language (our current codebase is over nine years old and includes hundreds of thousands of lines of Perl…we won’t be changing languages for our installable products, though we do look forward to Perl 6).

This got me to thinking about language popularity. PHP is, by far, the most popular language for the web. It’s also the worst among the popular choices. At least it’s the worst by every metric I consider important…except one: Availability.

PHP syntax is horrible. The vast majority of PHP code in the wild is horrible. The PHP standard library is inconsistent from top to bottom. It encourages bad practices, and dangerous behavior is the default for many PHP functions. Only in the past few years has PHP acquired the ability to safely and sanely use databases. The database problem is the cause of a huge percentage of the horrible PHP code in the wild…and folks are still writing code using the ugly way to access databases, despite better ways existing, because it’s all they’ve ever seen. PHP code, in general, is hard to maintain, hard to read, and often poorly documented (again, a lot of the documentation covers historic techniques that are simply dangerous). The object system is bolted on. It has no functional programming constructs to speak of.

So, why the heck is it so pervasive? Why on earth would smart people choose to use a language that is less effective for almost every non-trivial task in web development? And, I should be clear, smart people do choose to use it, and some fantastic applications have been written in it. This isn’t a situation where a bunch of idiots are using it because they don’t know better. Some of the coolest web applications are written in PHP, by some of the coolest developers working in the field.

The answer, and an answer that the Ruby On Rails folks, Pylons folks, Django folks, Perl Catalyst folks, etc. should all pay really close attention to (assuming they want their apps to be widely used), is that PHP is pervasive because it is pervasive. Tautology, I know, but what I’m trying to get at is that people code in PHP because they know it will work wherever they go or whoever they hand it off to. They know every web developer will be familiar with it, and will have access to a webserver that can run it. They know it will run on the cheapest DreamHost account, the midrange TextDrive account, or the most expensive RackSpace managed server, without modification and without jumping through too many hoops. They also know that if they have to hire someone to maintain it, or round up a community of users to help add features, there are no shortage of knowledgeable developers (it’s hard to find the good ones, as in any language, but some things don’t require good developers). Further, it’ll be reasonably fast, if they don’t do anything stupid or pathological with databases or files.

In short, if you want your installable application to be used, probably the single most useful thing you can do is write it in PHP. You also need to make it do something good, of course, and without huge glaring bugs. I’m guessing, but based on UI studies that show that for every field you add to a form, the completion rate drops by a huge amount (an order of magnitude in some extreme cases), I believe that an installable PHP application will initially get ten times the users of a Ruby On Rails application that does the same thing. I don’t have numbers to back up this bold assertion, of course, but I’m pretty sure that’s the way it will play out, unless and until Ruby On Rails is as pervasively available as PHP (which is unlikely to ever happen).

I don’t like this fact. I’m obviously not a fan of PHP as a language. I can’t think of a commonly used web development language that I would choose PHP over from a purely technical standpoint, except Java or C# (and even those have certain areas where they are particularly strong and well-suited for the task). But if I were building a small application (by small, I mean anything under 50,000 lines of code) that I wanted thousands of users to install, I would probably write it in PHP.

Of course, I’m not building an installable application this time…so I’ll be picking among the languages I like (Perl, Haskell, Python, Erlang). But my moments of blinding insight are few and far between, so I thought I’d share this one with the 12 people that visit my blog.