Obscene Art

In a brief bout of uncharacteristic gratitude, I’d like to thank the developers of both OpenACS and Joomla for choosing to use standard tools in the place where it matters most: the backend database.

As I mentioned a couple of weeks ago, I’m migrating Virtualmin.com off of OpenACS, due to a number of limitations and bugs in the platform. After some consideration, research, and tinkering, it appears that Joomla is the best suited for what we’re doing with our site. It has good forums, a good FAQ manager, a decent bug tracker, and a good ecommerce package. All of them, except the bug tracker, look strikingly similar to the OpenACS applications they’ll be replacing…which is an odd, but nice, coincidence.

But, the problem is that Virtualmin.com has been running for about a year on OpenACS. We’ve got a lot of data. Several thousand registered users, hundreds of forum posts, hundreds of bugs, and hundreds of products sold in the shopping cart. We also had a custom software license manager written for OpenACS, which keeps up with the begin and end date of our customers licenses, which has those hundreds of product sales information in its tables.

So, I’ve got to migrate that data. Luckily, OpenACS uses PostgreSQL and Joomla uses MySQL. Both are standard, popular, databases that can be accessed using just about any language. So, I’ve chosen to take this opportunity to learn a little PHP, and write some migration tools to automate the move.

The first result is a quick and dirty FAQ migration script that moves OpenACS FAQ entries over to a Joomla EasyFAQ. It doesn’t handle category creation, since that’s stored in a different table from EasyFAQ, and I didn’t want to learn how categories were created in Joomla, so you have to pre-create all of the categories and then fill in an associative array with the category numbers with OpenACS faq_id in the fields and Joomla catid in the values. Otherwise, it’s quite simple, and seems to work perfectly. Even the markup works after migration! I’ve posted the script since someone else out there might be making the same, or similar, move between two CMS systems or just two different databases and might find this a reasonable starting point. It’s not pretty, at all…it is my first PHP script, after all.

It took all of an afternoon, but I expect when I start working on the other datasets, like users and forums and bugs, it’ll go much faster (most of the time was spent figuring out why my various mysql_insert function attempts weren’t working…I ended up giving up on a generic function and just wrote the query out manually, since my PHP skills just aren’t up to the task of writing a generic insert function capable of handling arrays, yet).

Anyway, the reason I mentioned all of this is because I’ve tried to migrate data out of the Zope/Plone ZODB before…and I wouldn’t wish that task on anyone. Thanks to all those folks out their who understand that, sometimes, it’s OK to not reinvent every single wheel.

Here’s the script: faq-mover.php.gz

I hate yum. I hate apt-get. I hate urpmi. I really hate up2date and yast.

It’s blasphemous to say, but sometimes it’s the only possible emotion. Ordinarily, these tools are a miracle of modern software. It’d be nice if all of the distros could agree on the same miracle…yum for RPM-based systems, please, and apt-get will have to do for Debs, but that’s not my complaint today. My complaint today is with the fact that it is damned near impossible to convince any of these tools to install all dependencies with a single command and without asking any questions.

I know what you’re thinking. You’re thinking, “That’s their job. Of course they can do it, and I use it for just that purpose all the time.” But you’re wrong. And I know you’re wrong because I use them for that purpose and none of them work for some selections of packages, even when an acceptable resolution is available.

Here’s an example that applies to Fedora Core:

yum -y install caching-nameserver

When your BIND is not up to date. yum fetches the caching-nameserver package, sees that it needs a newer version of BIND, and bails with a fatal error. Did it bail because the newer version of BIND wasn’t available? No, it bailed because it wanted to bail. How about that “–install-all-of-the-damned-deps-I-really-effing-mean-it” option? It’d come in real handy right about now.

Just so you Debian and Ubuntu users don’t get all uppity and think you’re better than the RPM-based systems, try to non-interactively install a package that depends on postfix. It fails, despite postfix being available in the repository. The reason, I assume, is because exim is installed and exim is the super-exalted MTA on Debian and removing it non-interactively to resolve a dependency is not acceptable. Speaking of installing packages with apt-get non-interactively, do you know what kind of hoops you have to jump through to have it not stop and ask questions? apt-get not only needs an “–install-all-of-the-damned-deps-I-really-effing-mean-it” option, it also needs a “–you-better-not-ask-me-any-damned-question-about-it-either” option.

Just in case you think I’m overstating the problem, here’s how you install using apt-get without user interaction:

apt-get –config-file apt.conf.noninteractive -y –force-yes install

Holy crap! We need a “-y” and a “–force-yes”? Yep. Now, what’s in apt.conf.noninteractive?

APT::Get::Assume-Yes “true”;
APT::Get::Show-Upgraded “true”;
APT::Quiet “true”;
DPkg::Options {”–force-confmiss”;”–force-confold”};
DPkg::Pre-Install-Pkgs {”/usr/sbin/dpkg-preconfigure –apt”;};
Dir::Etc::SourceList “/etc/apt/sources.list”;

Holy crap, redux! Are you serious? There isn’t a one-letter option to get a quiet and non-interactive install with apt-get? I don’t know about you, but where I come from noninteractive is the default behavior of UNIX command line applications, and with good reason. Output is for errors. If you did what I wanted you to, I don’t need to hear anything about it.

Of course, apt-get isn’t the only guilty package manager, it’s just the most egregious example. yum has become more chatty, by default, in every major revision…it seems the technical users who speak up on mailing lists always want to see more output flying by, and developers always want to give it to them. It just confuses my users to see so much crap, so I’m filtering down the output as aggressively as I can in my scripts, as we speak. My users still get confused a lot, and so do I, particularly by the various dependency problems that pop up. The worst part is that these problems are extremely unpredictable; our install script might work one day and fail the next on the exact same system, even if no changes have been made to the system, because a new package rolled into the OS repository has different dependencies than the currently installed version. Actually, the worst part is that users either over-react to all of the output, believing it all to be errors, or they under-react, and watch serious errors go by without giving them a second thought. Oh, yeah, it also makes scripting with these tools very difficult.

I should make clear that we’re not dealing with broken or unavailable packages here (even though apt-get wrongly says that the postfix package is broken when you try to install a package that depends on postfix when exim is installed). We’re talking about OS-provided packages. They install without error when installed separately or via an update. But, without fail, every new OS and package manager that I work with has some quirk that prevents it from installing all of the packages we need with a single command. So, I have to have a two-stage process for all of them: install the deps, and then install the stuff that depends on them. yast is the worst, while yum is probably the best. But they all have issues that would probably be solved if the simply started thinking in terms of “what can we do to accomplish what the user asked of us without having to ask any questions?” It doesn’t even have to be the default, but if the user went to the trouble to stick a “-y” in there, take it as a given that they really wanted the dependencies installed without question or error if they are available.

And another thing, while I’m bashing package manager developers as a single entity. They’re kinda ornery about bugs. Sure, I’m all-around ornery most of the time, and you might assume that they’re just responding in kind (as would be their right), but when I file bugs I’m as sweet as sugar. I’m a developer myself, and I know that bugs are a difficult part of the process. I try to keep my bug reports friendlym concise, accurate, and I try to include all relevant details. The result has been less than stellar in almost every single case. Jeff Johnson, the RPM maintainer (which is not, strictly, in the same class as the other high level tools I’m speaking of, but I figured I’d pick on him anyway as it is trendy to do so lately), is renowned for surly responses to bugs, and I’ve tasted it more than a few times over the seven years I’ve been packaging RPMs. But I actually prefer Jeff’s surly but usually prompt response to what I’ve gotten from SUSE and Red Hat on rug and up2date bugs, which is effectively no response. SUSE closed a bug twice as a duplicate of a wholly unrelated bug, while Red Hat has had my up2date bugs open for over a year, one of which has no response at all in that time. Seth Vidal is a notable exception, as he has been extremely responsive and effective in dealing with yum bug reports. It’s no coincidence that yum is, by far, my favorite package management tool. When the system default package manager is just too broken to use (up2date and rug, at this time) I install yum to manage our packages. I don’t like the extra work, but they give me no choice.

And that’s all I have to say about that. I reckon I’m all bitched out for the day. I’ll feel much better tomorrow and I’ll be singing the praises of yum and apt-get from the highest rooftops. But you’ll never get me to say nice things about the package management features in yast–they’ve rightly taken those out and shot them. Unfortunately, they’ve replaced them with rug instead of yum, and introduced a whole new slew of problems.

Virtualmin.com runs on a mildly customized version of an Open Source CMS called OpenACS 5.1.5, but it won’t be running on it for much longer. OpenACS is one of the oldest content management systems available. It is written in TCL and runs on top of AOLServer and PostgreSQL.

Some folks are already saying, “Ah! It’s written in TCL and you said the word AOL. No wonder you’re leaving OpenACS!” But that’s not why. I’m predominantly a perl developer working on a large existing codebase…I know that dropping a mountain of working code just to work in a more fashionable language is foolish. I also know that, despite its bad reputation, modern TCL is actually a pretty good scripting language. It’s not as pretty as Python or Ruby (but what is?) and not as powerful as Perl+CPAN (but what is?), but it’s a pretty good mix of features and clean syntax. TCL even has a few cool Lisp-y bits thrown in; which, as we all know, means that it is closer to god or something. More importantly, OpenACS seemingly provides a mountain of working code.

No, the reason I’m leaving OpenACS is because large portions of that existing mountain of code, despite claims to the contrary, don’t work. Worse, the aspects of the code that don’t work are not becoming fewer with each new revision, they are becoming more common and more difficult to fix. Working applications are being deprecated in favor of incomplete, bug-ridden, and poorly documented rewrites. What was once a pragmatic workhorse is now an academic playground. Unfortunately, I have only experienced the academic playground era of the project–but I’ll take everyone’s word for it that ACS was once a helluva stable and cool set of web applications.

So far, I’m just being bitchy, and not really doing much to point out the real and serious flaws in OpenACS.

An example is in order. The OpenACS Bug Tracker. It’s a nice bug tracker. You can see it in action at OpenACS Bugs. Have a look around. It’s slick. Very powerful, entirely useable, fast, searchable, all sorts of good things. I approve. So what’s the problem? What you see is not the code you get when you download and install OpenACS. In fact, it shares almost no code with the new bug tracker. How can that be, you wonder? Me, too. The OACS bug tracker was rewritten a year or two ago by Lars Pind, a smart fellow who thinks very deeply about web application problems. The problem is that Lars didn’t set out to work on the bug tracker. He set out to make a cool workflow package, and somewhere along the way decided that a bug tracker would be a great example application. Apparently, the rest of the OpenACS folks agreed, because when the bug tracker was rewritten to use Lars’ new workflow package, it replaced the original bug tracker.

The new bug tracker looks the same on the surface. It has no new features or bugfixes. As one might expect of all new code, the new bug tracker has some new bugs of its own. The most egregious being that it doesn’t allow discussion of bugs except between the site administrator and the creator of the bug. Think about that for a moment. How many times have you googled about a bug, found an entry in bugzilla or roundup or trac, and dropped in to leave your two cents. Maybe you’ve got a solution or a workaround to contribute. Maybe you just want to get on the CC list so you know what’s happening. Or you have an extra data point that might help the developer reproduce or isolate the problem. It’s an absolutely vital and core part of the modern bug tracker. And the new OACS bug tracker can’t do it.

“Big deal!” I hear you cry. I said so, too, when I was excited enough about OpenACS to ignore all of the problems and push forward with our deployment. “I’ll fix that this weekend. How tough can it be? Really, we’ll just drop a comment button on that sucker for other users and all is well. I bet it’s three lines of code and a template change.”

A year later the bug still exists. I spent a couple of whole weekends trying to fix it, and then after a couple of months of periodically trying to fix it myself, I filed a bug. One of the OpenACS developers has spent quite a bit of time trying to fix it. It’s been discussed in the OACS forums. It’s been discussed ad nauseum in a bug about the problem. I’m not a software development guru, and I’ll be the first to admit it, but this makes me very itchy. How can such a seemingly simple problem be seemingly impossible to solve? Probably, Lars Pind could drop in and fix it in an hour, but he’s not around on the OACS project any more…so it’s gotta be someone else, and no one else is available. In fact, no one else on the project even seems to realize that the bug tracker they’ve been using at OACS.org is completely different code from the one that’s being distributed with the OACS downloadable packages. It does seem that a few folks are using Lars’ workflow in other projects, so the expertise is probably there somewhere…they’re just not working on existing OACS code.

Which brings me to an interesting and confusing aspect of the OpenACS project. The developers on the project are universally very smart. They’re well-educated in software development, many of the them have years of of web application development experience, they keep on top of modern development techniques, and are aware of the competition. The codebase they started from (ACS) was, by all accounts, among the best in the industry. But somehow, the current state of the project can only be described as “a mess”.

The scary part is that I can’t really point to any obvious thing that the OACS folks are doing wrong. They are aware of many of the problems, and have excellent sounding plans for resolving them. In the beginning of my time with OACS, I thought I was seeing a rebirth of a once-great CMS. I accepted all of the minor quirks, because I could see the incredible potential of the system. It has a lot of great applications, already built. I’m a big fan of “already built”, when it comes to software development. I like to put together components, and I don’t like starting from scratch. What could be better than a project that has dozens or hundreds of supposedly well-written, and time-tested, applications?

But, it’s over a year later. The ecommerce application is still just a bunch of incomplete components that have to be heavily modified to be useful. Well documented applications like “Edit This Page” have been deprecated with no documented replacement…the documentation still covers using ETP, as though it works, when it hasn’t for several revisions. The tutorials for creating your own OACS applications result in non-functional code.

In short, OACS is a mess and has been for the entire time I’ve been using it, and it hasn’t improved remarkably in the areas that I use. So, I’m jumping ship. Rather than upgrade to 5.2.x on my production server, I’m going to use that time to write a migration script to move our users, forum posts, and bugs, from OpenACS to some other CMS. I don’ t know what that CMS will be, yet, but Joomla is certainly looking like a nice package from the hour or two I’ve spent playing with it (I’m going to reserve judgement until I’ve spent more time with it, since early enthusiasm is what got me into the OACS mess to start with).

I’ve spoken about the problems with OpenACS several times in the past year in the OACS forums, so if you’re interested in more of my bitch-n-moan sessions, this’ll keep you busy for a good half an hour or so:

http://openacs.org/forums/message-view?message_id=351756
http://openacs.org/forums/message-view?message_id=352267
http://openacs.org/forums/message-view?message_id=352288
http://openacs.org/forums/message-view?message_id=352505

Note that all of these posts are several months old, some nearly a year. All issues addressed still apply, as far as I can tell from a glance at my devel server (which is running the latest version of OACS). Which is why I’m moving off of OACS rather than spend more time/money/frustration on it. I’d use PHP before I’d touch another line of OpenACS code, and if that aint a damning proclamation, I don’t know what is.

Slashdot linked to an interesting article today entitled 20 Smart Companies to Start Now. It’s probably worth a read for the entrepreneur wondering what kind of business to start, but only as an example of how to think about and present ideas that you want to get funding for, and not as a formula for what kind of business you ought to be starting.

Why not start one of the businesses defined and get your millions and live happily ever after? Well, there’s the sardonic answer that I could use: Most of the ideas are just plain stupid. But I would never say that, because I don’t want to offend any potential investors. (Quiet in the back! I am not saying that most of the ideas are just plain stupid, I’m merely saying that one could say that.)

The longer and more useful answer is that most of the ideas are mountain-top ideas. In the sense that the VC has come down from the mount of money upon which they normally perch and granted us a glimpse of their dreamworld. There is no interaction with users to shape these ideas. The value in a startup, and the value that VCs can’t bring to the equation because they don’t usually bring huge numbers of real users, is that you get real feedback and you shape a product that has value, even if the initial idea was somewhat different. If you created one of the products defined by the VCs here as being shoo-ins for big funding, and immediately took it to the investor in question, I’m almost certain they would pass because you’re not bringing customers to the table (though they might claim it is because the idea doesn’t actually match their vision). And, with many of the ideas, you couldn’t bring customers, because there are none to be found for those products, or too few to justify the costs of running the business.
This is certainly painting with a too-broad brush, but my essay hero, Paul Graham, does it all the time, so I’ll do so now without feeling too much guilt and I’ll try to refrain from constant parenthetical concessions that there are arguably many different and valid opinions.

The Good Ideas

Before I tear into the bad ideas, I’ll pluck out a couple of good ones, and explain why you, assuming you’re a small fry entrepreneur like me, probably shouldn’t bother trying to enter them even though they are cool ideas.

Car Computer with Heads Up Display

The first one in the list, a car-based computer with a heads up display, is probably going to be an extremely profitable area. I agree with them on this. I’d love to have a proper computer system in my car. Navigation has been hugely successful, despite being deeply flawed in its current incarnations. Everybody loves MP3 players and cell phones. Why not bundle them all up into a cool car computer? Great idea, indeed.

So what’s the problem? It’s already underway, and you won’t get there in time. The players already moving in this space have been there long enough to have a lot of relationships with car dealers in place, but are small enough to be reasonably nimble. Of course, if you have the team handy (I doubt you would need more than three or four to do the technical side of this, though, so I suspect the 20 number suggested is way too high, if your people are smart), and can get rolling immediately, you might stand a chance. If you can get big enough seed capital to build your prototypes.

Another problem with this suggestion, possibly the biggest one for folks who might have access to the team and the inclination to work on this problem, is the offered investment: $5 million for a deeply qualified 20-person team to deliver a prototype and a plan for pitching a commercial version to automakers within three years. Wow. $5 whole million dollars for 20 deeply qualified people over three years? I’m guessing he’s planning a trip to Bangladesh soon, or he doesn’t want to take a very large stake in this business (which makes me question how much he really believes in this idea). $5m would cover a shoestring team of 4 nerds, a part-time lawyer to handle the patents/copyrights/trademarks, a business manager, and a sales person pinched from Garmin or AC Delco (they’re needing to lose some employees anyway, so you could probably get them cheap) to line up your car manufacturer deals, plus all of the prototypes. It’s definitely not covering a 20 person team.

Mobile Health Monitoring

This probably has huge potential. I think it’s even coming within reach of small entrepreneurs within the next couple of years. Actually, this one is probably an idea worth pursuing. It isn’t Web 2.0. It isn’t cool. It won’t involve much Ruby On Rails (though your website can use it, if you want). In fact, it’ll involve C or assembly or that abominable embedded BASIC, and a lot of dealing with stupid and archaic systems (medical technology moves at a speed only attainable when a failure may result in someone’s death…i.e. really, really slowly). The boxes you’re building and integrating with will be shiny on the outside, but you’ll hate the crap that’s going on inside. You’ll probably get rich doing it though, so you won’t have to do it for more than a few years. And, hell, you just might help save someone’s life.
I stand corrected. This VC is almost certainly on the right track.

Super Batteries and/or Capacitors

This one is a no-brainer. Just about everything new that we do over the next 50 years is going to require good portable power.

You shouldn’t work on it because you aren’t the guys who came up with idea for capacitors with nano-fur, nor do you have the expertise. This is deep magic. At least, I’m not gonna try to touch it.

But it is a good idea. Not really interesting for someone to say so, though.

The Bad

When I say “bad”, I mean, merely, that the idea seems stupid to someone who has never made millions investing in tech companies. Everyone of them may very well go on to be the next Google for all I know.

Cell Phone Spam

This one clearly has a lot of demand from VCs, as it is the only idea that got two nods in the article (I guess portable power got two, as well, though one wanted capacitors and the other wanted LiIon). Basically, there is a desire from marketers to be able to call you up and spew ads at you any time they like. The buyers for the service are definitely there. I’m certain that if an ad service provider could convince a reasonable number of people to allow ads to be phoned in, the service would sell, and for a lot of money (to start with).

But, who in their right mind agrees to have ads follow them everywhere they go? There is, however, a way to turn this problem on its head and offer a service that people do want and occasionally throw in some ads. Y Combinator has funded Loopt, which seems like a potential service that ads could be injected into. I’m not entirely sure I would even want the “service” that Loopt is offering, but I’m a bit of an asshole who’s not entirely sure he likes people all that much, so I suspect I’m just missing the very real appeal of such a service. Maybe once they support T-Mobile and I get Loopt in and find loads of cool ways to meet up with my two friends, I’ll change my tune.
Offering ads as a service all by itself is doomed to failure, though once services like Loopt become commonplace, the need for ad networks that have small screen capabilities will grow rapidly. I reckon they can throw away the first prototype and re-use some of the code to repurpose it as an ad network, and come out alright. But they’ll waste a lot of money and time getting there. And the people who were working on the small-screen ad network to start with will probably already be the winners.

Web Search for Mobile Phones

Again, mobile phones are rightly seen as the next big market. A great convergence will occur, and the PC shall die a slow and agonizing death (OK, that part is probably over-reaching a bit). I reckon it’s true that billions are going to be made on mobile services, and I understand the desire of VCs to be there when it happens. I just think they’re all looking in the wrong direction.

The problem with this one is that web search is a gigantic problem. The actual presentation to the user is nearly trivial, in comparison (insert caveats here). With the biggest tech companies in the world battling it out, no mere $2 million startup is going to be able to solve this problem faster or better than Google, Yahoo, and Microsoft.

So to be a success with such a startup, you’d have to make use of one of the big search engines. Google, preferably. With that as a given, you now have as your primary business goal getting an iron-clad long-term contract signed with Google that allows you to use their results in your products. The API is there, so the technical side is easy. But if the market is really big (which it is), Google is going to want a stake in your profits when signing this deal or they’ll want to be able to inject the ads so they’re taking all of the profits from the actual searches. At this point, your only income is from the carriers that signup to carry your service. Maybe this can be profitable, I don’t know. I suspect they’d rather pay no one, and let Google solve the problem for free.

One way to turn this on its head into an idea that might actually work is to work on the small-screen interface problem in a generic way. It still hasn’t been solved, though there are a lot of people ahead of you in the race. The Palm interface isn’t bad, my Sidekick is pretty easy to use and reasonably quick, but both of those have access to a lot more input capability than a normal cell phone. So, if there is a solution to the “doing useful work with a numeric pad and a 240-pixel-wide 2-inch screen” problem, and you find it, you’re probably going to make some money. You’ll be in the patenting and licensing business, however, and not dealing directly with consumers, which is an area with wholly different skills than making technology products for end-users. That’s OK, of course, just not where I’m interested in going.

A New Database Company

Hmm…Really? A new database company? This new database…should it be: Lightweight? Cheap as free? Proven? Definitely!

I’ll concur that there’s plenty of room at the bottom for nimble companies to eat Oracle’s lunch. So, maybe this one isn’t such a bad idea after all. Everybody needs a database. But I’m doubtful of the ability of any small company to compete with IBM and Oracle for the big deployments and the Open Source options for the small to mid-sized deployments. I mean, I never stop to think, “Hmm, should I buy a database?” I always just install PostgreSQL for big jobs or SQLite for embedded jobs, and never think another thing about it. I guess there are customers out there who don’t think that way, but they probably just install SQL Server for big jobs and the embeddable SQL Server for smaller jobs or use Access when they don’t know better. I understand MySQL is also pretty popular.

A Make Your Own Video Kiosk for the Web (or “A Matchmaker for Mashups”)

This one is just sad on so many levels.

I’ll grant that there is a lot of content out there, and the studios would love to monetize it.

I’ll grant that people want to be able to mix up existing content with their own. Twenty minutes at YouTube reveals that people want to dance, play, sing along, make goofy faces, act out scenes, and pretty much live their (made up) lives on camera with a soundtrack of their favorite music and firmly immersed in popular culture.

So, there’s definitely demand for access to that precious content. But injecting video of myself into a scene in Glengarry Glenross? I don’t know anyone that wants to do that. Sure, there’s a small contingent of people that find it amusing to make a record of themselves singing over Celine Dion or something, at the local amusement park, once. Same concept here. Nobody is making millions off of this idea.

In fact, I’d suggest that if the studios figure out how to charge these folks for making ads for the studios products, they’ll move along to something else. It would suddenly become a job, and it wouldn’t be fun anymore, and they’d go do something else. Then YouTube (and the other fee-based services that the studios would like to replace it with) would have entirely become the ad-laden, boring, pointless, waste of bits that it has already started to become.
In short, it’s just a bad idea with nowhere to go. Though a web-based basic video editor might be cool, nobody is gonna get rich off of it.

Anyway, interesting read, for a look into what VCs are thinking right now. But I’m not planning to work on any of those problems today (I’ve got my own problem and it’s interesting and proving pretty successful), but I’m always happy to learn a bit more about the way VCs think and what their up to. They’re generally pretty damned smart, and they’ve got more money than I do, so they must know a few things I haven’t learned yet.

In Paul Graham’s latest essay, A Student’s Guide to Startups, he says the following:

Your final advantage, ignorance, may not sound very useful. I deliberately used a controversial word for it; you might equally call it innocence. But it seems to be a powerful force. My Y Combinator co-founder Jessica Livingston is just about to publish a book of interviews with startup founders, and I noticed a remarkable pattern in them. One after another said that if they’d known how hard it would be, they would have been too intimidated to start.

Ignorance can be useful when it’s a counterweight to other forms of stupidity. It’s useful in starting startups because you’re capable of more than you realize. Starting startups is harder than you expect, but you’re also capable of more than you expect, so they balance out.

I guess this might be a legitimate rule of thumb, but it certainly doesn’t apply to everyone. Many successful startups are started by people who’ve previously started companies (some successful, some failed, and some just getting by like my first). Folks like me know exactly what we’re getting ourselves into, and we do it anyway. I’m guessing there are people for whom entrepreneurship is the only work that is truly satisfying. Of course, I’m also too old to start another startup, by Paul’s reckoning.

Last year I closed down a small technology business, Swell Technology, that I had operated for nearly seven years. The business was profitable, it had no debt, and after so many years, the customer base was as large as it had ever been and still growing from very positive word of mouth. It was privately held, and I was the sole owner. It had several proprietary and Open Source projects under its belt, making me and Swell the de facto source of expertise on all things web caching in the Open Source world in America. I had my pick of small consulting jobs paying $800-$1000 per day on-site, as long as I didn’t mind travelling. So, why did I shut the darned thing down?

It just wasn’t fun anymore.

In fact, it hadn’t been fun in years, and it got less pleasant every day. So here’s an attempt to cover all of the reasons, small and large, that it wasn’t fun.

The Hardware Business Sucks

Swell Technology’s primary business was selling web proxy caching appliances (i.e. prebuilt, preconfigured, supported, easy to use servers) based on Linux, Squid, Webmin and a number of custom packages.
Being in the hardware business is just a crappy place to be. Dell sells systems to end users at the same price Swell could buy them from distributors, leaving us scrambling to offer huge additional value to make folks feel good about spending two or three times Dell prices for equal hardware. Our direct competitors (BlueCoat and NetApp), of course, charged five or ten times the Dell prices…but I just couldn’t bring myself to do that.

Our customers were almost universally happy, and they actually seemed to get happier the more we charged, but I was never able to bring myself to charge enough. And so, we never made money on the individual device sales. We generally managed to lose money on every box we sold. Volume sales were much more profitable, due to much lower per-unit support costs, but without a high pressure sales force to match BlueCoat and NetApp, we just weren’t in the running for the large sales.

In the end, in an effort to work fewer hours and bring in more help, I raised prices 20-40% across the product line. Sales volume increased slightly and the short term profits looked higher, but the support burden went through the roof. I had crossed the imaginary line in pricing that separates “enterprise” and “technical” customers, and the “enterprise” customers demand a lot of attention. I was working harder than ever.

The final issue with hardware is that it changes constantly, and in the case of OEM rackmount server hardware, it was constantly changing for the worse. For the first year or so, we shipped out crappy looking beige rackmount boxes with a simple plastic logo sticker to indicate they were Swell products. They were extremely reliable (at least a few of those original units are still in production use in the field today), and not too noisy, but they weren’t pretty. After that first year, I met the folks from Area Electronics at ISPCon, and they were just introducing Gigabyte OEM products to the US market. Those boxes were gorgeous. Looked better than the Dell stuff, had better components, and the price was right. Area would also do the hardware build for us at almost no cost, so we didn’t have to do it in-house anymore. Cool.

Then things started going downhill. Gigabyte had a couple of gaps in their line, requiring us to choose other options for the low end and the mid-range systems. Luckily, Quanta was in the mid-range, and they actually make some of Dell’s (and HP/Compaq, and others) systems…so the quality and “wow” factor was still good, though not as good as Gigabyte. Then Gigabyte left the OEM market entirely. So we switched over entirely to Quanta servers with MSI for the low-end box. And then Area stopped carrying Quanta, and I was unable to locate another US distributor, so we had to switch to all MSI, which were noisy, ugly, and simply cheap looking (with the exception of the low-end 9211, which was pretty and cool looking, but still noisy as hell).

Anyway, hardware sucks. Don’t do it. I never will again. I’ll let Dell figure out how to make it profitable, and I’ll focus on making great software. The pain of offering hardware was the straw the broke the camel’s back for my first startup. But it wasn’t the only problem.
A Telephone Ringing Now Makes My Stomach Hurt

On products that cost $2000-$12000, even if they aren’t actually profitable at that price, people expect to be able to call someone when something goes wrong. So, Swell offered 24/7 telephone support. First via a pager, that I carried with me everywhere, and then via a cell phone. The plan in the beginning was to grow enough to hire additional support staff, so I’d get a day off now and then…but that never happened. I eventually hired a bookkeeper, and I hired developers now and then, but in the end, I couldn’t afford to hire another me (or even a younger, less experienced version of me). The cost of providing support remained whatever my hourly rate for consulting and software development work was, or what I would make spending time selling more boxes.

So, every time someone called, it cost me between $95 and $500, because my hourly rate was $95-$125/hour and telephone support is extremely time consuming…something that would have taken ten minutes via email, takes an hour or more via telephone. I would regularly spend hours each day on the phone with a single customer, because of a misconfiguration on their router or a problem with their website. Not our problem, but I’m too nice to say, “Nothing I can do.”, when I knew exactly what the problem was and how to fix it.

Now, I hate telephones more than any other device on earth. I will never offer telephone support again, and I will never work in any position that requires me to answer a telephone. It’s a minor mental illness caused by a poor business decision. A good issue tracker makes telephone support seem absolutely barbaric, in contrast. Issues get resolved faster, they never get lost or missed, and I don’t have to interrupt dinner with my girlfriend to answer. We still have customers that ask for a phone number, but we’ve never lost a sale because I’ve refused to give them one (and I wouldn’t regret losing the sale if telephone is the primary mode by which the customer intends to contact me in the future…with software that starts at $69, I’m just not going to talk to anyone on the phone for any reason).
Open Source Users Can Be Kinda Bitchy

For the last three years that Swell was around, I was bored. My products were roughly complete. They looked good, they were predominantly free of bugs, and I finally had someone doing the actual build-install-ship process, so the tedious stuff was being handled too. So, I started branching out. We started work on a content filtering system, called PenguinFeet, with a Creative Commons licensed set of blacklists and a proprietary set of management tools (that ran on my server), and Open Source Firefox toolbar for submissions and blacklist management, and I hired four part time work-from-home employees (the Blacklist Wranglers) to validate our bot generated lists and find new sites that the bots missed and a part-time manager for those Wranglers. I paid for development of a gzip compression feature for Squid that users had been asking for for years. I hired Jamie Cameron (whom I’d worked with many, many times in the past) to write a virtual host administration tool (also known as a control panel) to integrate into Webmin, which we released under the GPL.
In response to the free PenguinFeet project, I received predominantly animosity from the Squid community (not the developers, of course, whom I’ve always counted among my favorite people). Some considered it censorship that Open Source developers shouldn’t be taking part in. Some just felt like it should have been more free, in the sense that my management tools should also be free, and that I shouldn’t be selling a product based on lists generated by the community (despite the fact that 99% of all entries in the database were from my bots and my paid Wranglers, and the lists were all under a Creative Commons license). And some just liked to bitch and complain because I wasn’t doing it the way they would have done it. Anyway, I never made a dime on PenguinFeet, and spent about $35 grand over 9 months, not to mention my own time spent on the project.

In response to Squid gzip, the animosity was nearly universal (again, not from the developers, who were excited about the project from the start). The reason, of course, was that I requested economic help from the users with completing the work before releasing the code. I’d invested about $10 grand and was out of development funds for the year, and another $4000 was needed to wrap it up, plus some money to develop Squid 3.0 to the point of stability. It was actually suggested by one user, with a straight face, that a collection be taken up from Squid users, to hire someone else to start from scratch and add compression to Squid. I’m not sure exactly what they expected to accomplish with this plan other than revenge for having the gaul to request a few donations to help finish up the work, but the plan was squashed quickly by Henrik (the real hero on the Squid project for the past four or five years, and the fellow I now highly recommend whenever anyone comes to me for Squid development or support work). Anyway, long story short, there are a few Open Source software users who are offensively ingrateful. Those few users helped convince me that an entirely Open Source-based company, which Swell was, just isn’t worth the heartache.
These are but two instances out of dozens in which a user (never a developer) of one of the Open Source projects Swell included in our products, and always contributed to, decided that because I was attempting to make a profit on Open Source software, or even just avoid incurring additional expenses when an Open Source project went over my budget, scorn was in order.

For what it’s worth, Squid gzip has been imported into the Squid CVS tree, despite never getting the finishing touches I was attempting to raise funds for. Hopefully, in a year or two, when Squid 3.0 stabilizes, gzip support will also get some attention and inline compression will be a reality for Open Source users (it’s been available in numerous, very expensive, proprietary products for several years). The PenguinFeet database is still live at http://www.penguinfeet.org and I’ll probably come back to it when I have some free time, though the Blacklist Wranglers have been off the job for a year and I’ll have to come up with a better way to handle the “the internet is HUGE” problem before calling it a success.
My new startup is still pretty fond of Open Source software, and my co-founder and I spend at least as much time on Open Source projects as on our commercial software, but the single product we sell is not Open Source. I made a sincere and persistent attempt to make a decent living with entirely Open Source-based products and failed, so I’m trying a slightly different path this time.

Note: I’m not complaining about not making any money on my Open Source development projects. I didn’t expect to make money, I was working on them (or funding the work on them, since I was busy running a company and wasn’t doing any development other than that which was core to my business) because they were needs I saw in the community and I had some resources that I could throw into getting those needs filled. My complaint is with the attitude of some of the users. Seven years of hard work, and frequent funding, on behalf of Open Source projects deserves at least a modicum of civility from the users of those projects.

It was also partly my own fault for not realizing the extremely short memory of crowds. The users who complained were new to the community, and didn’t bother to search through the mailing lists to see that I had been a contributing member of the community since before Swell existed. These users just assumed I was as new as they were and had only come to cash in on this new-fangled Open Source thing (perhaps the thought of doing so themselves had crossed their mind recently).

The Problem Domain Was No Longer Interesting

My enthusiasm for proxy caching simply faded. There really isn’t a lot to say about this one…the independent ISPs that I most enjoyed working for and with have gone the way of the buffalo in the US. The telcos killed them by repeatedly flaunting their power and fixing prices for access to the last mile. I’m opposed to regulation, but the so-called “de-regulation” of the telcos wasn’t. They still maintain their state protected monopoly over the last mile.
Also, I suspect I am not alone among entrepreneurs in having a roving imagination. I don’t like to stay on a single project too long, and seven years proved too long.

And, that’s all I have to say about that for now. More to come in part 2.

So, I’m starting a new blog today. I’ve operated a blog on this same domain for a few years, but it only saw a half dozen or so entries. Probably because I had to login to the server to write entries, due to the software I was using (Bloxsom). This year, I wrote no entries because I’ve been so busy with my new software company, Virtualmin, Inc.

But, that same new company, which is the second I’ve founded, has made me aware that I’m relearning a lot of the same things I learned during my first company spin up several years ago, though the lessons sometimes look a little different due to it being a different kind of company. Anyway, this time around, I’m going to write them down, so maybe I’ll be able to talk myself out of starting yet another company five years from now (or at least I’ll have to spend less time re-learning stuff, since I’ll have these crib notes).

And, of course, in this modern age, everybody keeps their journals on the web, so I’ll do the same. Maybe someone can learn something from my thoughts on my first barely successful company and my new and very promising company. I’ll also talk a lot about software development, since that’s a core part of my new business, though I’m not much of a developer myself.