The transformation of any idea into a thriving culture that disrupts the status quo is attributed to the zeal of the people at the center of that idea. It is the conviction and the unwavering commitment of the community that pushes forward an idea into something that can bring about a change.

Community is impactful in and of itself, but that impact reaches even greater heights when applied to the world of software. One of the central reasons that open source software brought about a revolution in how software is written, how systems are built, and how innovation happens is the dynamism of a community of people who form a part of it. It is the contribution of individuals from all walks of life who constitute this community that has paved the way for open source software to permeate every line of business and has played a vital role in democratizing technology.

Jim Salter is once such active contributor of the open source community who, through his initiatives and projects, is enriching this culture and inspiring many along the way. An author, a technologist, and an open source entrepreneur, he shares his thoughts about his association with FreeBSD and open source, his open documentation project, his consulting firm, JRS Systems, and All Things Open 2015.

Tell us about your experience when you set up your first FreeBSD server. What led you to FreeBSD, and how was it different?

Believe it or not, I was mostly a Windows guy and happy with it back then. But I knew Windows was wildly inappropriate for Internet hosted services, which I was very interested in. Back then, a little site called Walnut Creek CDROM—better known to most as “ftp.cdrom.com”—was doing leaps and bounds more traffic than anybody else in the world. Better yet, it was doing so on FreeBSD, on comparatively little actual hardware! Seemed like a no-brainer to me: It was time to learn FreeBSD.

I started out by moving my personal website to a small dedicated host in Florida who specialized in dedicated FreeBSD servers. He did the grunt work, though—by the time I had the box, it was already running FreeBSD 3.0, with Apache and the native FTP server installed and configured. All I really had to do was FTP my site up. Ho, hum. It was a start, though!

My first real, no kidding, I’m-root-and-it’s-all-on-me experience came a year or two later with FreeBSD 4.3 in early 2001. I built a box to sit in my living room with the eventual goal of it running an UltimateBB forum for my personal webpage. The first thing I did was get Samba running on it and dump all my files on it—this forced me to actually use the machine on a daily basis, which I knew was the only way I’d make any real progress learning about it. Fun times. By 2002, I’d learned enough that I wasn’t paying my webhost anymore—I was giving him technical support, so he comped me on my dedicated server. I should have gotten a T-shirt: Will Sysadmin For Server Space!

I’m not sure that tells the whole story of why FreeBSD was better, though. It certainly had a reputation for being more stable and performant than Linux in the early days, and as far as I could tell it was deserved. Honestly, what impacted me more was that it was much more logical. I struggled to get through Linux installs, much less actual usage, because the whole thing seemed to be a chaotic hodge-podge back then. FreeBSD, though… it had a reputation for being “hard.” In practice, though, I found FreeBSD much easier than Linux because it was so much more consistent. Sure, the installer might have been NCURSES text instead of graphical, but who cares? It made sense—as did the rest of the OS’s layout. A benefit of having the entire OS being under the control of a board, rather than just the kernel, I suppose.

What inspired you to start FreeBSDWiki.net? What is the goal of the site?

As I was learning FreeBSD, and Unix-like operating systems in general, I got really frustrated with the quality of the documentation. So frustrated that I really didn’t want to just contribute to the existing documentation because I felt like my documentation goals were just too different from the mainstream. Manpages and other official docs are usually an exercise in theoretical use and verbosity; blog posts were usually way too much “story time” and explanation, and not enough “here’s what we did/what you do.” I wanted to document by example—show me the configs, show me the commands! I also had gotten to the point where I would often need to recreate a task it had taken me five hours to figure out the first time… and it would take me another four hours to figure it out all over again the second time.

Back then, Wikipedia still seemed pretty new and magical. The whole idea of publicly editable documents still seemed pretty wacky, but Wikipedia had been doing it successfully for years, so I figured why not? And freebsdwiki.net was born. I tried to document the things and the tasks a newbie like I had been would need to know, and I also documented everything new I did or learned. All the articles were long on examples, short on exhaustive lists of every possible option. I subtitled the site “FreeBSD For The Impatient.” After a few years, I discovered kinda by accident that my FreeBSDwiki was one of the 500 largest wikis on the planet, which I thought was pretty cool.

Honestly it’s languished for a lot of years now, though. I’m no longer really active in the FreeBSD community, and the wikispam problem got just too berserk to deal with. I ended up writing my own spam filtering plugin for mediawiki, which was quite successful for a while but eventually got to the point where I was spending half an hour a day manually removing spam that the filters missed, and it was just too much. I finally took the site “private,” at least in terms of editing—anybody can still view it. It’s getting more and more out of date, though, and I really need to see if there’s anybody out there who wants to take it over.

With Linux used for running major enterprises all over the world, what do you think is the role of open source tools and technologies in system administration today and in the future?

I’ll be blunt: Learn to embrace open source, or get buried. It’s that simple.

Personally, I love open source. I love the ideology, I love the code, and I love the way it makes me feel to know that when I learn an open source app or operating system I can take that knowledge with me and use it anywhere for anything. That’s some serious power, right there! You learn to use Photoshop, and now you’re tied to $1,000 or more of software license—you might or might not be able to get an organization to buy that for you, or you might or might not have one of your own that you might or might not be able to use at any given organization. That just sucks, and I got bitten by licensing more times than I can remember in the bad old days. Now, though? Learn to use Krita or GIMP and you can take that anywhere. It’s yours. Those capabilities you gained when you learned how to use it are yours. You can use them—legally—however and wherever you want to. I wish more people understood what that really represents.

I digress, though. Whether you personally love FOSS or hate it, you’d better learn to leverage it or you’ll get left in the dust. Yes, there are still proprietary software powerhouses in the world—and there likely always will be. They have to learn to leverage and embrace FOSS too, though. When they don’t, they pay the price. Nobody has enough money to out-code the entire open source world. You can make proprietary coding a paying proposition when you’re only coding your own special sauce, but if you try to do everything that way you just won’t be able to keep up.

This trend isn’t going to stop. FOSS isn’t just hobbyists anymore—the hobbyists are still there, but they’ve been joined by professional developers employed full-time by industry giants specifically to work on open source. Companies like Google, HP, and IBM have long since realized that when it comes to the infrastructure under their “special sauce,” it makes more financial sense to cooperate with the rest of the world instead of buying proprietary or rolling their own—and that means that you’re running with a serious technical and financial debt if you try to compete on “in-house-only” product.

TL;DR: Get used to FOSS, and do it quickly if you don’t want to get left behind as a company or as an individual developer.

In your consulting practice, how do you use open source software to solve your clients’ problems? How do your clients perceive open source software as a solution to their business needs?

I work a lot in the small business space, where cost is a more significant factor. With that said, though, even at the microbusiness level cost isn’t the most important thing. Sound crazy? Well, let’s face it: if you pay three or four full-time employees, your payroll is going to dwarf your IT budget, even with proprietary and spendy stuff.

What’s really important is how reliable the technology is, how maintainable it is, how easily deployable it is, and how it improves the organization’s workflow. FOSS is fantastic for the first three. For the last, sadly, it frequently falls short. This usually isn’t because of anything wrong with FOSS itself—it’s just because so much of the world is still dependent on a monolithic block of proprietary software for everyday commodity processes.

What’s all this mean? It means that I’ve got Linux in every business, but it’s usually tucked away in the back where the end users don’t directly see it. Let’s look at an example: I think LibreOffice is fantastic—I prefer it to MS Office, and strongly—but it’s not usually appropriate for a business setting. Again, this isn’t because there’s anything wrong with LO—it’s a fantastic product—it’s because most of the businesses my customers interoperate with are using MS Office and are strongly dependent on its features and its bugs. Give a poorly-trained end-user access to an office suite and a few hours to hammer on it and they’ll frequently produce a document that looks mostly ok on the surface, but is a mass of godawful spaghetti code underneath. An end-user-produced document might have macros in it. Ugh. Section headings are likely to be manually formatted individually rather than defined from styles. Ugh. A sentence that’s “just in boldface” might actually also be in italics, then out of italics, run through four or five font settings, kerned differently, and every other possible horrible thing you can think of underneath the surface, because the user just kept hammering at it until it “finally looked right.” You can rely on LibreOffice to render a well-formatted MS Office document very well, but rendering “the kind of MS Office document you can expect to receive every now and then in the real world” is an entirely different ball game.

So as much as I’d like to, I rarely deploy LibreOffice in a business environment. When a contract that means a few hundred thousand dollars of revenue depends on users interacting well with a horribly formatted document written in a several hundred dollar office suite, you grit your teeth and you spend a few hundred bucks per user to buy that suite. Similarly, GiMP and Krita are usually going to be out of the question for businesses that are actually doing graphic design. Yes, there are outliers who understand the difficulties involved with bucking the mainstream and are willing to go with it, but for the most part it makes more sense to just go with the flow whether you like it or not.

Wait a minute, though. Didn’t I say “Linux in every client?” Where is it? Like I said, it’s on the back end. Even when there are Windows Server 2012 machines, they’re virtual machines running on Linux hosts using the Linux Kernel Virtual Machine for a hypervisor and openZFS for the storage. I developed my own converged infrastructure platform, Sanoid to facilitate this. This means I can do things like roll back a VM that’s been hit with Cryptowall literally in seconds instead of spending hours upon hours restoring it from clumsy backup, and I can replicate terabytes of data offsite daily over a cheap little 5mbps residential-style internet connection… but the users don’t see anything different that might impede their workflow.

At my customer sites you’re also going to see a lot of BIND9, a lot of openVPN, a lot of Apache, Varnish, and some Postfix—like I said, the back-end stuff. On the front end, you’ll see a fair amount of WordPress (I’m going to call that “front end” just because end users usually make posts directly on it), and cross-discipline software installs. “Cross-discipline?” Well, an architect is probably going to have and use a full copy of Photoshop—but a mechanical engineering firm that just wants to edit an image every now and then will probably have Krita or the GIMP. You get the idea.

FOSS is starting to make inroads into end-user territory, though, mostly by way of cross-platform software. More and more people are familiar with Thunderbird, Pidgin, LibreOffice, and GIMP. Everybody knows about Firefox. Chrome is the default browser at all of my clients—yes, I know, Chrome itself is proprietary, and Chromium has some controversial division between FOSS and proprietary code, but it’s still a lot better than Internet Explorer or Edge! I’d look for this to increase over the next decade or two—mostly in the same way that Windows itself invaded the business space—filtering upward from the users’ home systems, rather than the other way around.

Can you give us a sneak peek into what you’ll talk about at All Things Open this year?

My talk is Move Over, Rsync. I’ll go ahead and give you the punchline now: If you aren’t using file-system-level replication, your backups absolutely suck. Which of course is a statement that will make just about any veteran sysadmin bristle, but that’s the point!

I’m going to be talking about ZFS replication, and how it can beat Rsync performance-wise by multiple orders of magnitude in some common use cases—yes, really—as well as being inherently more secure and reliable. Did I mention I routinely, daily, replicate multiple terabytes of data offsite over 5mbps internet connections, usually in an hour or less? Because, well, I do that.

All Things Open
Speaker Interview

This article is part of the All Things Open Speaker Interview series. All Things Open is a conference exploring open source, open tech, and the open web in the enterprise.