What a Senior SysAdmin Should Know

Prior to taking on the System Architect role where I am, my title was Manager of System Administration. I've worked in system administration enough to know about all the things I don't know. Sadly, I've also learned a lot about what, does not, a sysadmin make when interviewing people.

In the Linux world, there are a growing number of people who call themselves system administrators. Technically, everyone who sits behind a computer administers it to some degree. The question, in the case of those that want to make it their livelihood is, to what extent do you know your field? Granted, these aren't the days of punch cards, nor do you have to be an electrical engineer to touch a computer (put down that soldering iron, if you please), but you do have to know your stuff, to be sure. But what stuff, and to what degree? Fair enough.

Networking concepts and Protocols

This is a hard and fast requirement; you must understand basic networking concepts and common protocols. It's that simple. You must be able to explain how TCP/IP works (if SYN, ACK, SYN/ACK doesn't mean anything to you, you're in trouble here). You must know the difference between stateful and non-stateful firewalls. You must know what an MTU size is and what changing it means. You must know what happens if a DNS response packet is larger than 512 bytes. You must understand, conceptually, how routing protocols work. This stuff isn't brain surgery (recently, I heard that brain surgeons joke that brain surgery isn't brain surgery), but it's a bit dry. You absolutely can not be a (structural) architect if you can't do basic math.

Linux as an Operating System

Should I need to say that you have to know about Linux-ish things to be a Linux sysadmin?

You must know as much as you can stand about the Linux kernel; building one, updating (more on tools like package managers later) kernels, where modules live, when they're loaded, why, how, how to manually load and unload them, and other exciting stuff. You must understand the Linux boot process and init. You must know about the boot managers and how they differ (lilo, grub, etc.). You must be able to write, modify, and maintain shell scripts (and understand concerns like quoting errors, for $deity's sake). You must understand basic system libraries such as (g)libc, and things like PAM (trust me, you'll thank me later). You must know how to tunnel everything under the sun over ssh. You must understand the interesting file systems like ext{2,3}fs (and you must understand what that {2,3} syntax means), udev / devfs, tmpfs, and especially proc. You must understand how to modify file system options (hint: tune2fs).

Security

You can not survive as a system administrator without a better than average understand of security. No excuses.

You must be able to read and interpret bug reports and know about things like the CVE list. You need to understand system hardening. You need to know iptables / netfilter. You need to know the security implications around certain bits that can be twiddled in /proc. You must understand common attacks and know how to mitigate them. You must understand what IDSs and NDSs are. You need to understand, at least conceptually, cryptography and how it applies to security (hint: lolzEncryptEverything!!!111 doesn't count as a valid answer). You must understand PKI. You must know how to respond to security incidents.

A Bit About Tools

Who doesn't like things that make our lives easier? If I were to read this, and admittedly, even as I type it, I would think that this sounds like a lot of stuff that is made easier by tools provided by vendors, open source projects, or even distros, themselves. That is entirely true. There are RPMs or DEBs of kernels, so why should you care where modules live or how the system boots? Because it's your job and knowing what tools do and how they work will not only help you, but will bail you out of trouble. I think projects like Webmin, Firewall Builder, and the like are great, but you still must know what they do and the concepts behind them before you can really understand how to use them properly. I'm guessing that the authors of those projects would agree.

This isn't about being elitest, but about doing what we do, and doing it well. Take everything as an opportunity to learn and grow and you'll quickly find that it's much easier to succeed when you know rather than when you guess correctly.