The Cloud - Not a Panacea

Everyone loves The Cloud. I mean, why not? There's no hardware to own, no storage systems to maintain, no networking hardware to deal with, it's infinitely scalable to the billionth degree; it's perfect!

Ok, so no one gushes about it that much (do they?). It has its notable benefits, but they aren't quite as obvious or simple as people sometimes make them out to be.

The crux of the issue is simple; simply moving traditional services, applications, or systems to a cloud-like environment yields little. Traditional systems with no knowledge of a cloud (or grid, or virtualized infrastructure, or...) environment can't take advantage of the dynamicity that it offers. Many systems (and arguably, people) don't understand what the true benefits of the cloud are.

A Common Cloud Case

Let's take a concrete - and common - example: an RDBMS on the cloud. I'll talk specifically about MySQL because it's on the tip of my tongue, but it will apply to many similar systems.

So there you are, running MySQL on Amazon EC2 on your Linux distro of choice. That was easy enough. You cron a nightly export to S3. You've taken advantage of some of the shared resources and that is good, to be sure. You can use EBS for large volumes. You lose some performance, but you can afford to get a larger instance so maybe you can increase some of the buffers and keep more in memory. Depending on your situation, it may all work out in the end. We won't get into the nuances of exactly what computational resources you get because it's damn near impossible to measure accurately and consistently. Let's call it a wash.

Perhaps the most important part of this is that you can easily set up one or more replicas. That's pretty damn nice.

That's a lot. The problem is that there isn't much one couldn't do in a traditional data center. Sure, something like EC2 can help jumpstart a start up - which is great - but for medium sized and larger companies, this isn't a concern. The question is what does the cloud do that the traditional data center (and approach) does not? The cloud is dynamic and on demand for a reason and this isn't it.

What the Cloud Does Well

Applications that natively and intrinsically know about the cloud and its properties and can actually react to changing conditions are the true candidates for cloud computing. For instance, stateless web serving is something that can take advantage of this kind of environment (with some additional functionality). The reasoning here is simple: with some simple measuring of load and knowledge of capacity, additional web server instances can be forked off and run, independently. These additional web servers (or their IPs) could be added to an external load balancing system to make them available to the public. There's no significant dependency here. Content to be served by the additional web servers can be made available with little fanfare and, provided you have enough resources, additional connections can be made to resources such as relational databases, caching servers like memcached, and so forth.

That's, of course, not the native case I mentioned. That's an adaption of a traditional service to the cloud using an orchestration process or resource manager, of sorts. The best case for the cloud are applications where change of the infrastructure is built in. People tend to easily go to places like map / reduce frameworks where jobs are self contained or transport state with them. Not to trivialize that case, but it's not everyday you find the perfect fit for such a model. Many times, you find hybrids where map / reduce jobs (a computation layer) requires access to a data grid (the storage layer) which limits your flexibility and ability to deal with changing conditions. In that case, it's usually not desirable to repartition or shift your data storage based on load (although you can repopulate caches based on load and network expansion). Maybe you don't care about data affinity because the footprint is small or access is infrequent, but with massive data stores or logically partitioned data, this is prohibitively expensive.

I'm not sure I have a good conclusion to this. What I'm driving at is that, while the cloud and dynamic infrastructure is a blessing, don't think of it as a panacea. Do the cost breakdown and consider what portions of your systems make sense in such an environment. Many times, running a traditional system where dynamic setup / teardown of nodes isn't feasible, on a platform like EC2, will wind up being more expensive after a year or so given even light usage. It's a buzzy topic and everyone wants to be on the cloud. Remember that the fastest way to kill something in the eyes of business is to push it for the wrong reasons or in the wrong situations.

Operations on the cheap

My current position puts me back into the world of tech ops, for lack of a better description. I'm primarily responsible for data center operations and infrastructure. It's not a bad gig, but as one might expect, it shifts the focus.

I have the omnipresent mandate of running the show on the cheap. This, of course, should be an anti-surprise for anyone in either a startup or a smart company. Waste is always wasteful even if you can afford it. Hint: you can never afford it.

Given this most recent brush with corporate poverty (which I fully support and feel is one of the things that keeps us honest about what is really necessary) I feel like I need to write something up on what has worked, in my experience. Maybe it's time for motivation.

Open Source and the IP Clause

This article reflects my lay-person's understanding of employment and intellectual property. It is not legal advice of any kind. I am not a lawyer. Don't assume that any of this is correct. This is based on drips of information I've picked up over the years. Corrections are very welcome.

As I've mentioned recently, I've switched jobs in the past month. One of the things I always worry about is the dreaded intellectual property clauses in most modern employment contracts. You may be surprised to find out that most companies in the know are willing to modify these, traditionally, draconian clauses if you phrase the request correctly.

The Problem

Commonly, these sections claim ownership of all IP, in all forms, for the company, with very little restriction. In more than one case, I've seen contracts that claim ownership of all IP developed on and off the clock, regardless of whether or not the work is related to what the business does or may do. This, of course, means that anyone interested in doing any kind of open source development, or even research of any kind, will not necessarily own such work.

I like to get involved in open source projects so having the freedom to do so is worth something to me. I found that simply being up front and asking the employer for modifications to the contract, if necessary, is the best way to go.

The Changes

In the most recent contract I was presented with, the company claimed all rights over everything developed while I was employed. Having one or two projects I'm interested in working on that are entirely unrelated to the company's stated business, I asked for changes to the contract.

I did not provide specific language for the changes I requested. I asked the company's management and general counsel for the following privileges.

• Ownership and the right to develop source code, written work, and IP in any other form provided it did not compete with or interfere in the company's stated goals and business.
• The right to release any IP, in any form, under a license of my choosing, citing the GPL, LGPL, BSD, APL, MPL, Creative Commons, and other licenses as examples of the type of license I may select.
• The right to participate in standards groups and bodies, user groups, and conferences, presenting any IP that I would own, under these changes.

Some of that is redundant. I asked for these things this way because I wanted to make very clear what my intentions were; to effectively disclose to the public, any IP I developed during my employment that was unrelated to the business.

As concessions, I offered that the company would be allowed to dictate the following.

• Whether or not I disclose for whom I work. They have the option of requiring me to state for whom I work (i.e. if they feel it helps them in some way).
• If any IP was in a shady grey area about whether or not it was in conflict with the business's goals or primary line of business, it would remain unreleased by me.
• I must select a license that allows the company to use said IP. Restrictions on their use of said IP was acceptable (i.e. I license something under GPL which limits their usage in some cases).
• They may opt to review work for conflict with their business.
• I am still (obviously) subject to, and aware of, the non-compete clauses in my contract.

Reality

It's not utopia. I don't even know if it's all enforceable; like I said, I'm really not a lawyer (and they are). I probably won't develop anything worth their time, but I want the option to try. More importantly, we, as potential contributors to open source work and standards, have the obligation of protecting projects and bodies to which we contribute. I don't know how effective the changes are that I requested, but certainly in its original form, that contract could be problematic to an open source project if I had contributed something and my employer wasn't happy about it.

I hope more companies, especially those that thrive on open source software and standards (even if only indirectly; I'm looking at you, Oracle, Java, and Microsoft shops), become more tolerant of what is required to generate such IP. Better, of course, are those that can directly contribute company owned IP, but by simply not attempting to exert ownership of the minds of their employees, a company is taking a huge step in the right direction. At least in my (non-lawyer) opinion.

What Makes the Job

As much as I have a burning hatred for interviewing for jobs, I've had to do so lately. I'm a creature of habit and comfort. I like working with and getting to know a core group of people. People are important like that. People are, absolutely, what make the job.

More so than anything, I've realized over the last week of interviews (I limit it to exactly one week of interviews) that a little extra money, the promise of equity in a company, even the business the company is in; all a far distant second to the people. Smart, talented, interested, people.

Almost unbelievably, I found two companies that seem to be full of people that are exactly what I would look for (well, at least the ones I met). For the record, I'm not sucking up; I've already received the offers and I don't think they know about or care about this blog. There shouldn't be anything remarkable about this, but there is. No pinball machine, free bagels, or company sponsored potato sack race can possibly come close to what I get out of picking the brains of smart coworkers.

That's all. Nothing notable, per se, just a realization about what's really important to me.

The development process - a look at OpenUP

I've encountered a number of companies that eschew any kind of formal process. The thought seems to be that a development process, any process, impedes flexibility and bogs down the process (lowercase p) of development. Personally, I've always been skeptical of processes for similar reasons. As a developer, you feel constrained by what your role is defined as, in any specific development methodology.

The longer I do what I do, the more I find that process isn't bad; it's the processes that we have (or had) that were bad. It's the type of process that needlessly constrains and restricts more so than having a process at all. Because of this belief, I was relieved when I first discovered agile methods. I was late to the game, to be sure, but I was equally impressed, none the less.

I read, on some random blog (sorry, I don't recall which one), about OpenUP - described as a lean Unified Process that applies iterative and incremental approaches within a structured lifecycle. In my reading of the wiki, I found that it's very easy to understand in about fifteen minutes, even for mere mortals (read: non-PMPs or trained project managers) provided you have a rudimentary understanding of software development. It has some interesting advantages:

• Low-ceremony
• Tool agnostic
• Project type independent
• Attention to small time windows (talks about days and months, not years)

Of course, it's also chock full of the normal agile development goodies like acceptance of change, iterative thinking, and so forth. It's interesting to see people defining simple, easy to understand and implement, processes that, themselves, are open to refinement. It just makes sense.

Give it a look. It's neat stuff.

OpenUP Wiki

Distributed teams in development

I suppose, at some point, an organization becomes large enough where teams of developers are geographically disparate. In my experience, this is never fun. Each team seems to develop a good rapor locally, for obvious reasons, but that doesn't necessarily translate to a strong connection to peers in other cities (or even countries). It's always difficult to spread common knowledge when the level of interaction between teams is low.

There's been some effort to bring people closer together, but these applications or tools are either too "low bandwidth" or don't allow for the kind of communication you really need for things like peer programming and code reviews. I'm referring to tools like MS Live Meeting, which is something I've been personally subjected to. Yes, subjected is the word I mean to use.

I've recently read a blog entry by Grady Booch where he briefly mentions using Skype and Second Life, which sounds interesting. This also appeals because it's available to us Linux folks (at our office, we have a projector connected to a box running Windows, a wireless mic with a polycom, a bridge phone conference thingie, all of which we use solely for team presentations). All bias aside (I have a distinct dislike of Live Meeting), Live Meeting is too slow and is a good representation of low interactivity, low bandwidth distributed tools. Skype (or other VoIP-ish tools) are a bit more natural for realtime conversations, but lack integration with any kind of tools. I'm not entirely sure what Second Life brings to the table; my exposure to it is rather limited.

It should be noted that many, if not all, open source projects are developed by distributed teams. I've been directly involved in a few popular projects such as Gentoo Linux and contributed patches and bug reports to others. In most cases, collaboration tools are critical to these kinds of projects; IRC networks, forums, mailing lists and numerous other bits of infrastructure exist to support this development model. The major thing I've found lacking is integration into the common toolsets of choice, be it vim, emacs, or a full blown workbench like Eclipse.

So why is it that a bunch of developers can't get on the same page about good, distributed, high bandwidth collaborative tools? Better yet, why can't our normal tools support collaboration as a standard part of development? Things like version control systems cover history management, tracking, auditing, and other functions, but they don't help with real time code review and pair programming. Distributed source code editors are interesting, but I'm not sure I've seen one that fits with the tools a personal already uses, as opposed to simply making them use other tools. In an age of modular language workbenches - I'm looking at you Eclipse - this should be something to pursue.

The role of the System Architect

My official title at my day job is System Architect. Simple, straight forward, theoretically well defined. The wikipedia definition of a system architect makes it seem like the world is made of butterflies and honey and everyone is in love. It couldn't be less true.

Admittedly, at my place of business - a tech-saavy company with a development team of about forty very smart people - the idea of what an architect is and what he or she does is still forming. The position didn't exist up until recently. It's a little less like the wikipedia definition cited above, and quite honestly, thank God. My position is definitely more technical than it is business oriented (i.e. I still write code, et al). I tend to focus on (and possibly obsess over) design, system structure, subsystem integration and communication, and other seemingly opaque and lofty topics. Luckily, I still have to eat my own dog food and practice what I preach, so it can't all be about purity and pie in the sky conceptual cruft. If there isn't any meat on the bone, I could easily get eaten alive by my peers. (My apologies for the list of inappropriately used cliches above.)

Sometimes, the responsibilities can get hazy, though. Given that I am in a consultative rather than authoritative position, the onus is on me to prove that the solution offered is the best there is. There's no guru hat here; it's put up or shut up.

Not often, but almost with certainty, there are situations where time constraints around a project force a sense of impending doom and severely limit the time people are willing to put into discussion of a system. It happens. It can be even worse than that, though. The time it takes to have these discussions, not to mention the emotional energy when everyone in the room actually cares, is immense. People can get their egos bruised, or worse. I'm certainly not above such things, at times.

So what is a System Architect, really? Specifically, how do we prove the value of things like design patterns, integration patterns, and the like to other developers?