Thursday, March 12, 2015

¬ Agile

Some of my thoughts on Agile got some Twitter press recently. I reproduced them below, with some light editing to make the series of comments into a single essay.

One complaint I have with Agile is that it takes the avoidance of a long-term plan too far — to the cartoonish extreme that user feedback is basically hooked up as somewhat of a control system. If user feedback was coherent, this might not be a problem, but in general user feedback is all over the map, contradicts itself, and calls for feature implementations or bug fixes that are highly speculative from an engineering investment point of view. If you are lucky enough to work on an Agile team where engineers are actually consulted for the feasibility of this kind of thing, then it might be OK and Agile might really be more of a set of bookkeeping conventions for you. But that’s not at all how it plays out in practice. In practice you’ve got sales-facing and business-facing managers in the loop, pushing various agendas and always looking for an angle to play based on the work in the back log and pointless metrics. The metrics are also a pretty substantial detractor. Burndown is a joke, particularly compared with linear burndown, and also when compared across teams and projects that vary wildly. Again, I suppose in some theoretical vacuum it might be OK if managers actually understood that you can’t just look at half a dozen burndown graphs and generalize that into any sort of meaningful management decisions. But that’s not what many middle managers do: instead they warp, twist, and misrepresent Agile metrics to suit their changing political circumstances and to lobby for their self-promotional agendas. It’s really depressing as an engineer to literally see in real time (as story points are being partitioned out) how some political agenda is being lobbied for with useless metrics and how it is directly affecting your personal work. I’ve always found this article about estimation games particularly enlightening about all that.

Agile is also extremely costly and wasteful, particularly of time. Take an average entry level developer salary, which is not trivial these days, and tabulate up what 15 minutes for a stand-up every day equates to in terms of actual dollars [1]. The false assumption is that such planning was going to have to happen anyway, but it’s not really true. For all the clamoring that Agile is more streamlined with planning, I’ve always found rapid iterations to basically result in more total  planning time over the long run, kind of like the fractal behavior of measuring the coast of Britain or whatever. It artificially feels like mini-planning every few weeks keeps you lean and focused on just the important stuff, but it’s a mistake. You aren’t aware of your global context or what the whole user story feedback loop is steering you towards, and so frequently the rapid planning is needed to pivot away from a bad development path that would have been avoided entirely with more long-term, up-front planning. I’m not saying that there should be 3+ months of Ivory Tower planning before you start a project, but the extreme micro-planning alternative is also too much. It ebbs and flows, changes by personnel and nature of the project. No framework will solve every problem, so why not be open to occasionally approaching something with a longer-term, research-focused type of planning period? Why always preclude it by the very nature of the tightly maintained iteration schedules?

From a philosophical point of view the lack of global context is the dealbreaker for me. Software is a fundamentally creative enterprise and the whole reason why it’s worthwhile to incur the insane cost of the complexity of software is because you can design it, meaning you can use intelligent, optimization power to consider the vast space of possible designs and you can intentionally and deterministically and predictably cause your work to steer you into a pocket of design space that meets your many needs.

When you agree to be steered by a feedback loop essentially made of a lightly business-filtered stream of customer feedback, you throw this optimization power out the window. You throw away something deterministic and efficient, like gradient descent, and you replace it with something random and inefficient like simulated annealing. Quite literally, you are making somewhat random tiny changes and sitting back to witness whether customers randomly accept or reject them. You are evolving your product in a manner more like Darwinian evolution, which is not sensible when the alternative is direct, creative optimization. If the nature of the design problem is so challenging that you are forced to optimize in this way, it should be lamented and not celebrated as if it was a superior business strategy. And you should be vigilant in a search for how to rescope the problems such that they are amenable to direct engineering, rather than control-system-steering-from-user-feedback.

There’s also a lot of the whole “the road to Hell is paved with good intentions” in Agile. Earnest tech managers may really want to prioritize refactoring workflows, re-designs, optimizations, etc. But all it takes is some slippage, some important business deadline for Q2 that has all the sales people in a nervous mood, and suddenly no matter how much you had been promising to take that big refactoring item off the back log and turn it into some legitimate user stories for the upcoming sprint, you’ll have to break the bad news again to everyone that you’ve got to hack even more new features on top of the bad infrastructure. This is not necessarily Agile’s fault, but Agile also doesn’t set up any kind of a framework that actively makes this entropy growth less likely. Part of the reason is that to do that — to create a brand of QA that explicitly makes it hard for short-term business concerns to override refactoring concerns — is antithetical to the Agile metrics (and this is not good for the middle managers who require such stats for their own advancement) and the time frame of Agile workflows.

Lastly, and this one is just my own personal gripe, Agile is infantile. Agile as a system oppresses me: it says that the system doesn’t care if I am clever, or if I have thought of a great new way to implement something, or if I have thought of a clever hack that I want to try out, or a new gadget that I can code up and share with my team. It discourages tinkering and discourages creativity big time, especially big picture creativity about the global optimization of the project. Agile infantilizes you because it says don’t be a whole person. Don’t bring all of your skills to bear on your work: only bring that minimal set needed to solve the specific user stories shoveled onto your plate. In fact, leave every other part of your professional self at home so as to avoid burning out, feeling bored, feeling shunted from progress, or otherwise becoming unhappy with the cog-like way you are expected to churn through discretized, measured progress. This might be necessary if you manage 10,000 programmers, most of whom suck, but it also means you will never ever be able to get those few really good programmers who can make all of the difference.

In response to the inevitable contortions made to defend whatever “pure Agile” might mean apart from Scrum and things like Scrum, I feel this captures my sentiment: “whenever a majority of users tend to misuse a tool, beyond a certain point it becomes the tool’s fault.”

If Agile (or any given framework) can be easily subverted by politics, then regardless of how good its theoretical intentions are, the framework is a failure. Given that nothing about the basic Agile principles addresses a need to explicitly encode a mechanism that makes it hard for short-term business needs or political concerns to supersede legitimate engineering concerns, and in fact that the Agile tenet of “Responding to change over Following a plan” can be viewed as an endorsement of exactly the opposite in many cases, it’s not particularly surprising or even interesting to point out what a failure Agile is, and it’s not disingenuous nor extreme to flatly say that Agile does not add anything to our communal wheelhouse that common sense had not already put there. Yet it does subtract a lot by way of all of the previous criticisms.

To restate it: there is nothing disingenuous about flatly saying that Agile fails, that Agile adds nothing worthwhile, that Agile is very easy to subvert, and that as a result Agile is not worth defending (certainly not defending in contrast to the likewise idiotic doctrine of Waterfall), and I don’t feel the least bit like I am being one-sided or extreme in saying so. I don’t give Agile any kudos. It didn’t fix anything and the creation of Agile does not deserve credit or praise. Young developers should boycott jobs that will debase them with mandatory adherence to Agile or any other fixed, reactive prescriptive framework that short circuits their ability to learn by tinkering on the job. The opportunity cost of accepting such jobs is often too high. Society at large should be pissed off over all of the aggregate value that has been frittered away due to Agile. That some value has nonetheless been created from within Agile implementations (indeed, in spite of Agile) is no credit to Agile.

To me, the many responses that faithfully recite the so-called Agile principles as a defense of the desirability of Agile always read as a No True Scotsman fallacy. If you want to define the term “Agile” to mean “only good things and never bad things” then “Agile” is vacuously always good and never bad.

All of these principles that Agile tries to appropriate for itself are desirable. I don’t know of any formal software development process, whether it is Waterfall or Agile or any other, that would ever claim to explicitly deny any of those points. Where in Waterfall does it say to deliver late, crappy, bad software that doesn’t meet the client’s new specs? Yet some development processes are better than others at meeting these goals.

Agile doesn’t own the copyright on whatever good quality software is. The idea of delighting your customer, planning when you need to plan, requiring specs when you need to require specs, using common sense, collaborating with business domain experts, … these things have been around since the dawn of commercial software itself in one form or another. They weren’t just dreamt up in 2001.

If someone wishes to define “Agile” as == all good software quality standards, more power to them. It becomes a useless definition and reminds me of the Yudkowsky quote, “If you are equally good at explaining any outcome, you have zero knowledge.”

I choose to define Agile by whatever common business practices emerge in its presence and I am confident that if Agile is defined this way, it removes the mask and reveals the ugly truth that as a system it merely pays lip service to quality while being all too easy to politically subvert.

[1] Here's a quick and dirty estimation of the cost of merely just the daily standup meetings that are prescribed in most implementations of Agile. Let's suppose conservatively that you've got about 5 engineers on a single team (in practice this is all over the map, even ranging upwards of 30 in a team I was a part of, though we never officially tried to have all 30 in the same standup). Junior to medium level developer salaries in most major metro areas are going to be between $\$$85k and \$$120k, let's call the average$\$$100,000. And let's say that you've got two more senior developers, with average salaries maybe in the \$$150,000 range. Again these are conservative estimates.

If we assume roughly 250 working days per year and 8 hours of useful productivity per day (again, conservative), this means the junior devs are paid $\$$12.50 for their time in a 15 minute meeting. The senior devs are paid \$$18.75. This amounts to$\$$3125.00 per year per junior developer and \$$4687.50 per year per senior developer. With 5 junior and 2 senior developers on a team, the team's total annual cost just for the daily standup meetings alone is $\$$25,000.00. That's 25 grand per Scrum team per year just for one daily meeting. And we're not even factoring in the costs of providing insurance, commuter benefits, etc., which will make the actual annual cost considerably higher. If we assume this ratio of 5 junior devs to 2 senior devs is roughly constant throughout an organization, it means that on average every 7 developers are costing you at least \$$25k per year just for Scrum standups. If you employ 70 developers, that becomes$\$$250k. If you employ 700 developers, it's \$$2.5 million. If you employ 10,000 developers, you're talking about more than $\$\$35 million dollars. Just. for. standups. (not even counting insurance or other benefit costs, nor whatever productivity drag there might be just from having the meetings.)

How to Write a Technology Job Ad

We are seeking an idealistic, quality-driven technologist to think creatively about unstructured, high-level problems and the trade-offs needed for their pragmatic solutions. If you have the maturity level to avoid considering yourself a ninja or guru of anything, read on!

You'll love working in our private, quiet offices from 8am to 4pm Monday through Friday, where we don't stock any beers or gourmet coffee!

You (an adult human mammal) are expected to figure out your own meals, but don't worry: we pay you enough money to eat and also to afford a structure to protect you and your family from the weather while you eat.

We're looking for someone who gets things done, such as the things in the job description that we mutually agree on and not other, surprise things.

Do you shop at Target instead of Wal-Mart? Well then you probably have more than enough higher education for us.

We don't want to know which buzzwords you selected for your resume. We just want to talk to you about your skills and experience and see if we can collaborate like amicable adult colleagues.

Benefits: actual insurance, actual vacation time, actual help for retirement.

Because it's 2015, please submit your resume online and don't fill out 6 pages of info that we could just read from your resume... we'll do that.

Thanks!

Saturday, January 3, 2015

A (an?) Haiku

Inspired to totally rip off a phrase from this:

I am a quilt of // loosely cobbled-together // coping strategies.

Tuesday, October 21, 2014

I've been unemployed for just over four months. I've been searching for jobs every day: I've conducted over 30 phone interviews (not counting the dozens of third-party recruiters I've spoken with), filled out well over 100 applications, participated in more than 10 later-stage on-site interviews, and rejected 5 formal job offers. It's definitely stressful and demoralizing to go through such a difficult searching experience.

I spoke recently with a third party recruiter who did not like my preferences: I expect to have a job with satisfactory compensation and benefits, satisfactory vacation time, a healthy work/life balance, freedom to use computational tools that are best for the job, the opportunity to gain quality work experience, the opportunity to learn new things, and the ability to work in a suitably quiet and private space to facilitate software development productivity.

I believe these are minimally acceptable preferences: a job that fails to meet even a single one of these items is not just less than ideal; it is unhealthy and should be flat out rejected.

The recruiter asked with a smug arrogance, "So, how's the job search going?"

"Great," I replied. "Here I am enjoying the fact that I'm not stuck in a horrible job."

Monday, October 13, 2014

Real Technology Agnosticism

Consider the two opinions:

"Use whatever tools will make you the most productive. If they cost a lot of money, just talk to us and explain why the tool is important. If we can afford it, we will. If we can't afford it, we will make the best compromise with other tools we can, and we'll work on affording the better tool. If it involves changing something that is a standard in our company, meaning that many people and established processes would have to change too, we're not going to adopt the change quickly -- but we will listen to evidence about why the change would be cost effective and we will be open with you about considering it."

"Use only the tools that we have decided, as a company, to use. These tools are our technology conventions. Whether you like them or not, they are the tools available to you and you will not be considered to be a 'team player' unless you find a way to get your work done with the tools we give you. If you recommend changes we will view it as a waste of your time to even contemplate tool changes, even if you have evidence of their cost-effectiveness. Changing something that is already a standard within the company is impossible unless the idea originates with senior-level employees; the more you ask about changing established policies, the more you will be viewed as uncooperative."

Which of these attitudes falls more under the banner of "technology agnosticism"? In most bureaucratic settings, the second attitude is trumpeted as a pragmatic, technology-agnostic view point. But really it is an excuse to avoid dealing with the consequences of depriving talented workers of affordable, productivity-enhancing technologies (generally for political reasons, like deflecting blame with standards, and emphatically not in the name of legitimate business concerns).

If anything, it is technology dogmatic.

Meanwhile, the first attitude strives to be actually pragmatic rather than merely paying lip service. If a better tech tool is available and affordable: just use it. If it's not affordable: justify it with numbers and be content to wait until it's affordable. If the scope of the change is massive within the company: expect that you will need to present equally massive evidence that the change is beneficial, but also expect that we will appreciate it when you do present this kind of evidence.

That sounds much more technology agnostic to me -- not to mention less dehumanizing, more pragmatic, and fairer. And you will have the benefit that technology policies will be influenced more by technology experts in an organization than by managers.