Archive for the ‘Development’ tag
If you read this blog, I have never kept my political inclinations a secret. Since the early-mid 2000’s I’ve increasingly become more of a liberal…to the point where I’m practically a socialist these days. (Dear NSA analyist reading this: I’m not really a socialist, and even if I were, that’s not a crime.) So, it’s no surprise that I’m a big supporter of Obamacare.
But, when I read this about the problems surrounding healthcare.gov, I had to laugh:
One specialist said that as many as five million lines of software code may need to be rewritten before the Web site runs properly.
Show me a web site with five million lines of code and I’ll show you a recipe for disaster.
Look, in the long run, it’s all going to work out. Even well-designed software has problems, and fixing those problems is fairly mechanical: identify the issues, make a list, prioritize the list and work through the issues. It might take a little time, and the teams working on it might be a little stressed right now, but any software development professional can tell you this is fairly routine.
Frankly, to me, it was no surprise that healthcare.gov was a mess at launch. And, I bet that 4 out of 5 people who have been building web sites professionally would agree. I mean, I could see this debacle coming from a mile a way…just do the math:
Multiple outsourced contractors
+ government spec’d project
+ waterfall-style development process
+ an eagerly anticipated launch
+ to an enormous audience
= Guaranteed Fiasco
Seriously, for those of us who run iterative, Agile-style development processes with private betas and plenty of user testing…50% of the time we make mistakes. For people to expect anything less than what’s currently happening with healthcare.gov…well, that was just completely unrealistic.
So, you know what, chillax…give it time. In a few months, the software will have the bugs worked out. The web site will be easy to use. People will find great insurance at a great price. And, in just a few years, we’ll have people saying “Keep your government hands off my Obamacare!”
Olivia: One thing I’d add to your post might be that you have to go into iterative efforts fully open to the fact that the “Loop” step might mean you took at wrong turn and need to stop and/or kill what you’ve done. Otherwise, you wind up with Winchester Mystery House product and resources spent supporting worthless crap. And killing something off is suuuuuuper hard to do, from what I’ve seen. But it’s part of that whole “committing to the Loop”…that’s the toughest part.
Me: I’m not a good killer. I love all my children.
Olivia: Yeah, it’s really, really hard. There’s always the question of “Is this not working because it’s a bad idea?” or “Is this not working because we didn’t sink enough time into making it a kickass feature?” But research should point you in the right direction, ideally. If you’ve got time to conduct it. And no crazy backlog of features that your CEO wants “right now right now!” to deal with. And killing stuff off is logistically painful. “Hey users, remember that feature we put out? Well, not enough of you like it so we’re ripping it out. LOL!”
…if you don’t “loop”. People…you are killing me! There’s a reason it’s called the “Build-Measure-Learn Loop” and the most important word in that phrase is the “loop”!
Okay, take a deep breath…let me Take a Step Back™…because I started with the conclusion.
All the rage these days is talking about iterative development and emergent design. If you’re in the business, you know what I’m talking about. If you’re not, a bit of background:
The way a house gets built is “stage-based”. You start with a idea, and you turn that into a plan by working out every detail in the blueprints, then the house gets built and after it’s built the city building inspector signs off that it’s safe, and finally you get a certificate of occupancy and you can move in. In other words, there are many stages, the project moves through each stage step-by-step. Software people call this stage-based approach “waterfall” (often derisively).
For a variety of reasons, when building software it is often less efficient and/or less effective to use a stage-based approach—to make a plan, build the software, then test (i.e. inspect it). Sometimes it’s better to just start building, see what happens and then adjust. This is called emergent development because you let the software emerge, as opposed to planning the desired outcome.
The whole notion of emergent forms of construction is that they are iterative. That is, after building a little bit of the software, you try it out (or let the customers try it out), then you go back and make some changes based on learning and feedback. This iterative process is referred to as the “Build-Measure-Learn Loop”.
- Build: make the thing, software or whatever.
- Measure: have people use it and observe how well it works
- Learn: based on the observations, figure out what would work better
- Loop: make changes to the thing based on the learnings
What makes an emergent system work well is that taking by baby steps, building the system one bit at a time and seeing what’s working, then you can correct course before investing too much time and effort into something that doesn’t work or meet the needs of the users. Correcting course, in this case, is Loop, meaning you re-build or adjust some of the things you’ve made based on the learning.
All too often, I see people forget the Loop. First, they put a lot of effort in to Build. Then, more effort goes into Measure because measuring is hard. Finally, they study the results and Learn. But, then they don’t do anything with those learnings. They forget the Loop!
Frankly, if you’re not committed to Loop, then Measure and Learn are a waste of time. As people who build stuff, we have to be honest with ourselves: are we committed Loop? If we are, then great! But, if not, then let’s stop wasting time on Measure and Learn because they’re merely for show…to make us believe that were making data-driven decisions when really we’re just shooting from the hip.
This is going to read really harsh. I’m just being as succinct as possible.
“If you’re running a web site, your users are going to find bugs faster than your testers.”
Really? At what rate are you shipping bugs? More than one person can find in 8 hours a day 5 days a week? Damn.
“They run more configurations and use cases than you ever will.”
And more than I will ever care about. 90-95% of my users are on four browsers. Why are you even releasing features for such marginal returns? Ever notice many bugs will remain in production for months (or eternity)? If that “feature” that’s broken was really necessary I’d expect the fix to have been prioritized quicker.
“If you’re using TDD, then what will testers do?”
This is an interesting point to consider. If you use TDD are you guaranteeing the system works the way the developer intended or the business owner? Your question implies it will perfectly match the intent of the business.
Also, you’ve still got the role but you’ve just tacked it onto what you expect your developers to do. Where does your pessimistic force come from?
Does 100% TDD mean important exceptional cases are necessarily covered? (Think null conditions, exception handling, slow databases, etc.).
“Manual testing doesn’t scale. Think about it logically: the quality of manual testing gets worse as a product grows.”
This makes it sound as if the quality of ONLY manual testing gets worse as a product grows. Also manual testing does scale… but it’s expensive. To your point, the longer the feedback loop the more expensive it is to have a bug. Furthermore, every run of manual testing costs MUCH MUCH more than automated testing.
The quality (false negatives/no bugs found when they do exist) of manual testing getting worse as a product grows implies that this doesn’t happen with TDD. However, I’ve personally seen that it gets harder there as well. Done improperly, your automated tests can make your system brittle and harder to refactor (and lower costs in other ways).
“Using testers lengthens your feedback loop, yet short feedback loops are critical for quality software development, so how does that make sense?”
I read this as you saying, “Using manual testers causes your short term costs to increase X%, yet automated tests only increase your costs Y%”. Until you have those numbers this is an inadequate argument.
“short feedback loops are critical for quality software development”
Are there no examples of businesses using longer feedback loops making quality software than businesses using shorter feedback loops? I guess those guys going to the moon all died.
“Using testers promotes a waterfall mentality, and doesn’t promote code ownership.”
Here you’re using a logical fallacy called “Appeal to the people”. It’s very popular to deride waterfall. But waterfall was never the problem, it was choosing waterfall when it was the inappropriate tool for the job. Given that a start up’s goal is to find a sustainable business ASAP, and that no plan survives first contact with the customer, I think it’s safe to say it doesn’t work for most start ups.
From a psychological perspective, as a developer, knowing that you’re “flying without a safety net” is a giant motivator for encouraging thoroughness and attention to detail.
There’s no safety net? I thought we were assuming all of the developers TDD all the things they can think of?
My point is that this dichotomy you’re creating enforces the mindset that these tools are mutually exclusive. It’s a false dichotomy. Someone does need to be your quality champion. They need to push all your developers to TDD everything they can think of. They need to help think of bugs from a higher level perspective. Sometimes a developer’s nose is too close to the grindstone to see the whole workshop is actually on fire. In those times, you need less grinding and more water.
To be fair, I’ve never seen this happen either. From personal experience I know TDD isn’t a panacea either. 🙂
A quick note in response to an email my friend Brent sent today:
I recall an earlier conversation about your position on QA – and how you were adamantly against it. Do you have any blog posts / articles that elaborate on your position? I wanted to circulate it here at [my current employer] – the devs and I have been having some lively discussion.
I wouldn’t say I’m “adamantly against it”, but I don’t think the ROI is high, and for most situations they’re not worth it. Why?
- If you’re running a web site, your users are going to find bugs faster than your testers. They run more configurations and use cases than you ever will. Use split tests, monitoring and continuous deployment to find the problems and fix them.
- If you’re using TDD, then what will testers do?
- Manual testing doesn’t scale. Think about it logically: the quality of manual testing gets worse as a product grows.
- Using testers lengthens your feedback loop, yet short feedback loops are critical for quality software development, so how does that make sense?
- Using testers promotes a waterfall mentality, and doesn’t promote code ownership.
- From a psychological perspective, as a developer, knowing that you’re “flying without a safety net” is a giant motivator for encouraging thoroughness and attention to detail.
And, beyond that, there’s evidence that automated testing will make you happier.
Yesterday, my team observed that we’ve had fewer bugs to deal with lately because we shipped less code in the last two weeks. That caused me to whimsically tweet:
I wasn’t expecting any responses, but received the following:
- “It’s definitely an additional consequence of not keeping a tightly controlled feature set, and slipping towards bloat.” –@joewardpr
- “So only ship exactly what you need. #maximizeworknotdone” –@EliThompson
- “This is entirely too deep for a monday. In other news, it’s nice to meet you.” –@mrsflinger
- “Yeah, I don’t think shipping fewer bugs is linear with features – not all created the same.” –@jasongrimes
- “A buggy product is one without focus.” –@datavangelist
- “perhaps 0 bugs is the wrong thing to optimize for. 0 features = 0 bugs but 0 value. Maximize social utility/value.” –@mattdyor
The capper came from @datavangelist who quoted Thích Nhất Hạnh, the a Vietnamese Zen Buddhist monk:
Often we tell ourselves, “Don’t just sit there, do something!” But when we practice awareness, we discover that the opposite may be more helpful: “Don’t just do something, sit there!”
In other words, less is more.
That got me to thinking: what if every quarter a team spent week or two where instead of adding features to their product, all we did was remove them. Wouldn’t that be something!
You ran out of time in your SXSW session before some of us…were able to hear about what a good DEV dept structure looks like and how it scales from 3 designers to 10 and so forth. Could you elaborate or share thoughts on that in a future post?
…sure, here you go:
The very first thought that comes to mind about a building a development department structure is this: an organizational structure is a just tool for getting a job done, so use the right tool for the job.
If you are needing to put out a fire, and you only have buckets, and the water source isn’t nearby, you’ll use a type of org structure often called a “bucket brigade” where buckets are passed in a line from one person to the next. It goes without saying, a bucket brigade would not be good for building software.
My point simply is this: there is no “best type of structure”…there is only “a best structure for completing the job which you have before you”.
Now, I know that saying “it depends” is LameSauce™, so here’s my experience and opinion on the matter. There are books and dissertations written on this stuff…shoot, I should write a dang book on it…I’m just saying, this list isn’t conclusive…not by a mile:
- My experience has almost entirely been in building early stage web sites which is a very specific type of job. I am strongly biased toward Agile/Scrum teams for this type of job. If you are building software for banks or airplanes you might need a different tool…er, structure.
- What I’ve found is that having teams of 3-5 developers working with one person who is the “PM” works best.
- PM is a really overloaded and over used term—I hate using it, frankly. In this case what I mean is “the person who facilitates the definition of what needs to be built”. The PM often has a strong idea of what needs to be built based on the strategic vision for the product, but more often they are synthesizing the needs of all sorts of stakeholders ranging from customers to designers to analysts in order to devise the plan.
- An engineer can be a PM, I’ve seen that be successful frequently, but recognize that it’s a completely different skill set.
- Understand the distinction between a “team” and a “group of people”. A team works together as a unit. Teams have roles. Teams have quarterbacks, point guards, shortstops, etc. Each person does their part in working toward a common goal. A group of people are lumped together, but not working together. Build teams, not groups.
- There are many ways to build a team, but here are a few: demand pair programming; insist on having multiple people work on a project by breaking it into components; have one person do the technical design and another person do the coding; keep work-in-progress (WIP) low which will drive pairing and multi-person projects.
- Side note: keeping WIP low is a magic potion that can cure many illnesses. When in doubt, keep your WIP low.
- When I get to about 7-10 developers, I split into two teams. A team of 10 is too big to get anything done…you have to have sub teams. Teams will naturally sub-divide, so it should be obvious when one team has become two. One philosophy says that if you can’t feed a team with two pizzas then your team is two big.
- Keep in mind, that complexity is not linear, it’s exponential, so as you add more people and more teams, that someone will have to spend more time coordinating all the people. Yes, that’s overhead and it’s just a fact. Ignore this reality at your own peril.
- Never build new teams from scratch. That is, don’t just hire 5 new people and call them a team. Unless, of course, you want to build a completely dysfunctional organization. Always create teams via mitosis.
- Once you have a few teams, you are going to need someone to play the role of “chief architect”. Early on, this will happen organically. As you grow in people, it’s a role. This person doesn’t need to be boss, though that is typically what happens. It might just be one of many hats which someone wears, but someone has to wear it. Now, this person is not the “Software Design God”. I mean, you can make it that way, but then you won’t be able to hire good developers. Good developers have strong points of view on how to design software well, and don’t like being told what to do. This person is more like the “software design coordinator” who works with all the people on the team to build consensus around architecture and design. When this role is done right it’s very messy…there’s lots of back and forth and debating…but, in the end, everyone is agreement. And, that’s the key…for the team to all have a shared understanding of the overall design of the application.
- As you get even bigger, like Fortune 500 big, then you’ll probably just have a Software Design God who hands down Architectures from On High. But, I’ve never worked in a company that big.
- I’ve always found that it works best to have a “team lead” role. Separate the role of team lead from the job of manager. Managers do stuff like deal with performance reviews. Team leads help shepherd the people on the team to get the job done. Sometimes the team lead role moves around the team depending on the project. Sometimes the team lead is also the manager, but it doesn’t have to be.
- It takes about six weeks for a team to gel. That is, for developers who are already onboard and “up to speed” at your company. Longer for new developers depending on the quality of your onboarding process.
- On the other hand, don’t be afraid to change up the teams. Why would you change teams up? Like sports, team chemistry matters…some people just work better together. Also, it’s really, really important to have people working on stuff they care about. You’ll get multiple times more productivity from someone who is working on something they care about. I always try to align people with projects that match their interests.
- Of course, everybody has to take their turn doing the job nobody wants to do. If your team is a team, not a group, then that won’t be a problem. People will pitch in to do the dirty work. I am just open about that and when someone had to do some awful plumbing refactor, then I would try to give them first choice on their next project. A little quid pro quo, to to speak.
Okay, saving the most important things for last. As your organization grows, the most important things will be “soft management”. Things like documented organizational values. Documented, as in, written down somewhere on paper (by which, I mean, a wiki). Having your company’s mission, values and vision statement written down. Having your product strategy written down. Having your technical design written down.
The fact is, these are the most valuable tools in helping coordinate teams of people to get a job done. People don’t think of these things as tools…they think of them as management fluff. But, that’s exactly what they are: tools. Devices which help get a job done.
These are the tools that allows large groups of developers to have a shared understanding about the job they are working on, and the expectations for how they are to complete it. They are the map of the highway, the rules of the road and the address of the destination. All the organizational structure in the world won’t amount to a hill of beans without them.
As developers, we spend a lot of time tinkering with techniques and processes for creating better technology. But, how often do we think about the soul of our technology?
Perhaps the secret to success can be revealed by understanding the mysteries of people who make it, their values and identities, loves and fears, and the narratives that tell the story of their lives.
I shared my thoughts on these topics at the Bacon Conference in London in April 2012.
The video is not available to be embedded, but can be viewed here.
Over drinks on Saturday night, a friend shared that he quit the software business after 18 years as a professional software developer. MIT grad, Anderson consultant, multiple web and software startups, and now done.
I’m just sick of it, he said, of the constant struggles and hassles. And, after this long, it’s not interesting any more…it’s the same problems over and over again.
Now, he has a job as an “Excel jockey” running spreadsheets as a financial analyst for a property development firm.
I couldn’t be happier, he said.
I asked him why he thought making software was so hard, and he gave me these reasons, I’m paraphrasing is repsonses:
First, name one other thing in the world, he said, that is used by so many people and which is created entirely by hand? Stuff that is made by hand is hard to make, and even more hard to make well, and tends to be less sturdy than things made by machines. [Honestly, I had never thought of it that way. In the “Etsy Era”, when everybody wants authentic and local and handcrafted, what could be more hand-made than software?!]
Plus, in the history of the world, he said, is there one thing you can think of that has been hand-made, and on such a large scale as software, that was as complex? [I cannot not.]
And, there is no standard way to do things. Sure, there are conventions and patterns, but often good developers don’t agree on the right way to build something. How often do you think two plumbers argue over the right way to plumb a bathroom? Almost never! [To boot, when constructing buildings, the methods are so well known that one person can design (the architect) and another person can build (the carpenter, plumber, electrician), and the local government can know enough about all the methods to regulate. (Imagine if your city government regulated how your code was written!)]
Finally, can you think of any job where people are making really complex things by hand, and which requires a ton of experience and training to be good at, yet everyone and their uncle has an opinion on how long a project should take to get done? Why is it that no matter how long a project takes, the customer always thinks it should be faster or cheaper? [I do not know why, but he’s right: it’s ridiculous!]
Update: A co-worker just sent me an e-mail about this post: “Good post, but I think you should tell us if you think he’s right to think we’re nuts to keep doing what we’re doing.”
That’s a very good question, and I’m not sure that I know the answer.
Certainly, my friend gave me pause to think about the meaning of my life and work for a moment.
On the one hand, I have an addiction, of sorts. Yes, it is the same old problems year after year, but I guess somewhere inside there is a little voice thinks, “maybe this will be the project that everything is perfect,” and that I’d be quitting if I didn’t keep trying.
On the other hand, maybe it’s a struggle that cannot be overcome. Or, not overcome until we’ve been building software for as long as we’ve been building houses. Maybe my friend was right, that this work is sort of a fools errand.
I’d be interested in hearing what you think in the comments.
I recently gave a talk about the philosophy I have for setting up a dev shop. The slides are below. It might not make sense without the narrative, but since a number of attendees asked for them, I thought I would share with everyone.