Why do web sites and software take so long to build? And why is it so hard?

with 105 comments

Over drinks on Saturday night, a friend shared that he quit the software business after 18 years as a professional software developer.  MIT grad, Anderson consultant, multiple web and software startups, and now done.

I’m just sick of it, he said, of the constant struggles and hassles.  And, after this long, it’s not interesting any more…it’s the same problems over and over again.

Now, he has a job as an “Excel jockey” running spreadsheets as a financial analyst for a property development firm.

I couldn’t be happier, he said.

I asked him why he thought making software was so hard, and he gave me these reasons, I’m paraphrasing is repsonses:

First, name one other thing in the world, he said, that is used by so many people and which is created entirely by hand?  Stuff that is made by hand is hard to make, and even more hard to make well, and tends to be less sturdy than things made by machines.  [Honestly, I had never thought of it that way.  In the "Etsy Era", when everybody wants authentic and local and handcrafted, what could be more hand-made than software?!]

Plus, in the history of the world, he said, is there one thing you can think of that has been hand-made, and on such a large scale as software, that was as complex?  [I cannot not.]

And, there is no standard way to do things.  Sure, there are conventions and patterns, but often good developers don’t agree on the right way to build something.  How often do you think two plumbers argue over the right way to plumb a bathroom?  Almost never!  [To boot, when constructing buildings, the methods are so well known that one person can design (the architect) and another person can build (the carpenter, plumber, electrician), and the local government can know enough about all the methods to regulate.  (Imagine if your city government regulated how your code was written!)]

Finally, can you think of any job where people are making really complex things by hand, and which requires a ton of experience and training to be good at, yet everyone and their uncle has an opinion on how long a project should take to get done?  Why is it that no matter how long a project takes, the customer always thinks it should be faster or cheaper?  [I do not know why, but he's right: it's ridiculous!]

Update: A co-worker just sent me an e-mail about this post: “Good post, but I think you should tell us if you think he’s right to think we’re nuts to keep doing what we’re doing.”

That’s a very good question, and I’m not sure that I know the answer.

Certainly, my friend gave me pause to think about the meaning of my life and work for a moment.

On the one hand, I have an addiction, of sorts.  Yes, it is the same old problems year after year, but I guess somewhere inside there is a little voice thinks, “maybe this will be the project that everything is perfect,” and that I’d be quitting if I didn’t keep trying.

On the other hand, maybe it’s a struggle that cannot be overcome.  Or, not overcome until we’ve been building software for as long as we’ve been building houses.  Maybe my friend was right, that this work is sort of a fools errand.

I’d be interested in hearing what you think in the comments.

Written by scottporad

May 6th, 2012 at 11:21 pm

Posted in Miscellaneous

Tagged with ,

105 Responses to 'Why do web sites and software take so long to build? And why is it so hard?'

Subscribe to comments with RSS or TrackBack to 'Why do web sites and software take so long to build? And why is it so hard?'.

  1. [...] Why do websites/software take so long to build? And why’s it so hard… 1 Upvotes Discuss Flag Submitted 1 min ago Rand Fishkin Entrepreneurship scottporad.com Comments [...]

  2. You need to spend some time on a job site, I’ve seen plumbers come to blows arguing over the best way to do something.

    Not saying your points is wrong, but the metaphor is broken.

    Fuzzy

    7 May 12 at 12:10 am

  3. There is a bug here. The software is made by hand. But is replicated by machines. You take any machine made thing that is used world wide, the first version is guided or made by hand/brain. Then the replication is done by machines.

    Unni Koroth

    7 May 12 at 12:12 am

  4. I can certainly relate to your friend’s frustrations. Over the past 14 years, I went from being a developer to a code tinkerer. I enjoyed keeping up on the latest development tools but was much more focused on the business strategy behind the project and the impact on society.

    Now I do communications and new media for a nonprofit and do the code tinkering and innovation on my own personal blog.

    Jesse Luna

    7 May 12 at 12:18 am

  5. 200 years ago, every machine in existence was hand-crafted. Software is still a very young industry, and it’s moving at break-neck speed. We’re a long way from where we were 20 or (shudder) 40 years ago. Take a look at http://www.automatedbusinesslogic.com for an example of automating software development. I’m not saying everything’s peachy, but it’s gotten so much better, I’m surprised someone would just give up.

    Max Tardiveau

    7 May 12 at 12:38 am

  6. I think programming is still in it’s infancy, in time things will become much more automated and take much less time.

    We have already seen huge leaps in automation and efficiency in the last 30 years which took hundreds in the comparable building or manufacturing industries.

    Think about how long it would take, and how much code you would write building the same piece of software with punch cards, to PHP, to Rails.

    Just my 2c

    Zen Savona

    7 May 12 at 12:39 am

  7. Your friend seems to be going through a sort of existential crisis. I do NOT intend to dispute his decision at all. Everybody has a right to the decision they make. All I intend to say that almost every work in the world is repetitive in nature, and the love for work doesn’t necessarily last forever.

    Bipin Upadhyay

    7 May 12 at 12:51 am

  8. [...] great post was by Scott Porad about “why do web sites and software take so long to build and why is it so hard“. An interesting read – makes you think about your role in life. Definitely the must [...]

  9. When I had my bathroom remodelled, at first I thought it was organized amazingly well and why couldn’t software be like that? There was a designer from Expo, a primary contractor and subcontractors for tiling, painting, etc. a project workbook containing all the documents, including the design, and a logbook for every contractor to record their visit. But it turned out to be just like a software project. As soon as they started ripping up the walls, the painstakingly drawn design for the tub/shower was hastily and arbitrarily adjusted because the casing for an outside fire extinguisher was in the way. Many of the components turned out to be incompatible with each other and many were not the ones originally ordered (the bathtub was not even the one labelled on its box!) Communication was terrible – the city inspector would tell the primary contractor to change something, and then a subcontractor would show up and ask me what he was supposed to do. When there was a disagreement about whether the tiling was done properly, one of the contractors disfigured the tiling to force another contractor to redo it. They all had other projects, so for long stretches I had a pile of dirt (for the cement) in my patio and a non-functioning bathroom for months. And toward the end there were some quick patches, e.g. the walls were slightly curved so edges of the tiles looked bad and they just painted the patches, which worked but cost me extra. And a couple of years later when the tub sprung a leak, the plumber couldn’t figure out how to get in there, said they must have done a shoddy job on connecting the pipes, then when he finally got a good look at it, realized why they did it that way. It was exactly like a software project!

    Phil Chu

    7 May 12 at 12:56 am

  10. The reason software is difficult comes down to one thing: change.

    If broadband never became mainstream, if processors never became faster, if mobile phones never became “smart”, if HTML5 never existed, our jobs would be a hell of a lot easier. Everything would be automated and efficient.

    But no, every time something changes & advances, we have to start all over again.

    How often do you think plumbing technology changes?

    Justin Ko

    7 May 12 at 1:24 am

  11. Phil has got it right. I think it’s nonsense to put software development in that category as if it was alone there – only the simplest things can be automated, and as soon as a human hand is required, chaos is likely to ensue. You can see this in software and plumbing, in graphic design, in public administration…Certain tasks (eg. running a bar, or a car factory i suppose) reach a certain stability once the creative work of making it all fit together is over and the human challenge consists of going through the motions. But neither software engineers nor plumbers ever reach that state, because it’s against the basis of their work.
    I mean not even plumbing – house cleaning should be the simplest thing in the world, right?Yet every second person I know with cleaning service is constantly complaining about how they can’t reach to satisfactory conclusions.

    Mario C

    7 May 12 at 1:31 am

  12. Something is clear for me: You(both) don’t have experience making bathrooms. The person that “designs” better supervises the person that builds, or disaster will come.

    Programming is per se the most automatic job that you could have, you can automate EVERYTHING, while making walls or chairs you can’t because there are real physical constraints.

    You could even make self writing code!(have you seen a hammer or a cutter that modifies itself?), multiple threads(rulers or bricks that auto replicate). You can create your own black boxes modules to control complexity, and parametrize everything.

    But you need to do it yourself, it is your responsibility to automate when you can’t handle it manually, nobody is doing it for you.

    Stop complaining, is not that hard. You don’t know what “hard” means.

    Jose

    7 May 12 at 2:04 am

  13. Of all the points your friend made, I felt the one about Complexity was the most relevant one. Our puny brains though very powerful, can process only so much of complexity over long periods of time.

    I feel more and more people are feeling frustrated at the increasingly complex systems and are quitting or changing their career-focus midway.

    shashi

    7 May 12 at 2:17 am

  14. I respect your friends opinions however, the difficulty of programming and software development is what makes it fun. It’s a challenge to us. What makes a good programmer in my opinion is someone who can tackle problems fiercely and not stop until it’s done (although the better ones know when to stop)

    Richard Bamford

    7 May 12 at 2:59 am

  15. Echoing some of the other comments, the web industry is both supremely young and constantly changing (at a very fast rate). It may not feel so because we’ve grown up with it but it is a very unique beast.

    iamkeir

    7 May 12 at 3:20 am

  16. Actually, comparing software development with the various activities you mention exposes a fundamental flaw that is unfortunately all to widespread. Software is constructed automatically and it is doe really, really quickly. The correct analog to building something is the compilation/packaging process, and we do in fact do that using a “software machine” a.k.a. a compiler.

    What we think of as software development is actually ever more detailed levels of design — and the most detailed level is writing the source code.

    When you look at software development in this light, you see that the difficulty in estimation is not too different to any analog in the non-software world.

    Jim

    7 May 12 at 3:43 am

  17. [...] Why is Software so hard to build? [...]

  18. Software is intelectual work, like writting a book.

    Mateus

    7 May 12 at 4:22 am

  19. Unless your friend is typing low-level byte code, he ain’t doing anything “by hand.”

    There is nothing inherently unique or “different” about writing software than there is building a building, creating a painting, or performing surgery.

    mauxfaux

    7 May 12 at 4:36 am

  20. There is in another field that does share many of the same problems… Literature. The hard part is that ours has to be readable by humans and a machine. At least with the machine we can get rapid feedback.

    Also many programmers don’t accept that we’re writing literature or that programming is anything like it, but many (like myself) do.

    (just Donald Knuth pfft) http://www.literateprogramming.com/

    JustinBozonier

    7 May 12 at 4:37 am

  21. I don’t remember verbatim, but in his ‘unbearable lightness of being’, Kundera says that life is a sketch. There can never be a ‘finished’ picture.

    I think most frustrations arise from an attempt at perfection or completion that is in essence an unachievable state. Programming lulls us into a false belief that it is somehow different, that there is perhaps an ideal flawless process of creating code.

    On an aside: I quit being an architect (buildings, not software) and moved to code 8 years ago, primarily because of the long timelines and disagreements between the concerned parties of any architectural or infrastructure project. I find programming to be way less frustrating. I commend your friend for taking the step he has. I know how difficult such a decision can be.

    V A

    7 May 12 at 4:38 am

  22. “is there one thing you can think of that has been hand-made, and on such a large scale as software, that was as complex?” LHC. Scientific equipments.
    A CPU or a nanomaterial is as hand-made a software (from the physical point of view, software is a particular configuration of spins in a harddisk. On the theoretical side, construction of a nanomaterials is a lot more harder, since learning quantum mechanics and making physical calculations in a lot more difficult that learning a programming language and writing software with it), and is fairly complex.

    pikachu

    7 May 12 at 4:48 am

  23. That’s why our tagline is “Handmade Software”.

    Stephen Eilert

    7 May 12 at 5:25 am

  24. Why would your friend hope that software development become automated and not done by hand. When that happens very few software developers will be needed.
    I love the fact that it remains complex, because i remain high in demand and high paid..

    joe

    7 May 12 at 6:13 am

  25. It looks like your friend sees to much the ‘builder’ part of our job, but not so much the ‘problem solver’ part. Creatively solving a problem by coming up with a systematic solution is NOT something that can be automated. And those problems are usually so damn abstract that most of the times our clients themselves don’t know exactly what they are, or how to put them into words.

    I think “I need a new house, can you help me?” and “I need a way to decrease production costs, can you help me?” are in very different levels of abstraction.

  26. I wonder if your friend is just attaching explanations to a deeper fundamental issue. When I talk to programmers there’s (in the older ones who have more perspective) a fundamental concern about the impact of their work, and it’s a core issue with Silicon Valley: little of it really matters.

    We just saw Instagram sell for a billion dollars, for an app that takes pictures, applies filters, and shares the pictures with friends. I heard a good summary which I’ll paraphrase as “Silicon Valley pumps out apps so you can find delicious tofu, review the delicious tofu, then snap a picture of it and share it with your friends.”

    Does any of that really matter? You can look at your work and take satisfaction in a job well done, but can you really take pride in what your work means to someone? If you build a house well, you give someone a home. If you do your job well as a doctor, you save a life. I’m not saying all programmers do work that is ultimately trivial, and I’m not saying that those doing trivial work are doing something wrong or that they should be unhappy, but I have seen a number of unhappy software developers who feel like your friend, and a sense of futility in their work is pretty common.

    Mike Hillyer

    7 May 12 at 6:41 am

  27. Extending on your building analogy let look at it from an outsider’s (client’s) perspective. I can personally relate to this because I am having a house built as we speak.

    Client’s can see every single step in the building of a construction project. Foundation pouring, framing, electrical, etc. It is so transparent that client’s understand the work involved with getting to the finished product.

    On the software site, client’s are clueless about the work that goes on setting up the DB, installing the framework, writing (and testing) the code. All they ever see throughout the process is a functioning website.

    Software developers, designers, and managers who downplay how easy are jobs are the reason why we hate our jobs some days. We need to stop saying “that’s easy” when a client asks for a feature request.

    Blain Smith

    7 May 12 at 6:48 am

  28. @Mike Hillyer
    It depends. Who do you think programs the fancy machines in the surgery rooms that save lives?

    Bob

    7 May 12 at 7:15 am

  29. Building on Shashi’s comment about complexity and limited brain capacity–I think it’s interesting to consider how humans typically respond to this type of overload through a process called “chunking.”

    If you’re not familiar, check it out on wiki, but essentially psych researchers have proven that when individuals are faced with remembering or processing large sets of information they tend to group the responses based on the items’ semantic relatedness.

    From an outsider’s perspective it appears that this phenomenon could be related to writing code, as it seems that every line is not written by “hand” but rather “chunks” of code are integrated or reconfigured.

    What interests me most is that the research shows that the groupings we create to help ourselves manage complexity are “highly subjective, as they depend critically on the individual’s perception of the features of the items and the individual’s semantic network.” As a result, how we group things differs significantly based on education, language and culture.

    So, as software development becomes increasingly more complex but also more open and global in nature, I wonder two things:
    1. How will we resolve the chunking conflicts that seem likely to arise (especially in a relatively unregulated and rapidly changing field), and
    2. Will there be any negative implications to our increasing separation from the things we create and the actions they perform?

    Averill

    7 May 12 at 7:21 am

  30. We’ve been houses for thousands of years. We’ve been building software for what, maybe 80 years?

    Tim

    7 May 12 at 7:24 am

  31. While your friend certainly has a point, it is not entirely as he describes, I’d say. There *are* other things and other jobs were you do all by hand (or, do an intellectual work), and the responsibility is very huge (I think this is what he meant when mentioned that many people use it). More related to what we can think of – think of an aircraft assembly for example – most of the job is done by hand. A surgeon? A judge? Being a good father for your child? For all this jobs – there are lots! of responsibility, lots of work to do, lots of choice every day and every time you think you do the same thing. In fact, after those, SW development looks not that much highly-responsible compared to those (well, unless you develop SW for aircraft). And believe me, there are tons of people judging and making their points on how to do this and that.

    And yet, people like there job. I like being a SW developer and a manager of a small team. I think, or hope :) that surgeons like their job (I think you have to like this job to do that), etc.

    I’m not judging on your friend’s choice. Whichever choice he made – good for him, because it is a personal *deliberate* choice – it is good therefore in any case. I’m only saying that this “issue” is not unique to SW development (and agree that this also applies to SW development).

    In the end, just for me personally – I adore working in these conditions, but also feel (yes, I feel that pretty often indeed) that I need a break. One needs to have rest. But some time later after a break – I feel this annoying need :) to get back to work where I can make a choice, where I take responsibility, where I can have a challenge.

    Timur

    7 May 12 at 7:24 am

  32. I agree with many of the sentiments expressed. Web development changes every month, and it’s difficult to keep up. But I am surprised that another troubling yet common aspect of this business was not mentioned. In how many other trades or professions is your work intentionally and maliciously attacked, by dozens if not hundreds of people around the globe, starting just moments after you have completed your job? When’s the last time you finished lovingly building a backyard barbecue only to wake up the next morning and find that someone from another country (who you do not know and who does not know you) has snuck into your backyard, disassembled half of it, and painted the other half bright orange?

    codeWhisperer

    7 May 12 at 7:25 am

  33. The friend experienced professional burn-out. That’s understandable when you’re making *anything* according to someone else’s wishes. When you are crafting something of your own design, then the story and motivation is different. This is why so many software engineers and programmers contribute to open source: they enjoy practicing their craft by following their own muse.

    !!Dean

    7 May 12 at 7:31 am

  34. Hehe. Nice one. A few things that I heard from customers in my life:

    “We want something standard that doesn’t cost much and is perfectly customized to our processes.”

    “Why do you need that much time for testing? Aren’t you any good?”

    “So when we cut of from the design phase, and skip the documentation and the training, and take only half of the testing, the project will be done much earlier and cost way less.”

    _Flin_

    7 May 12 at 7:39 am

  35. @Bob true enough, which is why I said that not all developers do trivial work and doing trivial work is not necessarily bad, just that it can lead to such feelings.

    Mike Hillyer

    7 May 12 at 7:40 am

  36. This whole thing wouldn’t be any fun if there were no problems. That’s what makes programming interesting, finding creative solutions to hard problems. That’s what makes ANYTHING creative interesting.

    Sounds like your friend is better suited to one of those drone jobs that suck the very emotion out of you. Or maybe that’s already happened…

    Tom

    7 May 12 at 7:51 am

  37. Your friend will be complaining about financial analysis soon.

    Anything involving people sucks. People are irrational, ego-driven, uninformed…and that’s in the best case. In the worst, they’re sociopathic, sadistic, passive-aggressive, and just plain evil.

    In any undertaking where people are involved, you’re going to have inefficiencies and frustrations.

    If there’s a trick to finding unfrustrating work, it’s in jobs that deal with the fewest people, but unfortunately no jobs deal with no people.

    The One

    7 May 12 at 8:26 am

  38. At the end of the day it comes down to the people you work with. If the people suck then the work sucks as well and this is amplified even further in the software field because it tends to attract people that are on the dangerous side of the autistic spectrum. The only sure way to mitigate the problem is to work with great people.

    david karapetyan

    7 May 12 at 8:41 am

  39. I wrote this a long time ago but I think it’s still relevant, so here’s my take on why software is hard to build and why we can’t build it like we build bridges: http://agilology.blogspot.com/2008/02/why-we-cant-build-software-like-we.html

    Jeff Tucker

    7 May 12 at 9:23 am

  40. The correct response to the difficulty of development is to help find a way to solve it, not to quit.

    Code generation and meta-development has made my projects at least 10 times more efficient than before.

    Kris

    7 May 12 at 9:30 am

  41. Yea, it’s hard to not get jaded in this biz. I’ve been in it since ’94, having done mainly startups and smaller software companies.

    There are a lot of frustrating aspects to the industry, but my guess is that’s the case with a 15+ career in any industry. With that much experience, you start to see the repeating patterns and the similarities between the different groups you’ve worked with. “Familiarity breeds contempt” kind of thing, I guess. I’m sure doctors and lawyers with that kind of experience can rattle off a litany of systemic problems with their industries.

    But software is such a young thing, and it’s changing so fast that it requires a lot of manual heroics to work right. On the upside, if it were easy, it’d be a commodity and not something done in the US anymore.

    Beyond that, I’ve seen a lot of software engineers with that kind of experience start to discount new ideas and technologies, having felt they’ve seen it all before. They fall into using the same patterns to solve problems because that’s what’s worked for them before. Being an only hands-on software engineer past 45 is a rough game. Your peers are right out of college with low salary expectations, have lots of youthful exuberance, and want to write their perfectly architected ‘killer app’.

    For me, I went into management, where I see myself as an engineer who’s trying to fix systemic problems and get rid of roadblocks for the coders on my team. The technology changes every few years and I’ll learn it it a high level. But where I used to really get into learning every arcane aspect of the framework/os/library/language/etc, now I’m more interested in the team and how to get a group of ridiculously smart people together to build kick ass software.

    Chris

    7 May 12 at 10:50 am

  42. It seems like the main issue here is this developers disgust for peoples misguided expectations.

    People are bad at predicting how long things will take. This is because you have to look at a micro level at the individual pieces and estimate the time it will take to complete each piece. Unrealistic predictions come when the developer has to complete something he or she has never before done ( yet still has to guess how long it will take, an impossible task!)

    When speaking to non developers I often use the analogy that building software is like building a car. For some reason, it is non intuitive for them to relate these two fields of engineering, because their experience with computers is limited and the details of computing are so beautifully abstracted away from them.

    In reality, software is a purer form of engineering because many assumptions are stronger than with hardware or mechanical engineering projects where theory gives way to the real world, and you have to account for a degree of variance and error that you will probably never be able to model perfectly.

    nate

    7 May 12 at 11:04 am

  43. Wats up ?

    Rahul

    7 May 12 at 11:52 am

  44. this whole blog post is a leaky abstraction.

    software is designed by engineers and built by machines. how is that any different than any other product in the world?

    it seems that you’re confusing the intricacy of software design with the actual process of building it. alot of things also have intricate design processes. software is hardly unique in this respect. everything from processed foods to military aircraft has a design process just as intricate as software design.

    this entire post is based on a fundamental misunderstanding of engineering.

    dbg

    7 May 12 at 12:02 pm

  45. Yes we have factories filled with machines that produce most of the worlds goods but these machines were created by people. The people that created one factories machine could say the same about when the next client asks them to build something almost the same as the last machine except that it outputs something a little different. People dream of machines and software that build its self but we are simply not able to create systems like this very well and the AI needed to make it better just hasn’t been dreamed up.

    Personally I write software because I like problem solving, everyday I am solving puzzels and always trying to create something beter than I was able to create before. As others have pointed out this often forced by new technologies which in my mind is a sign that I am not alone. I am part of a larger system of many developers people trying to make the world a better place. Simply shutting my brain off and using the same thing that is already been created is like giving up. There is always room for improvement and I look forward to the day everything I am working on now is replaced with something even better.

    Tyler Larson

    7 May 12 at 12:02 pm

  46. The plumper analogy comes alot with coding/designer sites/anything. From my experience, design (of any kind) and coding(of any kind) shares almost nothing in common with plumbing. Its put here really well, everything a plumber does is predefined, the hardest job in that process is that of the architect, and that work(and all the effort and complications that go with it) is, so some extent, invisible at the point the plumber comes round.

    If your hiring someone to make a website, that simple act highlights that your not capable of doing it yourself. Thats why I trust Casio to build me a watch, Apple to build me a phone, thats why I don;t build the houses I live in. I place my trust in those people to design and build those things os I can get on with something else, I pay a premium for that, because its worth it. The next phone by apple it worth the wait, even if its longer than I’d like and costs more than I wish to spend. I trust them, I relinquish control, so they can do their job.

    It’s a shame this level of trust isn’t implicit in the process of producing one off, very hard to build, very complex one off websites/apps etc.

    Only hire people you trust, realise that because you know less than them, you can have an input, but leave them to do their job in the time it takes, pay them on time, be nice, learn from the process, chill out.

    chris

    7 May 12 at 12:35 pm

  47. I think it’s wrong to compare software development to fields like construction or plumbing, and that is where your friend is going wrong.

    Sure, you do ‘build’ things in software development, but I think it’s more accurate to say you’re formalising ideas. When presented with a problem you must think of a solution. Software is a structured and repeatable representation of an idea, formed to solve a problem. As such it’s closer to literature than construction, and how many good books have you read that were written by a machine?

    Tom C

    7 May 12 at 12:49 pm

  48. Focusing on development practices will probably create the burnout that’s described here. Our software world (and the web in particular) is moving too fast for standards to form and it’ll be a hairy endeavor for the foreseeable future.

    That said, I find that looking for motivation in the products – and not necessarily the style of coding – makes it more likely for me to get up in the morning to go to work. The same web problems exist as they have a decade ago, but both the technology to solve them and the underlying problem to be solved have evolved.

    Working with good developers will, more often than not, result in doing the right thing. Setting a “coding standard” is admirable, but the cost comes in following a script that may not make sense, can be highly inefficient, which often places no direct responsibility and can kill speed and innovation.

    Allen

    7 May 12 at 12:59 pm

  49. [...] just read Scott Porad’s post “Why do web sites and software take so long to build? And why is it so hard?”, and it really got me thinking. He talks about how clients and customers think software should be [...]

  50. It’s hard because we’ve failed to make easy enough.

    Will

    7 May 12 at 2:35 pm

  51. *sigh*

    18 years is a long time (I’ve been at it 23 yrs now). It’s a long time that has seen some progress, but also a lot of big steps backwards. All the while, as many of these comments so clearly demonstrate, there is a constant flood of new people entering into an ever expanding field who choose not to believe that 18 years is a long time. Or that 18 years provides wisdom beyond their current experiences. Or that the work is not exactly what they think it is.

    As such, software is hard precisely because people running the show with 18 years of experience are rare, and people with significantly less than 18 years of experience are convinced that they get it (even when they clearly do not). You can lead a horse to water, but afterwards it’s best to head off to the beach …

    Paul.

    Paul W. Homer

    7 May 12 at 3:31 pm

  52. Writing the code might be boring or difficult sometimes. Building products that help people and change the world is a diffeent story.

    Sergey Kornilov

    7 May 12 at 3:45 pm

  53. I arrive at this very same question with each new challenge..

    - Building ‘vanilla kits’/'boiler plate’ should be a developers favorite past time to **avoid burnout**, improve deliver time, sharpen or innovate/assimilate techniques.

    - Don’t be affraid to changes languages if necessary.I’ve had an affair with c,python,php,javascript,ruby.. Still married to none and loyal to all.

    - Choose a technique, then choose->else->create the tools. The answers become apparent..

    With everything said and done, writting code – to myself anyhow – became a passionate idea to code.. insistant and burning like a papercut. Sometimes annoying as hell, but satisfying to scratch.

    Hope this help someone :)

    Victor Phelemba

    7 May 12 at 4:45 pm

  54. different languages make different jobs easier. If you’re finding that a job is miserably hard, you’re probably using the wrong language. My favorite these days is Squeak Smalltalk and friends.

    saijanai

    7 May 12 at 8:01 pm

  55. I think we’re all a little bit addicted to the challenge. New projects often do bring new challenges and complexities, and it can very fun and very frustrating try to hash out the most elegant solution we can in the time given.

    Steve Klebanoff

    7 May 12 at 9:08 pm

  56. I can understand how your friend would experience burn out. Software development is highly demanding career that takes proper management and coordination with team. A great book that addresses many of the issues he faced is entitled, Digital Work in an Analog World by John R. Fox. The author especially points out that looking for more and better technology to solve software problems appears not to be the answer. You can learn more about his book here: http://www.analogdevelopment.com/

    Lauren Smith

    7 May 12 at 9:17 pm

  57. It’s Andersen (not Anderson) Consulting (or was; now it’s Accenture).

    What Haveyou

    7 May 12 at 11:43 pm

  58. That’s the reason I love my job.
    Where do I get the chance to build such complex products, used by so many people, completely by hand?

    Christian Vester

    8 May 12 at 12:09 am

  59. Creating my own software product and making a living off of it is far more exciting, fulfilling, lucrative than working for a company and being told what to code via a large, boring spec document.

    BTCB

    8 May 12 at 6:05 am

  60. [...] Scott Porad’s post “Why do web sites and software take so long to build? And why is it so hard?” TweetVote on [...]

  61. It’s the hand crafting aspect that I really enjoy.

    “..is there one thing you can think of that has been hand-made, and on such a large scale as software, that was as complex”.

    This is the exact reason I find software development interesting. Actually, I would prefer to hand craft real physical objects, like model ships or cars etc, but any living to be made in that is all but gone now.

    Thank goodness I say for the current state of software development. In a few years, software creation too will be automatized to tedium. The only human parts will be UI design and high-level feature specification both simplified by powerful tools and the system architecture and implementation will be automatically governed by parameters and will adapt itself to scale. That is the point when I will quit.

    Kieryn

    8 May 12 at 6:42 am

  62. If your friend was tired of programming because it was “hard,” then he was in the wrong profession, and was right to leave. As in any profession, the great practitioners carry on BECAUSE of the challenges, not in spite of them.

    Ben Dilts

    8 May 12 at 6:59 am

  63. Every new satellite is unique, hideously complex, and completely hand made. Some attempts, like GPS satellites, have been made at repetitive assembly; but even these are not “mass produced”. And, after being shaken to within an inch of their lives, they have to work perfectly the first time they run.

    Budryerson

    8 May 12 at 7:19 am

  64. Hard? What’s hard? This stuff is a piece of cake. Try elementary school teaching, tax code law, micro-biology, materials science, integrated circuit design, oncology surgery. Those are hard.

    Imagine an industry where the tools used to build the most complex software are free. Are the same tools used and taught in high schools. Where the fundamental techniques can be basically “learned” in 21 days. Do not require a college education (Bachelors degree – ‘or equivalent experience’). Where you can jump right in and start using and building in an afternoon. Where even the simplest idea can make you rich (Angry Birds, Fadebook, etc.).

    But hard? Come on! Software is like pottery. Yeah it’s hand’s-on, hand crafted, creative from the start and builds on years of experience. But hard? Like pottery, after 20 minute of sculpting a cup or writing a file search utility, guess what? You can use whatever you just created. OK, sure the BEST software comes from decades of understanding, but that’s only experience talking – not difficulty.

    Anony Mole

    8 May 12 at 7:38 am

  65. I’ve been in software development and computer systems engineering since the late 70s.

    IMHO, the reason software development (web or other venue) takes so long is that developers tend to want to focus on the lower-level minutia rather than focus their creativity on higher-level functionality and quality.

    I remember when Microsoft’s successive line of QuickBasic (QB) and Visual Basic (VB) came out as rapid application developement (RAD) technologies. Other development technologies like PowerBuilder imitated VB for RAD. Developers who use the tools properly as designed could develop solid, efficient, stable, and scalable programs in a fraction of the time of their C and Pascal (and later C++ and Java) peers. Ever since then, many in the developer community have developed an intense, religious, elitist hatred towards VB and technologies like it. Granted, VB’s ease of use attracted a minority of less-than-professional developers who wrote crappy code, and that has contributed to the image, but hardly representative of the whole.

    But with QB and then VB, developers were given the tools that allow them to focus their limited project resources (e.g. man-hours) on delivering an application woith more features, more stability, that is very close to lower-level languages like C, C++, and Pascal in terms of performance and scalability.

    IT managers need to be leaders. Decide your department is going to be productive and make quality software in less time. Developers are like children. What they want is not always in their (or the organizations) best interest. They need to be led by an IT manager who equips them for success, sets the bar of expectations high (e.g. a servant-leader), and conveys (with effect) the bigger picture to the developers.

    Developers, like children, can do amazing things when properly motivated and led.

    If you want software done quickly, use the tools that enable that, but also enable quality, fewer bugs, good performance, maintainability, and scalability.

    Jeff Jones

    8 May 12 at 8:07 am

  66. Very interesting but I cannot quit. I’m sure it’s just because I don’t like routine stuff. Hmmm… well, it’s somehow hard but that’s what makes it enjoyable.

    Stephen Baidu

    8 May 12 at 8:25 am

  67. I guess this very same reason makes doing a website/software so interesting!

    Lucas Arruda

    8 May 12 at 9:08 am

  68. It’s only a matter of time before people stop making software themselves – the robots will take over – a human will simply tell the robot the problem and in a split second your program will be done.

    In years to come people will laugh or find incomprehensible that people actually developed software.

    This will happen sooner than most people realise.

    Joe

    Joe Newman

    8 May 12 at 9:16 am

  69. Because if it was easy, anybody could and would do it. And we would no get paid the large amounts we do for playing the ultimate electronic game…

    Ed Aymami

    8 May 12 at 9:38 am

  70. I see the problem somewhat differently. There was an article in computational physics in the late ’90s that looked at ecosystem stability. It showed that introducing even a single new species could result in mass extinctions.

    Software is an incredibly powerful means of spreading new ideas, and therefore of killing off old methods. Witness WalMart, whose software allowed it to disintermediate the distributor by linking the retailer directly to the manufacturer. Look at Facebook, whose “social media” infrastructure is slowly supplanting Google’s “search engine” ecology.

    So software is hard because it involves integrating into a rapidly changing “ecosystem” of systems and services, and painful because the consequences of failure are accelerated enormously. Business owners perceive these kinds of threats quite clearly, and when a team isn’t spot on with the technologies that they deploy, the situation can become frightening and unpleasant fairly rapidly.

    Brian Balke

    8 May 12 at 9:42 am

  71. Application software becomes complex because of Frameworks, too many Engineers working on the same product etc. If more than 10 developers are working on a product, its going to be complex, because as someone pointed out, humans cannot work in a group of more than 10.

    Raghu

    8 May 12 at 9:44 am

  72. “..is there one thing you can think of that has been hand-made, and on such a large scale as software, that was as complex”.

    The space shuttle springs to mind. Also, nuclear reactors. Any naval vessel (nuclear powered aircraft carriers, for example). Heck, any building.

    We don’t have “space shuttle manufacturing machines.” Parts of the shuttle, parts of the nuclear reactors, parts of naval vessels, parts of any building, are replicated by machines. But the assembly of anything large scale is done by entire crews of humans. The building you’re in had entire construction crews that hand built it with some aid of machines to do heavy lifting, and you’re trusting you life to its construction. Most of the software that gets written doesn’t have the same life-or-death consequences as these other large scale, hand-designed, hand-assembled projects.

    Manufacturing machines are good at creating a lot of the same thing. If something is done once or only a few times (like any large scale project), a specialized machine to do it is not cost effective. Generalized machines that are simply an extension of a person can be used on many projects, which makes it cost effective, but it no more responsible for the manufacture of the building or ship as a saw is in the hand-construction of a chair.

    So I have to say, your friend is exactly wrong. Every large scale project is hand made–software or otherwise.

    K.J.F.

    8 May 12 at 9:54 am

  73. The programmer may exit the programming loop,
    but the programming loop never exits the programmer.
    (it’s a Hotel California thing)

    GDFrank

    8 May 12 at 10:35 am

  74. The point has been brought up in the comments and it’s worth being posted in an update, IMO. Things that aren’t made by hand were made by hand until someone figured out they could create a machine to do it for them, then still, the machine needed to be created by hand to create the items.

    It’s an important note because it seems to me that your friend isn’t seeing how web applications [em]are the machines[/em].

    I could provide examples and expand but I think I’ll save it for my own blarg.

    Alex

    8 May 12 at 3:41 pm

  75. A classic opinion piece on why building software will always be hard is Fred Brooks’ “No Silver Bullet – Essence and Accident in Software Engineering” — there’s a version of this available at:

    people.eecs.ku.edu/~saiedian/Teaching/Sp08/…/no-silver-bullet.pdf

    And if you don’t agree, one of its rebuttals, “What if there’s a Silver Bullet … and the Competition Gets it First?”, by Brad Cox — with a version available at:

    http://www.virtualschool.edu/cox/pub/92ByteWhatIfSilverBullet/index.html

    …even gets into the idea of an “industrial revolution” for software.

    Good classic reading… 8-)

    Sharon Tuttle

    8 May 12 at 6:16 pm

  76. Oops — I didn’t paste the “No Silver Bullet” link correctly. Trying again:

    http://people.eecs.ku.edu/~saiedian/Teaching/Sp08/816/Papers/Background-Papers/no-silver-bullet.pdf

    That one works!

    Sharon Tuttle

    8 May 12 at 6:24 pm

  77. An interesting, but limited perspective I think. Even building websites we use toolsets, IDE’s, the framework / programming language, source control, even the o/s.

    Compare that to the mechanical manufacturing world, CNC machinery is quite a recent invention (last 20-30 years for mainstream), prior to that large machine shops were filled with various levels of craftsmen all turning and milling, etc. components by hand! Again they were using tools but they were controlling the machines by hand and the quality of the goods produced was dependent on the experience and care of the engineer! Not that much different to today’s coding environment! Similarly before word processors and copiers, we had large typing pools with many typists all bashing away producing material. Highly manual again. Each time technology has come along and “automated” the process but it takes time. Compared to these other industries software engineering is still in its infancy but has already taken great strides forward, it’s just a matter of time.

    The other option your friend has was to improve the game rather than leaving it. Of course it’s every ones own decision and glad he’s happy, but it feels like a missed chance to me.

    Paul Hadfield

    9 May 12 at 12:52 am

  78. Your friend has obviously no experience with engineering disciplines as well as with well-run software projects. He is wrong on all four accounts.

    “First, name one other thing in the world, he said, that is used by so many people and which is created entirely by hand?”

    Software is not at all created “entirely by hand”. For parsers, we use parser generators; for SQL, we use ORMs; etc.etc.etc.
    – This is exactly equivalent to a tool factory using its own tools to create tools to create tools that they sell.

    “Plus, in the history of the world, he said, is there one thing you can think of that has been hand-made, and on such a large scale as software, that was as complex?”

    Each and every DESIGN of a complex machine (your car, the Titanic, a house) is made exactly in the same way: Much thinking, much drawing and writing (by hand!), but also, as of today, many tools that support these processes. And: I challenge you to visit a railroad relay switching machine for a railroad station with say 200 switches – and then come back and claim what you said above (RELAY machine! – so that you do not mix this with software).

    “And, there is no standard way to do things.”

    What? Parsers and generators are written in a common way; layers are done in a standardized way. Many people might not know how to do things correctly – but many do!

    “Finally, can you think of any job where people are making really complex things by hand, and which requires a ton of experience and training to be good at, yet everyone and their uncle has an opinion on how long a project should take to get done?”

    Please: Go to the next car manufacturer and talk with their engineers about their engineering (NOT BUILDING! – that’s a total different story – we are talking about software ENGINEERING!!) of a new engine or an updated version of a model several years old. The stories will be exactly the same as in the software industry. You’ll find integration hell, wildly off estimates, last-minute patches to the plans, unexpected cross-interferences etc.etc.

    Harald M.

    9 May 12 at 4:10 am

  79. I think alot of the folks commenting are getting wrapped up in the details. Code is code, and it’s written by hand. Whether or not a library was used to avoid writing more code, is not the point. The article stresses that the challenge presented is not worth it (not that the challenge itself is too hard). He points out (and rightly) that there is little consensus on how to actually build things. That coupled with the fact that coders tend to be OCD, rigid, and sometimes over-zealous about learning every minute detail regarding an implementation, language, or environment. These two combine to make software a frustrating environment to work in. Why stay and deal with that when you can move on to greener pastures? Personally, I find I’m fairly good at software. Do I see myself doing this in 10 years? Probably not – there is just too many other interesting things to do in the world.

    Alex

    9 May 12 at 5:09 am

  80. Alex, I think there are three messages in the text:

    a) SWE is hard;

    b) ONLY SWE is hard (“can you think of any other job …”); and …

    c) … “therefore”, EVERYTHING else is a “greener pasture”.

    The first, as you point out, is correct. It’s the naivety of the other 2 points that most of us criticize. Of course there are greener pastures – if you choose to become a financial crook, you might rip off many people and collect 10s of millions in just a few years – engineering is just an idiotic pastime from this point of view.

    But if you are an engineer (by heart or by necessity), there are not so many green pastures around. And even if you start to compose music (which I do), you will find out that it is very hard work and has all the problems that any job has that wants to achieve something new.

    In 10 years, you might do something different – but I promise that you will also be stuck in social, conceptual and other deep (and interesting) problems that will look awfully similar to what is described above …

    Harald M.

    9 May 12 at 6:00 am

  81. True – How true ! …

    Just to mention – you forgot about the revisions ! … How many times we need to demolish a building and keep re-constructing it again and again … keeping the schematics of the building always approved and in control by the city regulatory ! …

    It sure is a TASK ! …

    Wished CUSTOMERS understood THIS ! …

    Anirudh

    9 May 12 at 6:06 am

  82. Anything that inolves two or people making decisions together is a hassle. People, on the whole, are terrible at making decisions, especially as a group. You can automate all the tasks you want, but until you automate decision making you will still have what feels like a manual, laborious process. I’ve been designing software for 10 years, and it’s almost always the same things that cause the same pain points, which is what can make a software veteran feel like they’re waking up to another groundhog’s day.

    Kristina

    9 May 12 at 7:29 pm

  83. [...] recently read an article about software design that got me thinking. Scott made some very good points as he thought about a [...]

  84. Still the best legal way to earn money as a maker (or at all really) without going back to school…

    Not many “makers” get paid as well as software engineers do (think sculptors, painters, writers, musicians, wood-workers, etc, etc – the average wage is all much lower).

    Dad

    9 May 12 at 10:20 pm

  85. I guess, you should do what you like, but don’t get obsessed.if your job is restricting you , you must explore other possibilities. It is hard to think beyond when you have got so addicted. But think like this, all subjects ever discovered are based on really questionable assumptions. Just think if every one were to do what he is made for, one has no choice at all! You must do, what you feel you should do. Whatever you choose just adore beauty. It happens with me, I try to correlate music and maths and suddenly links start appearing, and I discover a lot similarity. Think that way, just enjoy coding by adoring its beauty and not getting obsessed.

    Jaley

    10 May 12 at 11:44 am

  86. [...] was reading some random posts the other day and I came across a post by Scott Porad titled ‘Why do web sites and software take so long to build? And why is it so hard?‘. An interesting point is made during a discussion with a friend, the subject of the post, [...]

  87. “Why is it that no matter how long a project takes, the customer always thinks it should be faster or cheaper?”

    - Good
    - Fast
    - Cheap
    — PICK ONLY TWO!

    Richard

    11 May 12 at 8:38 am

  88. Never thought of it this way. But there’s certainly thruth in this story! In our projects I do see a shift towards standardization of common software functionality.

    Webmonger

    11 May 12 at 10:10 am

  89. I like the standardization of a content management system (WordPress), but even with a standardized CMS with agreed upon API and frameworks, each project is individual and unique.

    I’ve automated many of the standard processes (pushing the dev instance to testing to production, rapidly prototyping a site using WP, rapid import of client’s content via XML scripts, etc) but those are only tools to help me save time for the important things — putting the site together for my client.

    Nathan

    11 May 12 at 3:01 pm

  90. I’m willing to bet that what your friend is so discouraged about is not the process itself but the way it is managed, or micro-managed.

    RichardW

    12 May 12 at 9:43 am

  91. So much of what I think is already said here:
    Plumbers DO debate; there ARE patterns and standards a.s.o; developers ARE automating things, that should apply for their own job too (from “write you a tool” to model driven approaches).

    Indeed there are two more things:
    First, for me Excel jockeying is the most boring and repetitive job on earth. So it seems to be a question of the right person on the right place?

    Second, the arguments sound frustrated, over generalising and blurred.
    A typical symptom of burn out.
    I do not know if he got or even sought any councelling, but in his situation that would be recommended. With “councelling” I mean even so simple things like contact and talk to developers outside his company, trying to informally analyse the situation (discussing his view and if it fits reality) and to see if changes could be suggested and applied to improve it.
    It seems that on his work place he found no guidance and motivation by colleagues or by his management, so quitting may have been the right thing.

    All in all the questions remain: Is it indeed our industry (I think it is not)? Or is it his employer or his colleagues (environment)? Or had he simply chosen the wrong job?

    Dev

    14 May 12 at 12:19 am

  92. A while back I wrote about the very same thing. It didn’t evoke much response though. I suppose software developers don’t like thinking about it. If software creation is recognized to be a form of art (which in my opinion it is) then developers have to admit that only few of them will have natural talents while others will remain clueless no matter how they strive to succeed. Not something the majority would easily accept.

    Here’s the article:
    http://developerart.com/publications/11/why-software-creation-is-a-perpetual-challenge

    I’d be interested in your thoughts about it and more so in the opinion of your friend. Perhaps you could show it to him. Given his opinions expressed in this article he should find the reading interesting.

    Developer Art

    14 May 12 at 6:22 am

  93. [...] Porad schrieb neulich einen Artikel darüber warum es eigentlich so schwer ist Software zu bauen. First, name one other thing in the [...]

  94. Plumbing, Carpentry, Engineering and other disciplines are significantly older and more mature than Software engineering.

    That is why their are well known approaches that everyone agrees upon, tools to build them with. Because over the course of thousands of years the best approaches have come to the top.

    The discipline of building Software is 50 years old at best? I’m sure in a thousand years programmers won’t be arguing about the best framework, design pattern, or language.

    Jeffry Houser

    14 May 12 at 9:12 am

  95. [...] been meaning to share this thoughtful piece from Scott Porad, CTO of Cheezburger (squee!), that does an excellent job encapsulating some [...]

  96. They don’t take so long to build, they take the exact amount of time that they need to get done. The problem usually is because the project manager (or whoever estimated the project) did not collect the right project requirements or did not collect all the requirements.

    Fadi (itoctopus)

    16 May 12 at 8:07 am

  97. You use a computer, to write a blog, that you post to the Internet. You probably also use a cell phone, perhaps even a smart phone. All of that was built via that “fools errand”. I’ve lived before computers and cellphones, our lives are better for those fools.

    patbob

    17 May 12 at 10:23 am

  98. [...] can read the article here. Posted in [...]

  99. [...] Rzecz w tym, że w tej dziedzinie nie ma jedynie słusznych odpowiedzi – są standardy, są konwencje ale niczym w tej sentencji: „good thing about standards is that there are so many to choose from” wystarczy zestawić ze sobą kilku deweloperów by uzyskać kilkanaście rozwiązań, z których każde realizuje przedstawione założenia ale ma też swoje wady i zalety. Bardzo dobrze opisał to Scott Porad w notce Why do web sites and software take so long to build? And why is it so hard?. [...]

  100. 1) Really good methods to follow. Here is my small contribution this site will give you

  101. [...] Why do web sites and software take so long to build? And why is it so hard? at Scott Porad I can certainly relate to your friend’s frustrations. Over the past 14 years, I went from being a developer to a code tinkerer. I enjoyed keeping up on the latest development tools but was much mo… [...]

  102. Oh I agree entirely with your burnt out friend. But I still hope that the next project will be oh so much better to manage (hasn’t happened and I’m burning out too).

    Yes, government do define website design parameters which is why most such sites look bland while being very difficult to navigate. You have to really want the information contained to spend so much time guessing how to find it.

    Way back when I had a email byline generator that announced “If builders built built building the way (most) programmers white software, the first woodpecker that came along would mean the end of civilization as we know it!”

    Govind

    28 Feb 13 at 9:08 am

  103. Hello, I would like to subscribe for this blog to take latest updates, so where can i do it please help out.

    rebelmouse review

    16 Dec 13 at 9:24 pm

  104. Ciekawa fizyka sprawdziany pobierz

  105. You know, your friend got a point there. i dont completely agree with him. Navigation is a very important factor when it comes to website. show me 5 websites with those qualities

    Shalin

    20 Jan 14 at 10:53 pm

Leave a Reply


6 × one =