Hackers, Painters and Practices for Software as a Creative Process

In Paul Graham's book Hackers & Painters - Big Ideas From The Computer Age. A common theme that runs through the book is viewing software as a creative process, implied by the "Painters" in the book title.

There are some incredibly insightful observations in the book that resonate with my experience as a programmer - that is that the most successful projects I've worked on have been shaped and moulded to their best possible form rather than manufactured from a prescribed design. And while software development is also a engineering process requiring a solid foundation, sound principles and well defined interfaces, it seems to me that the environments where projects thrive embrace the creative nature of software development.

These are a few quotes from the essays in the book that allude to this idea:

Design and Research

Now almost every drawing teacher will tell you that the right way to get an accurate drawing is not to work your way slowly around the contour of an object, because errors will accumulate and you'll find at the end that the lines don't meet. Instead you should draw a few quick lines in roughly the right place, and then gradually refine this initial sketch.

Building something by gradually refining a prototype is good for morale because it keeps you engaged. In software, my rule is: always have working code. If you're writing something that you'll be able to test in an hour, then you have the prospect of an immediate reward to motivate you. The same is true in the arts, and particularly in oil painting. Most painters start with a blurry sketch and gradually refine it. If you work this way, then in principle you never have to end the day with something that actually looks unfinished. Indeed, there is even a saying among painters: "A painting is never finished, you just stop working on it." This idea will be familiar to anyone who has worked on software.

Taste for Makers

Good design is redesign. It's rare to get things right the first time. Experts expect to throw away some early work. They plan for plans to change.

It takes confidence to throw work away. You have to be able to think, there's more where that came from. When people first start drawing, for example, they're often reluctant to redo parts that aren't right; they feel they've been lucky to get that far, and if they try to redo something, it will turn out worse. Instead they convince themselves that the drawing is not that bad, really-- in fact, maybe they meant it to look that way.

Dangerous territory, that; if anything you should cultivate dissatisfaction. In Leonardo's drawings there are often five or six attempts to get a line right. The distinctive back of the Porsche 911 only appeared in the redesign of an awkward prototype. In Wright's early plans for the Guggenheim, the right half was a ziggurat; he inverted it to get the present shape.

Morale is key in design. I'm surprised people don't talk more about it. One of my first drawing teachers told me: if you're bored when you're drawing something, the drawing will look boring. For example, suppose you have to draw a building, and you decide to draw each brick individually. You can do this if you want, but if you get bored halfway through and start making the bricks mechanically instead of observing each one, the drawing will look worse than if you had merely suggested the bricks.

Hackers and Painters

As far as I know, when painters worked together on a painting, they never worked on the same parts. It was common for the master to paint the principal figures and for assistants to paint the others and the background. But you never had one guy painting over the work of another.

I think this is the right model for collaboration in software too. Don't push it too far. When a piece of code is being hacked by three or four different people, no one of whom really owns it, it will end up being like a common-room. It will tend to feel bleak and abandoned, and accumulate cruft. The right way to collaborate, I think, is to divide projects into sharply defined modules, each with a definite owner, and with interfaces between them that are as carefully designed and, if possible, as articulated as programming languages.

The Age of The Essay

Essayer is the French verb meaning "to try" and an essai is an attempt. An essay is something you write to try to figure something out.

So What Processes Do Truly Creative Companies Use?

If software really is a creative process, what can we learn from companies running big creative projects. How about Pixar? Their work is no doubt technical in nature, incredibly complex and yet produces some of the most engaging digital works in the world. These enormous projects require process and organisation just like every other, so how do they do it? Clearly a talented team is required, but they need a structure within which to work and communicate. Apparently, Kanban and the Andon system are their secret weapons. This does not surprise me given my experience of working in both Kanban and Scrum agile teams.

Looking at Pixar's practices:

Kanban

Kanban aligns well with creative flow and is accepting of the fact that teams need to be able to change direction, even during the time that would span a typical scrum sprint.

Some key features of Kanban as compared with Scrum are:

• Continuous flow
• Continuous delivery or at the team's discretion
• Focus on cycle time
• Change can happen at any time
• Limits on concurrent work in progress
• Collective responsibility for goals

Andon

Andon allows a team to stop and reassess.

From the article linked above:

Another key Lean tool used by Pixar is the Andon System. This consists of the idea that anyone in the company can stop the production line at any time if waste, bottlenecks and obstacles are spotted. In Pixar’s case, if something that might hurt the animation’s value is detected, the creative process is temporarily suspended to undergo analysis.

This reminds me of a quote:

If the ladder is not leaning against the right wall, every step we take just gets us to the wrong place faster. - Stephen R. Covey

Kanban vs Scrum for Creative Process

I don't want to over generalise, there are situations where Scrum is a more optimal choice than Kanban. According to the Agile Alliance:

If the team is simply a group of individuals with some expertise, use Kanban.

If the work is innovative, creative, or new and requires stakeholder and customer feedback/engagement, use Scrum.

While I would disagree with the use of the word 'creative' in the "use scrum" statement (because it implies that Kanban is not suited to creative work), I agree Scrum should be considered where frequent reporting to stakeholders and product owners is required.

However I believe that Scrum comes at a cost to the creative nature of software development in the following ways:

Commitment

Commitment provides clarity of goals, and key metrics that indicate whether a team is being successful in their short term goals. However even during short sprint cycles, new information, discussions or realisations can invalidate the initial goals of the sprint within the first few days. Repeated failure to meet sprint goals are bad for team morale. A burn down chart that repeatedly doesn't burn down reduces faith in the process. Often this happens not because Scrum is not properly organised or because of team competence, but because the project is exploratory in nature, and therefore requires a greater degree of creative process.

Ceremony

Ceremony can be valuable to the product owner, giving them confidence that their goals are being met and keeping them in the loop and providing an opportunity to provide important feedback. The cost of ceremony is the regular break of flow state. More frequent scheduled stops discourage team members from self organising adhoc communication, which is likely to be more focused and timely.

Kanban and CI/CD / DevOps

Kanban fits better with CI/CD. Commonly, Scrum and CI/CD are combined. But for delivery to be continuous, the scheduling of releases needs to be reduced. Also, the collective responsibility and reduced focus on roles also fits better with modern DevOps practice.

Conclusions

The reality may be that Kanban is more likely to be acceptable to your organisation if you own your own product, and so are less accountable to external stakeholders. But I feel that Kanban is more closely aligned with the creative nature of software development, and that is the environment that projects are most likely to reach their potential. Kanban likely requires more trust in the team from external stakeholders since fewer meetings, reduced commitment to short term goals, limiting work in progress and an openness to change in direction can look disorganised to those not close to the low-level details of a project.

More case studies are probably needed, I'd be interested to hear of highly creative teams using Scrum successfully. If any come to my attention, I'll add them here.