This is more a brain dump rather than a fully-formed post, mostly inspired by my current endeavours in my practice of writing software.
TL;DR - Tools help us get our job done. They matter HUGELY, especially in a task as complex as writing software. Software tools are evolving in a way that has more of a "fix the symptoms" philosophy, rather than trying to "do what's empirically right" We need a tool-builders renaissance if we want to sustain exponential progress in the realm of software implementation. I don't know how to do it, but will be focusing a chunk of my efforts to the attempt.
==== ACTUAL POST BEGINS ====
Flushing a toilet starts a multi-step process, of which the ultimate goal is to bring the waste in the toilet efficiently to a facility which can process and dispose of it in a safe and economical manner.
This works millions of times a day (lots of shit in this world), and the fact that we rarely see residential sewage pipes and canals spilling their contents out onto the streets is a testament to the great engineering feat that is our sewage system.
It works because someone, somewhere knows how the sewage system works. (or at least, the part that they're responsible for) Be it through best practices, extensive literature, etc, these engineers manage to create and maintain a system that works within the contraints of nature (eg: fluid mechanics -- shit is viscous) to create a deterministic system that just works and works and works.
These people can reason about the sewage system, and that gives them the power to evolve it to meet increasing future requirements (shit volume increases monotonically, shit density increases exponentially).
The framework that these sewage architects have is so solid, that urban planners can (almost) take for granted that the sewage system will be successfully completed. It's not a matter of "Will it be correct?" but rather "When? And at what Cost?"
At risk of sounding derogatory, it's a problem that money can solve. Somehow, somewhere, you can without too much effort, find the right contractor, give them adequate resources, wait a couple of months, and you've got a system that takes Shit out of your life.
Programming isn't like that of course. In the words of Fred Brooks, "Adding Programmers to a late software project makes it later". (ref: 'The Mythical Man-Month')
More generally though, any artistic venture doesn't face resource constraints as the primary bottleneck. Rather, the main bottleneck is in evolution of the implementation to meet project requirements while maintaining the conceptual integrity of vision.
I figure this is much easier to show first then tell, so here's a video of stills captured during the process of Pablo Picasso painting a woman (Inspired by this post by Derek Sivers) Keep an eye on the legs as the stills go by. During the process, Picasso makes drastic changes to their position, color, and texture. This is the "changing requirements" so to speak, but more importantly, he has a thorough understanding of the constraints of the underlying canvas, and a masterful grasp of the tools he has at hand. He thus has the skill and the confidence to be able to make such on-the-fly changes to his work, all in line with his vision of "the Reclining Woman".
In building software, we have none of these. This is of course, because we deal with much more complex requirements. Complex requirements typically demand a cooperative group effort, which brings about its own set of challenges.
For starters, Group cooperation requires specification. The fact is that Software specification and documentation is notoriously hard, and hence notoriously bad. Worse still, once they have to change (and they will have to change), having the courage and the vision to actually make that change is knowledge bestowed upon only a few.
Also because of this complexity, the tools that we depend on sometimes fail. More often, we find that our tools are inadequate for the specific task at hand. This is when one tries to augment them by modify the code to make it fit for purpose.
But complexity always wins, and we find ourselves grapping with uncomprehendable code bases, novel abstractions, and foreign conventions. We have no choice but to hack the tool -- to forcibly make it do what we need it to do under our circumstances, knowing full well that a horde of bugs is bound to stem from this disrespect of natural complexity.
Our budgets and our bosses push us on. (Even the great John Carmack has to struggle with his tools)
Perhaps it is just that software lives at the intersection of Art and Science, giving rise to phenomena we've yet had the language nor the intellect, nor the time, to explain. Or maybe it is a region that is unformalisable -- we cannot form a rational model for its creation, and it's mastery lies only with those unique individuals with that je ne sais quoi.
I don't really buy that. Rather, I think that we just don't understand the underlying constraints well enough. Just as the ideal sewage architects need to understand the immutable laws of fluid dynamics to make their systems work, so does the ideal software architect need to understand the immutable laws of scoping, concurrency, types, inheritance, continuations, etc, etc, etc. It's a far longer list, but more crucially, it is enumerable -- possible to fit into a single human brain.
As software developers (and as artists), I think we should remember that our ultimate goal is to leverage our tools to show the world something novel. Which implies that we need to have a vested interest in the study and improvement of our tools.
Music is a good example here. To the audience, music is purely for consumption. To the musician, music is not just about the experience, but also about the conversation with nature regarding the possible. One of my favourite pieces, 'CAFO', shows how you can take a simple tapping technique, overlay it over a 13 crotchet-beat rhythm, insert a 5-beat quaver motive, and make something completely mind-blowing (this is admidst other compositional innovation, which is why I like it so much). It was an innate understanding of the constraints of the instrument (guitar), along with some creative application of well-studied musical theory, that allowed one (Tosin Abasi) to compose something of that sort. It was a random moment of inspiration that was allowed to flourish given a set of deterministically-derived contexts.
That leads us to another defining feature of good tools -- they need to allow the user an adequate degree of abstraction, giving the user the flexibility of expression neccessary.
Most of software today however, comes in the form of black boxes: an API that you call, or a framework that you pray "just works". As a tool-obsessed geek, I find this incredibly hard to sit with. (ref 'Zen and the Art of Motorcycle Maintenance') It's almost as if we're building software with a planned obsolescence cycle. (ref: 'Built to Last' by James C. Collins and Jerry I. Porras)
In any case, it probably is time to start considering how we can start anew.
But let's not forget that with we're fighting up -- against strong economic forces. People make a crap ton of money making software, and grabbing mind share from the current breed of technology is far from an easy task.
We need our new tooling solution to be disruptive (in the definition of Clayton Christensen). We should not fall for the romantic notions of "build it (a better product) and they will come", nor should we think that "if we can drive cost lower everything will be good".
Bittorrent made huge advances in P2P streaming technology, Skype showed us that NAT hole punching (and other proprietary stuff) can actually deliver (almost) toll-quality voice service. But both failed to affect the art of software development in a big way, either for cultural (Bittorrent == PIRATES!) or economic (Skype ain't telling us how they work anytime soon) factors.
Free (or Cheap) alone ain't going to cut it either. While cheap(er) video cameras and free distribution allowed the indie musician or indie film-maker to reach an unprecedented audience, we find that production is only the first hurdle in getting a movie out to an audience, (See the Critical Path episode 28 for a great preview into the financial challenges faced in movie making) not to mention the whole host of other challenges.
In that sense, we need to push software forward with metaphors that are both culturally and intellectually accepted, all for the price of FREE. That's not possible today, but I think it's becoming ever more likely.
Once we've got all that done, we then need to figure out a way to defend our intellectual sovereignty of the project. One of the reasons why I don't favour taking on Venture Capital Funding is because of the power that you give away to a committee (the Board of Directors). If anything, a complex project needs a small group -- preferably an indivitual, at the helm who calls all the shots. (that's probably why a project like Linux has continually made progress all these years, Linus Torvalds kept everything together)
Worse still, since we're fighting up against established forces, we're liking going to run into legal issues. Witness the patent wars that ensued from the Right Brothers' "invention", and the telcomm lockdown of Ma Bell (ref: 'The Master Switch' by Tim Wu).
We must of course, fight this, but in a way that's compatible with future interests. Of contemporaries, Cory Doctorow has just most recently warned us about the Coming War on General-Purpose Computing. (His books, freely available under CC 3.0 are available on his website http://craphound.com/. I'd recommend starting with the latest at the time of this writing - http://craphound.com/gbbt/)
In some ways, we should admire the entrepreneurial hotbeds like Silicon Valley -- where people chase culturally-driven delusions of granduer, almost irreverant to the obvious chance of failure, lavishing in the thought of "doing well by doing good".
We should respect that, but at the same time, I deny myself patronage from such ideals. I'm happy that people do it, just like I'm happy that people "waste" time playing Starcraft 2 -- the replays and pro tournaments are Epic!
In the same way, I view the chronic focus on the ends and not on the means stifling to "true technological progress". We celebrate the entreprenuer, the risk-taker, the glamourous exits, but never the artist. I'm glad that Silicon Valley does that, but it's not for me.
Instead, I want to focus on the underlying forces that drive such things to existence. To do that, we need a celebration of the people who actually play the game of creation. Those who have a vested desire to elucidate new knowledge, and bring new visions to the world -- tool builders. If I have any goals, it would be to play in that field.
Well, I've just begun to scratch the surface of the challenges at hand, but that's the price of Art. As Van Gogh would say, "no painting ever sells for as much as it cost the artist to make it". (ref: 'Ignore Everybody' by Hugh Macleod, Point 39)
The only thing driving such a quest is that at the end of it, we'll have a deterministic way of making software. We will understand our implementations, and we will hence have power of them. That, I think, is something worth working for.
Until then, I'll have Compiler anxiety. And until then, I'll be fighting to cure this ailment.