Lots of people think about software architecture and systems design and system validation and software economics, etc. Research has been done on the group dynamics of engineering and there's an abundance of project management books. And complex systems do get built: bridges, airplanes, phone switching software. But i'm still convinced it's an order of magnitude harder than it should be.
The problem is human beings. Not that i'm all that crazy about Syknet either, but while you could at least imagine a technique for describing and optimizing the relationships between correctly specified subsystems, you can't do it with people. Even describing inter-human relationships in a statistical way is not particularly useful, because average behavior can't be applied to a single situation in a meaningful way (if i know that 50% of all projects fail, then i know that my project has a 50% chance of failing, that's true; but in practice you almost always have a better sense at the outset of what your chances of success are).
Probably the best statement of the problem is Mark Pilgrim's
"Morons and Assholes" post. In it he argues that all developers are either morons or assholes. However, most readers miss the point that a comfortable mix of assholes and morons is the best case scenario. Not infrequently, projects get too many morons or too many assholes, or at least one particulary strident sociopath (Pilgrim's term for morons who become assholes). Two sociopaths can destroy a project before it even gets started. The chance of any given person being a sociopath i'd estimate at about 10%.
If all workers were equal and you graphed productivity vs. number of people you'd get a curve that increases less and less rapidly up to about 6 people, then levels off, then starts to asymptotically drop back to 0. So 2 people do slightly less than 2x as much work, while 5 people do maybe 3x as much work, and 20 people do nothing useful. The traditional approach to address this problem is to create a management hierarchy so that the pool of people is divided into either functional or project groups (or both in the dreaded "matrix") with one person in the group designated as the main intergroup communicator.
The opposite extreme in my view would be to create an environment where systems emerge that address specific needs because there is a market for it. Groups will naturally optimize to the right size because the best, most timely solution wins. Communication between groups becomes less of an issue because the solution of group A either meets the needs of group B and so succeeds, or it doesn't and fails. The main problem with this evolutionary/market process is that it requires a fair number of participants and enough capital to get all of the variations to a point sufficient to be evaluated. It's not quite as farfetched as it sounds though. Think of the way that large companies tend to often buy companies that address their business needs rather than trying to compete with them.
My opinion is that complex systems in the future will more frequently get built according to the second model and its inevitable refinements. Much of the Internet itself and software surrounding it have has some of this flavor. I think the reason for this is the lower barrier to entry. It requires much less capital to start an interesting Internet service than it does to develop a useful aircraft component. I assume this to be because of the commodity pricing of computer hardware, the availability of nearly-free software, and the standards that prevail in the Internet world, both by committee and de facto.
