Here is a comment that I made to the Linked-In "Software Engineering Productivity" group in the thread "Can agile methodology be applied to all software development projects". Since I took the time to organize a mini-novel around my thoughts on the topic, I figured that it would be a good idea to save that work here:
"As for my position on this thread, I believe the best benefit comes from taking a hybrid approach that puts the leg work in up front to ensure that business as well as IT has enough research vested in the business case as well as the overall Business/IT Strategy to understand at least enough to validate the investment and to set expectations on initial cost and time boundries. Milestones should be set to allow for measures and metrics and to enable departments such as marketing and product management to know when the business value is expected to be delivered and to be able to do thier jobs effectively. Also, similar to what CMMI outlines.. standard metrics should be captured throughout the project so that management and executive teams see progress and understand the affects to the bottom line for items such as scope creep and unexpected project delays.
I see great benefit for iterative development during the development cycle and I think that software designs benefit from having the flexibility to solidify UI, Component,Service, or data designs designs during the development cycle iteration/sprint. I don't believe that Enterprise Architectural goals are best determined during multiple development iterations and advocate that thought should be put in up front on how to weave the projects deliverables into the overall architectural roadmap which should be aligned with the long term strategic business/product roadmaps.
I also believe the size or type of a project can and should impact the amount of up front requirements gathering, analysis, and design. I tend to work projects that have multi-datacenter infrastructure requirements to support the software development effort .. not to mention hardware costs, software licensing, and vendor relationship management considerations. Both Systems Engineering and Software development team timelines have to line up to meet the milestone targets to deliver value to the customer. This is where I find more structure and repeatable processes invaluable. Also I believe that good project management and capturing business and IT metrics throughout the project help keep costs and expectations within target.
On the other side of the coin, requiring business, architecture, engineering, and development teams to deliver tons of documentation doesn't work either. You have to target the documentation artifacts that are useful ... not the ones that noone will ever read. The documentation should benefit communication and understanding across the projects stakeholders. Also, I agree that you can't know every requirement up front and that business and customer needs change and iterative development and reasonable change management procedures really help with this.
Also, getting funtionality to the customer as soon as possible is a win .. I definately advocate taking iterative approaches whenever and whereever possible.
If I have to deal with a project that has to deliver stringent requirements around scale, capacity, and reliability (or any of the other architectural "abilities") to fullfill promised SLA's, non-functional requirements or to maintain Customer Trust then I believe that this is a different argument that doesn't necessarily lean toward iterative approaches.
I believe in working to find the right amount of process and structure to be effective but not detrimental to project progress and success. I think that Waterfall, Spiral, Lean, and Agile methodogies all have their golden nuggets. But I don't believe that one methodolgy is the end all best and only approach. Methodologies just like any other set of principals or ideas requires you to consume the best points from all of them and apply them to the problems that you face on a daily basis."