Software Development Methodologies, does one size fit all?

Here is a comment that I made to the Linked-In "Software Engineering Productivity" group in the thread "Can agile methodology be applied to all software development projects". Since I took the time to organize a mini-novel around my thoughts on the topic, I figured that it would be a good idea to save that work here:

"As for my position on this thread, I believe the best benefit comes from taking a hybrid approach that puts the leg work in up front to ensure that business as well as IT has enough research vested in the business case as well as the overall Business/IT Strategy to understand at least enough to validate the investment and to set expectations on initial cost and time boundries. Milestones should be set to allow for measures and metrics and to enable departments such as marketing and product management to know when the business value is expected to be delivered and to be able to do thier jobs effectively. Also, similar to what CMMI outlines.. standard metrics should be captured throughout the project so that management and executive teams see progress and understand the affects to the bottom line for items such as scope creep and unexpected project delays.

I see great benefit for iterative development during the development cycle and I think that software designs benefit from having the flexibility to solidify UI, Component,Service, or data designs designs during the development cycle iteration/sprint. I don't believe that Enterprise Architectural goals are best determined during multiple development iterations and advocate that thought should be put in up front on how to weave the projects deliverables into the overall architectural roadmap which should be aligned with the long term strategic business/product roadmaps.

I also believe the size or type of a project can and should impact the amount of up front requirements gathering, analysis, and design. I tend to work projects that have multi-datacenter infrastructure requirements to support the software development effort .. not to mention hardware costs, software licensing, and vendor relationship management considerations. Both Systems Engineering and Software development team timelines have to line up to meet the milestone targets to deliver value to the customer. This is where I find more structure and repeatable processes invaluable. Also I believe that good project management and capturing business and IT metrics throughout the project help keep costs and expectations within target.

On the other side of the coin, requiring business, architecture, engineering, and development teams to deliver tons of documentation doesn't work either. You have to target the documentation artifacts that are useful ... not the ones that noone will ever read. The documentation should benefit communication and understanding across the projects stakeholders. Also, I agree that you can't know every requirement up front and that business and customer needs change and iterative development and reasonable change management procedures really help with this.


Also, getting funtionality to the customer as soon as possible is a win .. I definately advocate taking iterative approaches whenever and whereever possible.
If I have to deal with a project that has to deliver stringent requirements around scale, capacity, and reliability (or any of the other architectural "abilities") to fullfill promised SLA's, non-functional requirements or to maintain Customer Trust then I believe that this is a different argument that doesn't necessarily lean toward iterative approaches.

I believe in working to find the right amount of process and structure to be effective but not detrimental to project progress and success. I think that Waterfall, Spiral, Lean, and Agile methodogies all have their golden nuggets. But I don't believe that one methodolgy is the end all best and only approach. Methodologies just like any other set of principals or ideas requires you to consume the best points from all of them and apply them to the problems that you face on a daily basis."

Where is Microsoft Taking BizTalk? How does BizTalk fit into the cloud?

If you are like me, you have a good deal of expertise and time invested in BizTalk as a platform for integrations and service orientation. With the release of the Azure Service bus I have become curious about the path that Microsoft has planned for BizTalk and where MS stands with the platform moving forward.

See the link below for the path that MS sees for BizTalk

http://blogs.msdn.com/b/biztalk_server_team_blog/archive/2010/10/28/changing-the-game-biztalk-server-2010-and-the-road-ahead.aspx

Scenarios for Integrating BizTalk and StreamInsight

There is a lot of buzz around StreamInsight and Complex Event Processing in the industry. Jesus Rodriguez and the team at Tellago have put out a great document for showing how these two technologies can fit together. Its well worth a read.

http://www.microsoft.com/downloads/en/details.aspx?FamilyID=6c1da607-4785-4668-a274-6898c38ea603

BizTalk 2010 Developer Edition is Available for Download

http://www.microsoft.com/downloads/en/details.aspx?FamilyID=938102b8-a677-4c20-906d-f6ae472b3a6a&displaylang=en

App Fabric Cache - Grid Dynamics White Paper that demonstrates ability to Scale Web Application

I have been looking at caching solutions for a number of scenarios lately to speed up enterprise application performance. In the process I ran across the white paper that shows the benefits of caching in a Web Application scenario.

"Grid Dynamics, a Microsoft partner, created three sample applications, designed to be typical use cases for data caching, and ran extensive benchmarking tests to evaluate their performance characteristics. The applications were a blogging engine, demonstrating the most basic features of Velocity caching technology; a simple e-commerce website, demonstrating Velocity’s capabilities for managing session state, and; a market data application, demonstrating Velocity’s event processing capabilities."

Click Here to download the full White Paper

New Microsoft whitepaper: Windows Server AppFabric Caching Deployment & Management Guide

Microsoft has released a New Microsoft whitepaper: Windows Server AppFabric Caching Deployment & Management Guide

Gartner: What they say about the place for Enterprise Architecture 2010 forward

The following article :
http://biztech2.in.com/india/news/enterprise-solutions/enterprise-architecture-enters-trough-of-disillusionment-gartner/88972/0 argues to the point of how EA when mature should be the bridge to help Business Strategy become IT Strategy ... not vice versa. Business Strategies should always drive Technology. You don't start with a Technological solution or Tool and try to mold the business around it.

Here is an intresting quote from the Article:

"The artificial walls between business and IT are crashing down, and EA is the bridge to integrate business and IT," said Philip Allega, Research Vice President, Gartner. "EA's original promise was its ability to provide future safe guidance given the desires and vision of an organisation's senior leadership team. As IT roles shift away from technology management to enterprise management, EA is suited to bring clarity to these blurred boundaries, and, by 2015, increased adoption of EA processes and uses by business will further IT's alignment with the organisation's culture, future-state vision and delivery of business value outcomes."


I agree with this assessment.

SOA, The Good - The Bad - The Ugly

Jens Andexer Executive IT Architect @ IBM and Willem Bekker, Head of Architecture Initiatives @ Standard Bank created this comparision chart that does a great job to outlining the benefits, challenges, and pitfalls of moving toward a Service Oriented Architecture within your company. From my experience I have found that this information is very true. Here is a link to the original article. I have included the matrix in my blog for my personal future reference. The credit for the article goes to Jens and Willem.

Good Business Implications of SOA	Bad Business Implications of SOA	Ugly Business Implications of SOA
Agility - SOAs allow business processes to be developed more quickly and it allows them to be changed more easily. This allows organizations to adapt to changes in their business environment more rapidly. This translates into a real market advantage since it allows products and services to be brought to market more quickly than competitors.	Change in the organization structure - At the heart of every SOA is a Centre of Excellence (COE). The COE is a new entity that controls the technical development of the SOA and provides expertise to the rest of the organization. The SOA COE is a new addition to any organization and hence its introduction can lead to conflict when it is resourced and when decisions made here effect the rest of the organization.	It's not easy - Transforming and organization to become services oriented is not easy. The transition is to a SOA is characterized by an evolution and not a revolution. Organisations that have traditionally been organised as silos my need to change their structure to take full advantage of becoming services oriented. This shift can be complex, expensive and there is no shortage of opponents to the change.
Alignment - A closer cooperative relationship between business and IT removes traditional barriers the hindered the IT implementation of business requirements. The footprint of a service in the business domain is a business function and it is described in business terms. The details of its implementation are hidden.	Change in organizational power structure Placing the ownership and control of services into the domain of business changes the power structure in organizations. This is typically met with resistance from those who have a vested interest in keeping the status quo.	Change in culture - A SOA represents more than a change in technology. Attendant to this type of transformation comes a change in an organization's culture as it becomes value driven. The organization must learn what it means t be agile and how best to exploit this agility for itself. The ugly truth is that this is one of the most difficult lessons to learn.
Business Process Improvements - Typically any SOA is associated with a rethinking of business processes. This business process reengineering is an opportunity to optimize how the organization operationally is going about its business. Doing a good job of this reengineering can make a significant improvement to a business's operational efficiency.	New challenges for business - Business must get more involved in giving direction to IT. Business lines must take ownership and responsibility for services and their enhancements hence initiating the development and change cycle as they will drive this process. This is not a role typically filled by business lines and will make for an uncomfortable change.
Flexibility - The adherence to good software engineering practice in SOA allows IT to be more responsive to business needs. Time to market for products and services is shortened and the cost of development and change process is reduced.	IT landscape becomes more complex before it can become simpler - SOA is enabled with a set of technologies like a business process execution engine and ESB. Adding technology to an IT landscape does not make it simpler even when the advantages out way the costs. But just because the IT landscape is more complex does not mean it can not appear simpler. The introduction of the service allows the complexities in the IT to be a secret. Consumers of the services need to know nothing about what goes on inside a service. As a result any rationalization that takes place in the back end can be hidden behind the service interface.	Technology alone will probably not show value - A SOA that focuses on infrastructure and technology is likely to fail. A SOA initiative is founded on the promise of delivering business value more quickly and more cheaply than before. A SOA that is too technology focused is unlikely to deliver on that promise since they will not show value in terms business people want to see it. Flexibility is only recognised as business value when it accelerates operationalising business requirements or reduces the cost of operational systems by allowing their rationalisation. Technology focused initiatives typically do not do this.
Data Unification - The service interface provides an opportunity to unify characteristics of the data so that service interfaces consume data that adheres to a unified data model. Common here means common: Structure - the structural relationships between elements is the same so that for example an address consistently consists of a house number, street, city, region and post code. Semantics - semantic refers to the meaning and use of the data. Data must have a consistent meaning and must not be used in a way that can be misleading. For example the concept of a customer might be hits on a web site in contrast to a set of owners of accounts. Format - how data is represented is important. Dates that are formatted DDMMYYYY must not be confused MMDDYYYY. Type - A type is determined by the representation of data and the set of behavior that can be performed on it. Timing - Timing refers to when an attribute is updated. In some cases attributes are updated by the front end system in real time. However, some attributes are only updated periodically by batch operations. Life cycle - under what circumstances data is add to data bases and when it is updated and when and how it is finally deleted from data bases	No One view of data - Standard interfaces for services require a common view of the data. This common view typically does not exist and trying to develop it often shows how disparate views in the organisation may be.	Unification may not be possible - To get all the characteristics of data consistent is seldom possible. Beyond the issues above are the ever present problems associated with "dirty data". Dealing with the inconsistencies is one of the great challenges of designing service interfaces. The ugly truth is that a uniform service interface is very difficult to build.
Operational Monitoring - The technologies and principles used to enable a SOA make monitoring business processes easier. This type of monitoring allows feedback from daily operations. This feedback can be used to measure how well the organization is doing against its strategic goals Traditionally business processes (BP) and presentation logic where all written into the same programs that contained business logic. The best one could hope for was that the different types of logic where at least grouped together but even this is often not the case. The upshot of this is that it makes monitoring of categories of logic very difficult. For example business processes could only be monitored as part of the application since it could not be separated out. SOAs, typically come with a business process execution engine. The introduction of this technology encourages business process logic to be located in one spot. Having the BP logic in one spot makes BP monitoring possible without the need to have the BP logic in the application logic. This is not unique to a SOA but the technology used to enable a SOA along with the discipline that comes from SOA's good software engineering practices make it easier in a SOA.	Monitoring complexity - Developing a monitoring model for business processes that feeds back to organizational goals is a significant piece of work that requires specialist skills.
Leveraging operational systems - A SOA uses operational systems to supply the business functionality for services. This means that the investment in existing systems can be used by repackaging it into services.	Technical misfits - In some cases operational systems do not relent easily to being repackaged as is the case when the structure of business functions does not match the requirements. For example if a transaction to add a new customer has a commit point after the name and address has been added to the customer data base and a second commit point after the security data has been added then these two operations are inexorably linked.	Changes may be needed - In some cases operational systems may need to be changed. When the changes need to be made in ERPs vendors are reluctant to make these changes. If the organization decides to make the changes in house the maintenance cost must be considered in the funding of the service.

ESB Toolkit 2.0 Multi-Server Installation - ESB ToolKit Installation

This is the third installation for installing the Microsoft ESB 2.0 Toolkit. This blog outlines the steps for installing the ESB Toolkit in a 64-bit multi-server environment. This article assumes a two node BizTalk Cluster. If you have questions please feel free to email me tjthompson5150@yahoo.com. Again sorry for the images .. its was my most direct approach to getting the article up. BizTalk Install Instructions for installing BizTalk, SSO, and BAM can be found here. UDDI Install One thing to note is to make sure that you have UDDI installed on each BizTalk node. You can follow the instructions for installing UDDI on a passive node found in this post. The goal on the BizTalk nodes is to connect to the existing UDDI3 database, so you use the "use existing" options in the UDDI Configuration console. http://tjthompson5150.blogspot.com/2010/04/esb-toolkit-20-multi-server.html In our environment we have a three server load balanced web farm that is our primary host for the UDDI 3.0 Web Site, so BizTalk is not our primary UDDI server in our scenario. If you do not install UDDI on the BizTalk tier the Microsoft.Practices.ESB.UDDIPublisher.exe will fail and not put the appropriate entries in the UDDI resposity and that will cause the ESB Managment Portal to throw errors, among other things. General Notes I am sure this is not the only way or necessarily the best way to install the Toolkit but it worked for us on what proved to be a challenging installation. I will continue to add information to this post as to provide clarity in response to questions. A shout out goes to Mark London for helping with the deployment of UDDI, SSO, BizTalk, and the Toolkit in production. Physical View Diagram This is a target production environment. Currently we have 2 BizTalk nodes per Datacenter but will be scaling over the coming year. Our BizTalk Tier is Active/Passive from a Windows Clustering perspective and Active/Active from load balancing perspective for the OnRamps.
Pre Installation Notes
AD Groups and Permissions Service Accounts
AD Groups
BizTalk Server Local Group Permissions
Administrators Group
IIS_IUSRS Group
BizTalk Tier IIS 7 Configuration
App Pool Sites
SQL Permissions
ESB Management Portal Web.Config In order to get the portal working remotely we had to modify the web.config and enable asp.net impersonation. The Portal runs in the user context of the svcesbportal service account. Also, we modified the webHttpBinding to have the following setting: transport clientCrendentialType="Ntlm"
Microsoft.Practices.ESB Application Bindings
During the installation we went with the default bindings provides by the MSI. Doing so will leave the OnRamps running under the BizTalkServerIsolatedHost Receive Handler. We manually created a WCF-WSHttp Adapter Hander and associated it with the OnRamp.Itinerary.WCF ReceiveLocation since it is our primary ESB OnRamp.
SQL Scripts Also, I mention running a script against the ESBExceptionDB. The script is shown in an image. Here is the text for the script.
USE EsbExceptionDb GO GRANT SELECT ON [dbo].[Alert] TO [ESBPortal] GO GRANT SELECT ON [dbo].[Alert] TO [ESBPortalAdmin] GO GRANT SELECT,INSERT,UPDATE,DELETE ON [dbo].[AlertCondition] TO [ESBPortal] GO GRANT SELECT,INSERT,UPDATE,DELETE ON [dbo].[AlertCondition] TO [ESBPortalAdmin] GO GRANT SELECT,INSERT,UPDATE,DELETE ON [dbo].[AlertSubscription] TO [ESBPortal] GO GRANT SELECT,INSERT,UPDATE,DELETE ON [dbo].[AlertSubscription] TO [ESBPortalAdmin] GO GRANT SELECT,INSERT,UPDATE ON [dbo].[Configuration] TO [ESBPortal] GO GRANT SELECT,INSERT,UPDATE ON [dbo].[Configuration] TO [ESBPortalAdmin] GO GRANT SELECT,INSERT,UPDATE ON [dbo].[UserSetting] TO [ESBPortal] GO GRANT SELECT,INSERT,UPDATE ON [dbo].[UserSetting] TO [ESBPortalAdmin] GO GRANT SELECT ON [dbo].[vw_AggregatedFaults] TO [ESBPortal] GO GRANT SELECT ON [dbo].[vw_AggregatedFaults] TO [ESBPortalAdmin] GO GRANT EXECUTE ON [dbo].[usp_select_Fault_Count_Over_Time_By_Application] TO [ESBPortal] GO GRANT EXECUTE ON [dbo].[usp_select_Fault_Count_Over_Time_By_Application] TO [ESBPortalAdmin] GO GRANT EXECUTE ON [dbo].[usp_select_Fault_Count_Over_Time_By_Application_ServiceName] TO [ESBPortal] GO GRANT EXECUTE ON [dbo].[usp_select_Fault_Count_Over_Time_By_Application_ServiceName] TO [ESBPortalAdmin] GO GRANT EXECUTE ON [dbo].[usp_select_Resubmission_Count_Over_Time_By_Application] TO [ESBPortal] GO GRANT EXECUTE ON [dbo].[usp_select_Resubmission_Count_Over_Time_By_Application] TO [ESBPortalAdmin] GO GRANT EXECUTE ON [dbo].[usp_select_AlertSubscription_Count_Over_Time_By_Application] TO [ESBPortal] GO GRANT EXECUTE ON [dbo].[usp_select_AlertSubscription_Count_Over_Time_By_Application] TO [ESBPortalAdmin] GO GRANT EXECUTE ON [dbo].[usp_select_AlertSubscription_Count_Over_Time_By_Application_ServiceName] TO [ESBPortal] GO GRANT EXECUTE ON [dbo].[usp_select_AlertSubscription_Count_Over_Time_By_Application_ServiceName] TO [ESBPortalAdmin] GO GRANT EXECUTE ON [dbo].[usp_select_Resubmission_Count_Over_Time_By_Application_ServiceName] TO [ESBPortal] GO GRANT EXECUTE ON [dbo].[usp_select_Resubmission_Count_Over_Time_By_Application_ServiceName] TO [ESBPortalAdmin] GO
Note: this step may be redundant with the SA permissions that are added later. It was a step that we took in the process ... so its included here.
Installation Instructions