12 Steps to Managing EAI Projects

virtualtechpartners rmayers@[Link]

Overview
Many technology services organizations have now implemented Enterprise Application Integration methodologies or are just now scratching the surface. In many cases organizations find out the hard way that the success of an EAI project has inherited risks that are unique to EAI projects. Therefore normal project management methodologies (while they can and should be applied) are not complete if one does not perform the proper planning activities to ensure the success of the EAI project. This presentation will cover some of the most important steps to ensure the success of your EAI project. These are not the only steps one should consider but they are the most practical steps in approaching EAI. The 12 step strategy is taken from best practices and follows the Keep it Simple rule, as the more complex one makes the EAI domain the more difficult it will be to manage and deliver your project. Familiar concepts and terms such as metadata, schemas, object oriented modeling, analysis and design (OOA & OOD), patterns and requirement gathering techniques are all key to the success of your project. Assigning a project manager with good EAI technical and/or architecture knowledge is also a good thing to have for the success of your project from a risk mitigation perspective (Engage an EAI Project Manager).

Step 1-Understanding your problem domain

Clearly define the problem domain from an end-to-end business perspective; Identify and partner with your Business Owner/s (BO) and Subject Matter Expert/s (SME) Document how they do business; what information systems do they use, business processes, information content, interfaces and business requirements for the new system. Identify and partner with your Technology Services and Technology Support SMEs Document existing integration patterns of the existing system Does the Enterprise have Architecture Standards and/or Products in place? Utilization of the Unified Modeling Language (UML) to document your requirements is highly recommended This step is a basic requirements gathering process. It demands you to engage with the three Ps (below) to identify and document the information that will define the EAI problem domain correctly so it can then be analyzed, modeled and re-engineered 1) People 2) Paper 3) Processes

Step 2-Make sense of the Data

Since all EAI projects exist merely for movement of data, this reality alone justifies your quest to understand what exists in the databases which may also be scattered throughout the enterprise. Ultimately, the implementation of any data-level EAI project comes down to understanding where the data exists, gathering information about the data (i.e. Database Schema and/or Data Dictionary) and applying business principles and/or rules to determine which and what data flows where, why and when? You must document & verify the enterprise metadata required for the success of your EAI project.

Step 3-Make sense of the Processes

Once you understand the enterprise metadata, you have created a baseline for developing your project level metadata model, however you must also decide how to approach the enterprise business model for your EAI project. This decision will depend of several things. First, there is your (potentially limited) view of the enterprise at the process or method level. Your success here will depend on how well you understand and document the business processes and the way they relate to each other. Verification with your business partners is highly recommended. Traditional process modeling techniques such as UML are the best way to document business processes in a way that everyone can understand (your Business & Technical partners). Work from existing and/or known business process in an iterative fashion to better understand the end-to-end business processes; what they do, the sequence of events, are they technology or people processes or a combination of both? Verify assumptions, the only bad question is the one you do not ask

Step 4-Identify Business Events

When something happens (a business event), then there is a resulting reaction. For example when someone applies for a credit application online, this represents an event. This event will trigger something else to happen such as executing a real-time credit check with a consumer credit agency for his/her credit rating, The returned credit rating will invoke other events such as approval or rejection processes to occur. These events are usually asynchronous in nature, however they can also be synchronous as well. In attempting to understand your EAI problem domain your must strive to capture the business events that occur within the domain. It is important to know what invoked a business event, what takes place during the event and any other up or down stream events that your system may be dependant upon. If a business event is dependant on manual intervention remember a true EAI solution should look to eliminate manual intervention entirely (where applicable). Therefore you must understand the business events within your EAI problem domain to analyze where and how an EAI strategy will help.

Step 5-Identify all application interfaces

Application interfaces can be challenging as they will differ from system to system. What may be called an interface in one system may not e deemed an interface by another, whats more complicating is what may be referred to as an application interface by an application support team may not truly be an application interface at all. So the time you spend validating assumptions about application interfaces will be well worth it. The best way to begin with interfaces is to catalog all application interfaces into a directory for the project. As with any other repository of data you will want to store high level data elements about your interfaces; Source System & target System, XML schema or file formats used, communication protocols, is publish or subscribe messaging utilized, messaging middleware, XML over HTTP, Web Services, etc. This Interface Directory along with the Enterprise Metadata captured from step 2 along with the Business Process Model defined in Step 3 will help you to understand the points of integration within all systems that ultimately comprise of the EAI problem domain for your project.

Step 6-Identify data movement

As part of your interface directory, you will want to map the movement of data from system to system. Mapping what data element or elements are moving from what source system to what target system where it will ultimately be stored, further processed or both. You will want to document where the data is physically located, what security may be present, what enabling technology exists and how the information is extracted from the source system and method used to push or pull the data to the target system. Is this an event driven process or if no event is required what condition causes the movement of the data? Is it a batch or real-time process or simply when the state of the data changes (i.e. data replication). Are the systems loosely or physically coupled? Physically couples means both systems must know about each other and if the data formats passed is changed on either side this will trigger change on the other side. Loosely coupled means the exact opposite, the systems do not need to know about each other and there is some type of data transformation logic or product (i.e. middleware) component managing the exchange of data between the two systems.

Step 7-Identify data transformation scenarios

This leads us to Data Transformation scenarios, once you know the data movement you will want to also document in your interface directory the rules for data transformation. This is especially important for several reasons which was touched on in Step 5. 1) The data in one system may not make sense to another system until the data schema and format are reformatted to be compatible with the target system. 2) The EAI strategy and solution must take into account the transformation requirements as well as data formats required by the source and target systems.

Step 8-Apply Technology

There are a wide range of choices of technology toolkits to use within the EAI space including but not limited to application servers, distributed objects, message brokers, business process management engines, etc. The choice of technology will likely be a mix of products and vendors or the Enterprise Architecture team may already have standards on which vendors and or technologies are acceptable for utilization. Technology selection requires a great deal of time and effort, however the requirements you have now compiled in the previous steps will help you to determine the right mix of products and vendors to meet your EAI problem domain. The time it takes to select your technology could take as long as the development of the EAI solution itself, however consider the consequences of choosing the wrong technology that is not in line with the Enterprise Strategy for EAI solutions. This wrong decision will ultimately result in the failure of the EAI project itself.

Step 9-Test, Test, Test

Testing is expensive, time consuming and usually a thankless job. Still if an EAI solution is not tested properly , then disaster looms large. For many EAI toolkits available today, the EAI solution tends to be a black box where all the EAI processing occur with no real sense of tracking once the black box is invoked. Data comes in from the source system and goes out to the target system and life is good. Well this is where negative testing is very critical, the described scenario above is typically called positive testing, its what happens when all in the world is good and everything executes as expected. Negative test cases are developed to see what happens when the data received and/or the systems behavior is not as expected. Develop a test plan that incorporates all positive and negative test case scenarios. Most EAI solutions are mission critical and therefore cannot be taken offline. As result the better your test plan is to thoroughly perform all positive and negative testing, your new EAI solution will be resilient when handling error conditions and reliable when processing valid transactions as expected.

Step 10-Consider performance

The EAI solution must run efficiently with better than average response time in performing end-to-end transactions. If the EAI solution runs less efficiently than the old system it replaced - it will be deemed a failure. While much of the system performance is tied to the Infrastructure the EAI solutions runs on. The choice of EAI enabling product needs to be equally matched with infrastructure components with the processing power capable of handling the expected load of the system. Therefore, Capacity Planning is a very important planning activity that must be done with the right involvement from the right people within the enterprise. As well, load balancing techniques which provide high availability and/or higher system throughput must also be considered as part of the Capacity Planning activities.

Step 11-Define the value

OK the difficult part is over. You must now try to determine the business value of integration the systems in question. The 2 things that must be considered are the hard dollars cost save and the soft dollar savings. The hard dollar savings should be easy to identify, such as the value of having eliminated manual processes, labor cost reduction, reducing error rates, processing transaction more efficiently, etc. However, the soft dollar savings may be harder to define. These would potentially be increased productivity by the Business Unit, ease of use of the system which fosters customer satisfaction and/or the ability of the two disparate Business Units to implement change within their respective systems without effecting their Business Partner up or down stream.

Step 12-Monitoring and Maintenance Procedures

Lastly, you will want to define how system health monitoring will be accomplished. Many EAI products lack having good monitoring tools as part of their EAI toolkit. You may need to look at what monitoring tools the company already has as enterprise monitoring tools first to see if the ones already in usage will help monitor your new EAI solution. Otherwise you may need to look at other monitoring tools and source one from another vendor. Maintenance procedures must also be defined, document all of the maintenance procedures that need to occur. For instance, who will administer the message broker server, middleware servers, application servers and database servers, etc. Who will manage security, who will monitor system performance, what will your problem notification call list look like, who is the primary and secondary contacts, etc. What are your disaster recovery requirements for the applications in question? This may incur additional hardware being procured

12 Steps to Managing EAI Projects

In Summary
1) 2) 3) 4) 5) 6) 7) 8) 9) 10) 11) 12) Understand the Problem Domain Make sense of the Data Make sense of the Processes Identify Business Events Identify all Application Interfaces Identify Data Movement Identify Data Transformation scenarios Apply Technology Test, Test, Test Consider Performance Define the Value Monitoring and Maintenance procedures

Some top 10 reasons why EAI projects fail

1) The EAI implementation is more complex and expensive then expected 2) The Business Unit does not communicate and/or cooperate in drafting/documenting the end-to-end Business Processes 3) There are no standardized methodologies in place and/or a lack in competencies when designing EAI blue prints 4) Entitlements are often not considered for initial implementations 5) EAI implementations differ greatly from application implementations 6) Un-Normalized Data Models and Schema inconsistencies between applications complicate communication and/or data mapping 7) Transactional and Data integrity issues not identified in testing 8) Poor performance of the EAI solution post production implementation 9) Poor end-to-end system management and monitoring 10) Performance testing tools for load testing and profiling tools do not exist