[ Team LiB ]

Chapter 1. Solving Integration Problem

Using Patterns
This chapter illustrates how the patterns in this book can be used to solve a variety o
problems. In order to do so, we examine common integration scenarios and present
integration example. As we design the solution to this example, we express the solut
patterns contained in this book. At the end of this chapter you will be familiar with ab
integration patterns.

[ Team LiB ]
[ Team LiB ]

The Need for Integration

Enterprises are typically comprised of hundreds, if not thousands, of applications tha
acquired from a third party, part of a legacy system, or a combination thereof, opera
tiers of different operating system platforms. It is not uncommon to find an enterpris
different Web sites, three instances of SAP, and countless departmental solutions.

We may be tempted to ask: How do businesses allow themselves to get into such a m
any CIO who is responsible for such an enterprise spaghetti architecture be fired? We
cases, things happen for a reason.

First, writing business applications is hard. Creating a single, big application to run a
is next to impossible. The Enterprise Resource Planning (ERP) vendors have had som
creating larger-than-ever business applications. The reality, though, is that even the
SAP, Oracle, Peoplesoft, and the like perform only a fraction of the business function
typical enterprise. We can see this easily by the fact that ERP systems are one of the
integration points in today's enterprises.

Second, spreading business functions across multiple applications provides the busin
flexibility to select the "best" accounting package, the "best" customer relationship m
software, as well as the "best" order-processing system for its needs. Usually, IT org
interested in a single enterprise application that does it all, nor is such an application
number of individual business requirements.

Vendors have learned to cater to this preference and offer focused applications aroun
function. However, the ever-present urge to add new functionality to existing softwa
caused some functionality spillover among packaged business applications. For exam
systems started to incorporate customer care and accounting functionality. Likewise,
software maker takes a stab at implementing simple billing functions, such as disput
Defining a clear, functional separation between systems is difficult: Is a customer dis
considered a customer care or a billing function?
secure data exchange between multiple enterprise applications.
[ Team LiB ]
[ Team LiB ]

Integration Challenges
Unfortunately, enterprise integration is no easy task. By definition, enterprise integra
with multiple applications running on multiple platforms in different locations, making
integration pretty much an oxymoron. Software vendors offer Enterprise Application
suites that provide cross-platform, cross-language integration as well as the ability to
many popular packaged business applications. However, this technical infrastructure
small portion of the integration complexities. The true challenges of integration span
business and technical issues.

 Enterprise integration requires a significant shift in corporate politics. Business

generally focus on a specific functional area, such as customer relationship ma
billing, and finance. This seems to be an extension of Conway's famous law: "O
which design systems are constrained to produce designs which are copies of t
structures of these organizations." Many IT groups are organized in alignment
functional areas. Successful enterprise integration needs to establish communi
between multiple computer systems but also between business units and IT de
integrated enterprise application, groups no longer control a specific application
application is now part of an overall flow of integrated applications and services

 Because of their wide scope, integration efforts typically have far-reaching imp
business. Once the processing of the most critical business functions is incorpo
integration solution, the proper functioning of that solution becomes vital to th
failing or misbehaving integration solution can cost a business millions of dollar
misrouted payments, and disgruntled customers.

 One important constraint of developing integration solutions is the limited amo

integration developers typically have over the participating applications. In mo
applications are legacy systems or packaged applications that cannot be chang
connected to an integration solution. This often leaves the integration develope
where they have to make up for deficiencies or idiosyncrasies inside the applica
differences between the applications. Often it would be easier to implement pa
inside the application endpoints, but for political or technical reasons, that opti
available.

 Despite the widespread need for integration solutions, only a few standards ha
themselves in this domain. The advent of XML, XSL, and Web services certainly
significant advance of standards-based features in an integration solution. How
around Web services has also given grounds to new fragmentation of the mark
in a flurry of new "extensions" and "interpretations" of the standards. This shou
the lack of interoperability between "standards-compliant" products was one of
stumbling blocks for CORBA, which offered a sophisticated technical solution fo
integration.

 Existing XML Web services standards address only a fraction of the integration
 solutions make deployment, monitoring, and troubleshooting complex tasks th
combination of skill sets. In most cases, these skill sets are spread across man
individuals or do not even exist within IT operations.

Anyone who has been through an EAI deployment can attest that EAI solutions are a
component of today's enterprise strategies—but they make IT life harder, not easier.
between the high-level vision of the integrated enterprise (defined by terms such as
processing, T+1, agile enterprise) and the nuts-and-bolts implementations (What pa
[Link] take again?).

[ Team LiB ]
[ Team LiB ]

How Integration Patterns Can Help

There are no simple answers for enterprise integration. In our opinion, anyone who c
integration is easy must be incredibly smart (or at least a good bit smarter than the
incredibly ignorant (okay, let's say optimistic), or have a financial interest in making
integration is easy.

Even though integration is a broad and difficult topic, we can always observe some p
much better at it than others. What do these people know that others don't? Since th
thing as "Teach Yourself Integration in 21 Days" (this book sure ain't!), it is unlikely
know all the answers to integration. However, they have usually solved enough integ
that they can compare new problems to prior problems they have solved. They know
problems and associated solutions. They learned these patterns over time by trial-an
other experienced integration architects.

The patterns are not copy-paste code samples or shrink-wrapped components, but ra
advice that describe solutions to frequently recurring problems. Used properly, the in
can help fill the wide gap between the high-level vision of integration and the actual
implementation.

[ Team LiB ]
[ Team LiB ]

The Wide World of Integration

We intentionally left the definition of integration very broad. To us it means connecti
systems, companies, or people. While this broad definition gives us the convenience
whatever we find interesting into this book, it is helpful to have a closer look at some
common integration scenarios. Over the course of many integration projects, we rep
across the following six types of integration:

 Information portals
solutions that integration architects build. Many integration projects consist of a com
multiple types of integration. For example, reference data replication is often require
applications into a single, distributed business process.

Information Portal

Many business users have to access more than one system to answer a specific ques
a single business function. For example, to verify the status of an order, a customer
representative may have to access the order management system on the mainframe
system that manages orders placed over the Web. Information portals aggregate info
multiple sources into a single display to avoid having the user access multiple system
Simple information portals divide the screen into multiple zones, each of which displa
from a different system. More sophisticated systems provide limited interaction betw
example, when a user selects an item from a list in zone A, zone B refreshes with de
about the selected item. Other portals provide even more sophisticated user interact
line between a portal and an integrated application.

Data Replication

Many business systems require access to the same data. For example, a customer's
used in the customer care system (when the customer calls to change it), the accoun
compute sales tax), the shipping system (to label the shipment), and the billing syst
invoice). Many of these systems have their own data stores to store customer-relate
When a customer calls to change his or her address, all these systems need to chang
customer's address. This can be accomplished by implementing an integration strate
replication.
In the same way that many business applications store redundant data, they also ten
redundant functionality. Multiple systems may need to check whether a social-securi
whether the address matches the specified postal code, or whether a particular item
makes sense to expose these functions as a shared business function that is impleme
available as a service to other systems.

A shared business function can address some of the same needs as data replication.
could implement a business function called Get Customer Address that could allow ot
request the customer's address when it is needed rather than permanently store a re
The decision between these two approaches is driven by a number of criteria, such a
control we have over the systems (calling a shared function is usually more intrusive
into the database) or the rate of change (an address may be needed frequently but c
infrequently).

Service-Oriented Architecture

Shared business functions are often referred to as services. A service is a well-define

universally available and responds to requests from "service consumers." Once an en
assembles a collection of useful services, managing the services becomes a critical fu
applications need some form of service directory, a centralized list of all available ser
each service needs to describe its interface in such a way that an application can "ne
One of the key drivers of integration is that a single business transaction is often spr
different systems. A previous example showed us that a simple business function suc
order can easily touch half a dozen systems. In most cases, all relevant functions are
inside existing applications. What is missing is the coordination between these applic
we can add a business process management component that manages the execution
function across multiple existing systems.

The boundaries between an SOA and a distributed business can be fuzzy. For examp
expose all relevant business functions as services and then encode the business proc
application that accesses all services via an SOA.

Business-to-Business Integration

So far, we have mainly considered the interaction between applications and business
single enterprise. In many cases, business functions may be available from outside s
business partners. For example, the shipping company may provide a service for cus
compute shipping cost or track shipments. Or a business may use an outside provide
tax rates. Likewise, integration frequently occurs between business partners. A custo
a retailer to inquire on the price and availability of an item. In response, the retailer
supplier for the status of an expected shipment that contains the out-of-stock item.

Many of the above considerations apply equally to business-to-business integration.

communicating across the Internet or some other network usually raises new issues
transport protocols and security. Also, since many business partners may collaborate
"conversation," standardized data formats are critically important.
[ Team LiB ]
[ Team LiB ]

Loose Coupling
One of the biggest buzzwords in enterprise architecture and integration is loose coup
least use a common intermediate language or byte code). The calling method has to
number of expected parameters using agreed-upon data types. The call is immediate
called method starts processing immediately after the calling method makes the call
calling method will resume processing only when the called method completes (mean
is synchronous). Processing will automatically resume in the calling method with the
immediately following the method call. The communication between the methods is i
instantaneous, so neither the caller nor the called method has to worry about securit
form of eavesdropping third parties. All these assumptions make it very easy to write
applications that divide functionality into individual methods to be called by other me
resulting large number of small methods allows for flexibility and reuse.

Many integration approaches have aimed to make remote communications simple by

remote data exchange into the same semantics as a local method call. This strategy
notion of a Remote Procedure Call (RPC) or Remote Method Invocation (RMI), suppo
popular frameworks and platforms: CORBA (see [Zahavi]), Microsoft DCOM, .NET Re
and most recently, RPC-style Web services. The intended upside of this approach is t
synchronous method-call semantics are very familiar to application developers, so w
what we already know? Second, using the same syntax and semantics for both local
remote invocations would allow us to defer until deployment time the decision about
should run locally and which should run remotely, leaving the application developer w
to worry about.

The challenge that all these approaches face lies in the fact that remote communicat
many of the assumptions that a local method call is based on. As a result, abstractin
communication into the simple semantics of a method call can be confusing and misl
colleagues reminded us back in 1994 that "objects that interact in a distributed syste
dealt with in ways that are intrinsically different from objects that interact in a single
space" [Waldo]. For example, if we call a remote service to perform a function for us
want to restrict ourselves to only those services that were built using the same progr
we use? A call across the network also tends to be multiple orders of magnitude slow
call. Should the calling method really wait until the called method completes? What i
interrupted and the called method is temporarily unreachable? How long should we w
be sure we communicate with the intended party and not a third-party "spoofer"? Ho
against eavesdropping? What if the method signature (the list of expected paramete
method changes? If the remote method is maintained by a third party or a business
longer have control over such changes. Should we have our method invocation fail, o
attempt to find the best possible mapping between the parameters and still make the
becomes apparent that remote integration brings up a lot of issues that a local meth
to deal with.

In summary, trying to portray remote communication as a variant of a local method

asking for trouble. Such tightly coupled architectures typically result in brittle, hard-t
poorly scalable solutions. Many Web services pioneers recently (re-)discovered this f
[ Team LiB ]
[ Team LiB ]

One-Minute EAI
To show the effects of tightly coupled dependencies and how to resolve them, let's lo
simple way of connecting two systems. Let's assume we are building an online banki
ll d i i h i f h b k T f
identical in C or Java).

String hostName = "[Link]";

int port = 80;

IPHostEntry hostInfo = [Link](hostName);

IPAddress address = [Link][0];

IPEndPoint endpoint = new IPEndPoint(address, port);

Socket socket = new Socket([Link], [Link], Pro

[Link](endpoint);

byte[] amount = [Link](1000);

byte[] name = [Link]("Joe");

int bytesSent = [Link](amount);

bytesSent += [Link](name);

[Link]();

This code opens a socket connection to the address [Link] and sends tw
amount and the customer's name) across the network. No expensive middleware is r
tools, RPC toolkits—just 10 lines of code. When we run this code, it tells us "7 bytes
can integration be difficult?

There are a couple of major problems with this integration attempt. One of the stren
protocol is its wide support so that we can connect to pretty much any computer con
network regardless of the operating system or programming language it uses. Howev
independence works only for very simple messages: byte streams. In order to conve
byte stream, we used the BitConverter class. This class converts any data type into
using the internal memory representation of the data type. The catch is that the inte
representation of an integer number varies with computer systems. For example, .NE
integer, while other systems may use a 64-bit representation. Our example transfers
the network to represent a 32-bit integer number. A system using 64 bits would be i
bytes off the network and would end up interpreting the whole message (including th
as a single number.

Also, some computer systems store their numbers in big-endian format, while others
little-endian format. A big-endian format stores numbers starting with the highest by
little-endian systems store the lowest byte first. PCs operate on a little-endian schem
passes the following 4 bytes across the network:

232 3 0 0

232 + 3 * 28 equals 1,000. A system that uses big-endian numbers would consider t
mean 232 * 224 + 3 * 216 = 3,892,510,720. Joe will be a very rich man! So this app
under the assumption that all connected computers represent numbers in the same i

The second problem with this simple approach is that we specify the location of the r
our case, [Link]). The Dynamic Naming Service (DNS) gives us one leve
between the domain name and the IP address but what if we want to move the func
Finally, the simple communication also relies on a very strict data format. We are sen
amount data and then a sequence of characters that define the customer's account.
insert a third parameter, such as the name of the currency, we would have to modify
receiver to use the new data format.

Tightly Coupled Interaction

Our minimalist integration solution is fast and cheap, but it results in a very brittle so
two participating parties make the following assumptions about each other:

 Platform technology— internal representations of numbers and objects

 Location— hardcoded machine addresses

 Time— all components have to be available at the same time

 Data format— the list of parameters and their types must match

As we stated earlier, coupling is a measure of how many assumptions parties make a

when they communicate. Our simple solution requires the parties to make a lot of as
Therefore, this solution is tightly coupled.

In order to make the solution more loosely coupled, we can try to remove these depe
one. We should use a standard data format that is self-describing and platform-indep
XML. Instead of sending information directly to a specific machine, we should send it
channel. A channel is a logical address that both sender and receiver can agree on w
of each other's identity. Using channels resolves the location dependency but still req
components to be available at the same time if the channel is implemented using a c
oriented protocol. In order to remove this temporal dependency, we can enhance the
up sent requests until the network and the receiving system are ready. The sender c
requests into the channel and continue processing without having to worry about the
data. Queuing requests inside the channel requires data to be chunked into self-cont
that the channel knows how much data to buffer and deliver at any one time. The tw
depend on a common data format, but we can remove this dependency by allowing f
transformations inside the channel. If the format of one system changes, we only ha
transformer and not the other participating systems. This is particularly useful if man
send data to the same channel

Loosely Coupled Interaction

has to depend on the receiver's internal data format nor on its location. It does not e
attention to whether or not the other computer is ready to accept requests. Removin
dependencies between the systems makes the overall solution more tolerant to chan
of loose coupling. The main drawback of the loosely coupled approach is the addition
is no longer a 10-lines-of-code solution! Therefore, we use a message-oriented midd
infrastructure that provides these services for us. This infrastructure makes exchang
loosely coupled way almost as easy as the example we started with. The next section
components that make up such a middleware solution.

Is loose coupling the panacea? Like everything else in enterprise architecture, there
answer. Loose coupling provides important benefits such as flexibility and scalability,
a more complex programming model and can make designing, building, and debuggi
difficult.

[ Team LiB ]
[ Team LiB ]

A Loosely Coupled Integration Solution

In order to connect two systems via an integration solution, a number of issues have
These functions make up what we call middleware—the glue that sits between applic

Invariably, some data has to be transported from one application to the next. This da
address record that needs to be replicated, a call to a remote service, or a snippet of
a portal display. Regardless of the payload, this piece of data needs to be understood
needs to be transported across a network. Two elements provide this basic function.
communications channel that can move information from one application to the othe
could consist of a series of TCP/IP connections, a shared file, a shared database, or a
carried from one computer to the next (the infamous "sneakernet"). Inside this chan
message—a snippet of data that has an agreed-upon meaning to both applications th
integrated. This piece of data can be very small, such as the phone number of a sing
has changed, or it can be very large, such as the complete list of all customers and t
addresses.

Basic Elements of Message-Based Integration

Now that we can send messages across channels, we can establish a very basic form
However, we promised that simple integration is an oxymoron, so let's see what is m
mentioned that integration solutions often have limited control over the applications
address. As the number of systems increases, this becomes very tedious and require
system to have knowledge about all other systems. Every time a new system is adde
care system would have to be adjusted to the new environment. Things would be a l
middleware could take care of sending messages to the correct places. This is the ro
component, such as a message broker.

Integration solutions can quickly become complex because they deal with multiple ap
formats, channels, routing, and transformation. All these elements may be spread ac
operating platforms and geographic locations. In order to have any idea what is goin
system, we need a systems management function. This subsystem monitors the flow
sure that all applications and components are up and running, and reports error cond
location.

Our integration solution is now almost complete. We can move data from one system
accommodate differences in the data format, route the data to the required systems
performance of the solution. So far, we assumed that an application sends data as a
channel. However, most packaged and legacy applications and many custom applicat
prepared to participate in an integration solution. We need a message endpoint to co
explicitly to the integration solution. The endpoint can be a special piece of code or a
provided by an integration software vendor.

[ Team LiB ]
[ Team LiB ]

Widgets & Gadgets 'R Us: An Example

The best way to understand message-based integration solutions is by walking throu
example. Let's consider Widgets & Gadgets 'R Us (WGRUS), an online retailer that bu
gadgets from manufacturers and resells them to customers.

WGRUS Ecosystem
  Change Address: Customers can use a Web front-end to change their billing a
address.

 New Catalog: The suppliers update their catalog periodically. WGRUS needs t
pricing and availability based on the new catalogs.

 Announcements: Customers can subscribe to selective announcements from

 Testing and Monitoring: The operations staff needs to be able to monitor all
components and the message flow between them.

We tackle each of these requirements separately and describe the solution alternativ
using the pattern language introduced in this book. We will start with a simple messa
architecture and introduce more complex concepts, such as a Process Manager, as w
increasingly complex requirements.

Internal Systems
As in most integration scenarios, WGRUS is not a so-called "green field" implementat
integration of an existing IT infrastructure comprised of a variety of packaged and cu
The fact that we have to work with existing applications often makes integration wor
our example, WGRUS runs the following systems (see figure).

WGRUS IT Infrastructure

WGRUS has four different channels to interact with customers. Customers can visit t
site, call the customer service representative at the call center, or submit orders via
can also receive notifications via e-mail.

WGRUS's internal systems are comprised of the accounting system, which also includ
functions, and the shipping system that computes shipping charges and interacts wit
packaged application, while the Web site is a custom J2EE application. The inbound f
requires manual data entry into a small Microsoft Access application. We want to trea
equally, regardless of their source. For example, a customer should be able to place
call center and check the order status on the Web site.

Because placing an order is an asynchronous process that connects many systems, w

implement a message-oriented middleware solution to streamline the order entry pro
packaged call center application was not developed with integration in mind so that w
it to the messaging system using a Channel Adapter. A Channel Adapter is a compon
attach to an application and publish messages to a Message Channel whenever an ev
the application. With some Channel Adapters, the application may not even be aware
of the adapter. For example, a database adapter may add triggers to specific tables s
the application inserts a row of data, a message is sent to the Message Channel. Cha
also work in the opposite direction, consuming messages off a Message Channel and
action inside the application in response.

We use the same approach for the inbound fax application, connecting the Channel A
application database. Because the Web application is custom built, we implement the
Endpoint code inside the application. We use a Messaging Gateway to isolate the app
the messaging-specific code.

Taking Orders from Three Different Channels

Because each system uses a different data format for the incoming orders, we use th
Translators to convert the different data formats into a common New Order message
Canonical Data Model. A Canonical Data Model defines message formats that are ind
specific application so that all applications can communicate with each other in this c
the internal format of an application changes, only the Message Translator between t
application and the common Message Channel has to change, while all other applicat
Translators remain unaffected. Using a Canonical Data Model means that we deal wit
messages: canonical (public) messages and application-specific (private) messages.
specific messages should not be consumed by any component other than that applica
associated Message Translator. To reinforce this policy, we name application-specific
starting with the name of the application: for example WEB NEW ORDER In contrast
The NEW_ORDER Message Channel is a so-called Datatype Channel because it carries m
one type: new orders. This makes it easy for message consumers to know what type
expect. The New Order message itself is designed as a Document Message. The inten
is not to instruct the receiver to take a specific action, but rather to pass a document
recipient, who is free to decide how to process the document.

Processing Orders
Now that we have a consistent order message that is independent from the message
ready to process orders. To fulfill an order, we need to complete the following steps:

 Verify the customer's credit standing. If the customer has outstanding bills, we
new order.

 Verify inventory. We can't fulfill orders for items that are not in stock.

 If the customer is in good standing and we have inventory, we want to ship the
the customer.

We can express this sequence of events using a Unified Modeling Language (UML) ac
Activity diagrams have relatively simple semantics and are a good tool to depict proc
parallel activities. The notation is very simple; sequential activities are connected by
Parallel activities are connected by a thick black bar representing fork and join action
causes all connected activities to start simultaneously, while the join action continues
incoming activities have been completed.

Our activity diagram (see figure) is designed to execute the Check Inventory task an
Customer Standing task in parallel. The join bar waits until both activities are comple
allows the next activity to start. The next activity verifies the results of both steps: D
inventory, and is the customer in good standing? If both conditions are fulfilled, the p
fulfill the order. Otherwise, we transition to an exception-handling activity. For exam
remind the customer to pay the last invoice or send an e-mail letting him or her know
will be delayed. Because this book focuses on the design aspects of message-oriente
rather than on workflow modeling, we leave the details of the exception-handling pro
now. For a very good discussions of workflow architecture and workflow modeling, re
and [Sharp].

Activity Diagram for Order Processing

It turns out that the activities in the activity diagram map relatively nicely to the sys
IT department. The accounting system verifies the customer's credit standing, the in
check the inventory, and the shipping system initiates the physical shipping of goods
system also acts as the billing system and sends invoices. We can see that the proce
a typical implementation of a distributed business process.

To convert the logical activity diagram into an integration design, we can use a Publis
Channel to implement the fork action and an Aggregator to implement the join action
Subscribe Channel sends a message to all active consumers; an Aggregator receives
messages and combines them into a single, outgoing message (see figure).

Order Processing Implementation Using Asynchronous Messaging

In our example, the Publish-Subscribe Channel sends the New Order message to bot
system and the inventory system. The Aggregator combines the result messages fro
widgets and one for gadgets. As a result, we have to route the request for inventory
system. Because we want to hide the peculiarities of the inventory systems from the
insert another Content-Based Router that routes the message to the correct inventor
on the type of item ordered (see figure). For example, all incoming messages with an
starting with W are routed to the widget inventory system, and all orders with an ite
with G are routed to the gadget inventory system.

Routing the Inventory Request

Note that the intent of messages on the Point-to-Point Channels between the Conten
and the inventory systems is different from the previous channel. These channels co
Messages, messages that instruct the system to execute the specified command, in t
the inventory of an item.

Because the widget inventory system and the gadget inventory system use different
formats, we again insert Message Translators to convert from the canonical New Ord
to a system-specific format.

What happens if the order item starts neither with W nor G? The Content-Based Rou
message to the INVALID_ORDER channel so that the invalid order can be processed a
notifying the customer, for example. This channel is a typical example of an Invalid M
It highlights the fact that the meaning of a message changes depending on what cha
the NEW_ORDER channel and the INVALID_ORDER channel transport the same type of m
one case a new order is being processed, while in the other case the order is deemed

So far, we have assumed that each order contains only a single item. This would be
inconvenient for our customers because they would have to place a new order for ea
would end up shipping multiple orders to the same customer and incur unnecessary
However, if we allow multiple items inside an order, which inventory system should v
inventory for this order? We could use a Publish-Subscribe Channel to send the order
system to pick out the items that it can process. But what would then happen to inva
would we notice that neither inventory system processed the item? We want to main
control the Content-Based Router gives us, but we need to be able to route each ord
individually.

Therefore, we insert a Splitter, a component that breaks a single message into multi
messages. In our case, the Splitter splits a single Order message into multiple Order
Naturally, when the inventory for all items has been verified, we need to recombine t
a single message. We already learned that the component that can combine multiple
single message is the Aggregator. Using both a Splitter and an Aggregator, we can lo
the message flow for individual order items from the flow for a complete order.

When designing an Aggregator, we have to make three key decisions:

 Which messages belong together (correlation)?

 How do we determine that all messages are received (the completeness condit

 How do we combine the individual messages into one result message (the agg
algorithm)?

Let's tackle these issues one by one. We can't correlate order items by the customer
customer may place multiple orders in short succession. Therefore, we need a unique
order. We accomplish this by inserting a Content Enricher into the part of the solutio
(see figure). A Content Enricher is a component that adds missing data items to an i
In our case, the Content Enricher adds a unique order ID to the message.

Taking Orders with Enricher

a Composed Message Processor. To simplify the figure, we insert the symbol for a Co
Processor into the original message flow diagram:

Revised Order Process Implementation

Checking Status
Despite connecting the systems via Message Channels, fulfilling an order can take so
time. For example, we may be out of a certain item and the inventory system may b
Inventory Check message until new items arrive. This is one of the advantages of as
messaging: the communication is designed to happen at the pace of the components
inventory system is holding the message, the accounting system can still verify the c
standing. Once both steps are completed, the Aggregator publishes the Validated Ord
initiate shipment and invoicing.

A long-running business process also means that both customers and managers are
know the status of a specific order. For example, if certain items are out of inventory
may decide to process just those items that are in stock. Or, if the customer has not
goods, it is useful if we can tell him or her that the goods are on their way (including
company's tracking number) or that there is an internal delay in the warehouse.

Tracking the status of an order with the current design is not so easy. Related messa
multiple systems. To ascertain the status of the order in the sequence of steps, we w
the "last" message related to this order. One of the advantages of a Publish-Subscrib
we can add additional subscribers without disturbing the flow of messages. We can u
listen in to new and validated orders and store them in a Message Store. We could th
Message Store database for the status of an order (see figure):

Adding a Message Store to Track Order Status

In situations where we use a Point-Point Channel, we cannot simply add a subscriber

because a Point to Point Channel ensures that each message is only consumed by a
Storing message data in a central database has another significant advantage. In the
each message had to carry extraneous data in order to continue processing the mess
line. For example, the Validate Customer Standing message may have had to transp
customer data even though it required only the customer ID. This additional data is n
the resulting message still contains all data from the original order message. Storing
message in a Message Store at the beginning of the message flow has the advantage
subsequent components can refer to the Message Store for important message data
intermediate steps having to carry the data along. We refer to such a function as Cla
messages can "check in" data for later retrieval. The downside of this approach is tha
central data store is not as reliable as sending messages across asynchronous Messa

Now, the Message Store is responsible for maintaining data related to the new messa
progress of the message within the process. This data gives us enough information t
Store to determine the next required steps in the process rather than connecting com
fixed Message Channels. For example, if the database contains reply messages from
systems and the billing system, we can conclude that the order has been validated a
message to the shipping and billing systems. Instead of making this decision in a sep
component, we can do it right in the Message Store. Effectively, we are turning the M
a Process Manager.

A Process Manager is a central component that manages the flow of messages throu
The Process Manager provides two main functions:

 Storing data between messages (inside a "process instance")

 Keeping track of progress and determining the next step (by using a "process t

Processing Orders with a Process Manager

This architecture turns the individual systems (e.g., the inventory systems) into shar
functions that can be accessed by other components as services, thus increasing reu
maintenance. The services can be wired together via a message flow (for example, u
Message Processor to check inventory status for each order item) or orchestrated via
Manager. Using a Process Manager makes changing the flow of messages much easie
previous approach.

The new architecture exposes all services to a common services bus so that they can
any other component. We could turn the WGRUS IT infrastructure into an SOA by ad
look up ("discover") a service from a central service registry. In order to participate i
service would have to provide additional functions. For example, each service would
interface contract that describes the functions provided by the service. Each request-
needs to support the concept of a Return Address. A Return Address allows the caller
consumer) to specify the channel where the service should send the reply message.
to allow the service to be reused in different contexts, each of which may require its
reply messages.

The Process Manager itself uses a persistent store (typically files or a relational datab
associated with each process instance. To allow the Web interface to query the statu
could send a message to the Process Manager or the order database. However, chec
synchronous process—the customer expects the response right away. Because the W
custom application, we decide to access the order database directly to query the ord
form of Shared Database is the simplest and most efficient approach, and we are alw
the Web interface displays the most current status. The potential downside of this ap
Web interface is tightly coupled to the database, a trade-off that we are willing to tak

One difficulty in exposing systems as services results from the fact that many legacy
built with features such as Return Address in mind. Therefore, we "wrap" access to t
with a Smart Proxy. This Smart Proxy enhances the basic system service with additio
that it can participate in an SOA. To do this, the Smart Proxy intercepts both request
messages to and from the basic service (see figure)
The Smart Proxy can store information from the request message (e.g., the Return A
by the requestor) and use this information to process the reply message, (e.g., route
reply channel). A Smart Proxy is also very useful to track quality of service (e.g., res
external service.

Change Address
WGRUS needs to deal with a number of addresses. For example, the invoice has to b
customer's billing address, while the goods are shipped to the shipping address. We
customer to maintain all these addresses through the Web interface to eliminate unn
steps.

We can choose between two basic approaches to get the correct billing and shipping
billing and shipping systems:

 Include address data with every New Order message

 Store address data in each system and replicate changes

The first option has the advantage that we can use an existing integration channel to
additional information. A potential downside is the additional data flowing across the
infrastructure; we pass the address data along with every order even though the add
much less frequently.

When implementing the first option, we need to consider that the billing and shipping
packaged applications and were likely not designed with integration in mind. As such
to be able to accept addresses with a new order but rather use the address that is st
database. To enable the systems to update the address with the New Order message
execute two functions in the billing system (and the shipping system): First, we mus
address, and then we must send the bill (or ship the goods). Because the order of th
matters, we insert a simple Process Manager component that receives a New Order m
includes the current shipping and billing addresses and publishes two separate messa
(or shipping) system (see figure).

Including Address Data in the New Order Message

orders). It can also reduce coupling between systems. Any system that uses an addr
to the ADDRESS_CHANGE channel without affecting any other systems.

Because we are dealing with multiple types of addresses (shipping and billing addres
make sure that only the right type of address is stored in each system. We need to a
address change message to the shipping system if the address is a billing address. W
by using Message Filters that pass only messages matching certain criteria (see figur

We also use Message Translators to translate the generic Address Change message i
message format used by the applications. In this case, we do not have to use a Mess
the Web interface because we define the Canonical Data Model as equal to the forma
interface application. This could limit our flexibility if we want to introduce other way
addresses in the future, but for now it is sufficient.

Propagating Address Changes via a Separate Publish-Subscribe Channel

Both the shipping and the billing systems store addresses in a relational database, so
database Channel Adapter to update the data in each system.

How do we decide between the two options? In our situation, the message traffic is n
concern because we process only a few hundred orders a day, so either solution wou
efficient. The main decision driver is going to be the internal structure of the applicat
be able to insert the addresses directly into the database, but rather through the app
layer. In this case, the applications may perform additional validation steps and reco
change activity. The system may even be programmed to e-mail a confirmation mes
customer every time the address changes. This would get very annoying if the updat
every order. Such a condition would favor propagating address changes using dedica
are sent only when the customer actually changes the address.

In general, we prefer well-defined, self-contained business actions such as "Change A

"Place Order" because they give us more flexibility in orchestrating the businesses pr
comes down to a question of granularity and the associated trade-offs. Fine-grained
to sluggish systems due to an excessive number of remote calls being made or mess
For example, imagine an interface that exposes a separate method to change each a
approach would be efficient if the communication happens inside a single application
those fields that changed. In an integration scenario, sending six or seven messages
address would be a significant overhead, plus we would have to deal with synchroniz
messages. Fine-grained interfaces also lead to tight coupling. If we change the addre
catalog is driven by the offerings from the respective suppliers. However, one of the
WGRUS provides to its customers is allowing them to view widgets and gadgets on th
to order both types of items in a single order. This function is an example of an Infor
scenario—we combine information from multiple sources into a single view.

It turns out that both suppliers update their product catalog once every three month
makes relatively little sense to create a real-time messaging infrastructure to propag
changes from the suppliers to WGRUS. Instead, we use File Transfer integration to m
from suppliers to WGRUS. The other advantage of using files is that they are easily a
transported across public networks using FTP or similar protocols. In comparison, mo
messaging infrastructures do not work well over the public Internet.

We still can use translators and adapters to transform the data to our internal catalo
However, these translators process a whole catalog at once instead of one item at a
approach is much more efficient if we are dealing with large amounts of data in the s

Updating Catalog Data via File Transfer

Announcements
In order to improve business, we want to announce specials to our customers every
avoid annoying the customers, we allow each customer to specify which messages in
also want to target specific messages to a specific subset of customers. For example
announce special deals only to preferred customers. When we need to send informat
recipients, a Publish-Subscribe Channel immediately comes to mind. However, a Pub
Channel has some disadvantages. First, it allows any subscriber to listen to the publi
without the publisher's knowledge. For example, we would not want smaller custome
special offers intended for high-volume customers. The second downside of Publish-S
is that they work efficiently only on local networks. If we send data across wide-area
Publish-Subscribe Channel, we have to send a separate copy of the message to each
recipient is not interested in the message, we would have incurred unnecessary netw

Therefore, we should look for a solution that allows subscribers to issue their subscri
and then send individual messages only to interested (and authorized) customers. To
function, we use a Dynamic Recipient List. A dynamic Recipient List is the combinatio
Routing patterns. A Recipient List is a router that propagates a single message to a s
The main difference between the Recipient List and a Publish-Subscribe Channel is th
List addresses each recipient specifically and therefore has tight control over who rec
If customers receive announcements via e-mail, the implementation of these pattern
mailing list's features typically supplied by e-mail systems. Each recipient channel is
an e-mail address. Likewise, if customers prefer to receive announcements via a Web
interface, each recipient channel is implemented by a SOAP request, and the channe
URI of the Web service. This example illustrates that the patterns we use to describe
design are independent of a specific transport technology.

Testing and Monitoring

Monitoring the correct execution of messages is a critical operations and support fun
Store can provide us with some important business metrics, such as the average tim
order. However, we may need more detailed information for the successful operation
solution. Let's assume we enhance our solution to access an external credit agency t
our customer's credit standing. Even if we show no outstanding payments, we may w
customer's order if the customer's credit ranking is particularly poor. This is especial
customers who do not have a payment history with us. Because the service is provid
provider, we are charged for its use. To verify the provider's invoice, we want to trac
and reconcile the two reports. We cannot simply go by the number of orders, becaus
logic may not request an external credit check for long-standing customers. Also, we
quality of service (QoS) agreement with the external provider. For example, if the re
exceeds a specified time, we may not have to pay for the request.

To make sure we are being billed correctly, we want to track the number of requests
time it takes for the associated response to arrive. We have to be able to deal with tw
situations. First, the external service can process more than one request at a time, s
able to match up request and reply messages. Second, since we treat the external se
service inside our enterprise, we want to allow the service consumer to specify a Ret
channel where the service should send the reply message. Not knowing to which cha
being sent can make it difficult to match request and reply messages.

Once again, the Smart Proxy is the answer. We insert the Smart Proxy between any
and the external service. The Smart Proxy intercepts each request to the service and
Return Address specified by the service consumer with a fixed reply channel. This ca
send all reply messages to the channel specified by the Smart Proxy. The proxy store
Return Address so that it can forward the reply message to the channel originally spe
consumer. The Smart Proxy also measures the time elapsed between request and re
Besides tracking our usage of the external credit service, we also want to make sure
working correctly. The Smart Proxy can report to the management console cases wh
message is received within a specified time-out period. Much harder to detect are ca
external service returns a reply message but the results in the message are incorrect
the external service malfunctions and returns a credit score of zero for every custom
up denying every order. There are two mechanisms that can help us protect against
First, we can periodically inject a Test Message into the request stream. This Test Me
the score for a specific person so that the result is known. We can then use a test da
not only that a reply was received but also the accuracy of the message content. Bec
Proxy supports Return Addresses, the test data generator can specify a special reply
separate test replies from regular replies (see figure).

Inserting Test Messages to Verify Accurate Results

Another effective strategy to detect malfunctioning services is to take a statistical sa

example, we may expect to decline an average of less than one in 10 orders due to c
standing. If we decline more than five orders in a row, this may be an indication that
service or some business logic is malfunctioning. The management console could e-m
to an administrator, who can then take a quick look at the data to verify whether the
justified.
[ Team LiB ]
[ Team LiB ]
The integration scenario in this chapter focuses primarily on how to connect existing
detailed description of how to publish and consume messages from inside a custom a
examples in Chapter 6, "Interlude: Simple Messaging," and Chapter 9, "Interlude: Co
Messaging."

The remainder of the book contains detailed descriptions and code examples for each
that we used in our solution design, as well as many related patterns. The patterns a
their primary intent between base patterns, channel patterns, message patterns, rou
transformation patterns, endpoint patterns, and system management patterns. This
makes it easy to read all patterns in sequence or to look up individual patterns as a r

[ Team LiB ]