Closing the Loop: Using SOA to Automate Human Interaction
At a high level, the flow for both business analytical feedback loops and business exception correction mechanisms are very similar. However, the technical details reveal how different the two concepts really are.
Both systems start with data flows from real-time transactional systems into some sort of central data repository. We've used the term data mart in this paper to indicate a potentially focused collection of specific data artifacts designed to support the functions discussed in this paper, but the term is not intended to require any specific organization of data or level of detail. In fact, depending on the data requirements for the analytical tool, the repository could be an Operational Data Store (ODS), a Data Warehouse (DW), a Data Mart (DM), or something completely different. We've used a fairly neutral term to focus on the analytical output and how it is used rather than the type(s) of data repository. In general, we assume that most of the data flows would be implemented using a traditional ETL tool. The key would be to house data in a repository that does not effect the performance metrics of a real-time transactional system.
In a business exception correction mechanism, the analytical tool behaves like a Business Activity Monitoring solution (BAM), generating an event to which the SEAI subscribes. The SEAI does not need to filter or transform the event to act upon it because the event is specific to a pre-defined business condition. The SEAI may need to transform the event during its processing, but this type of transformation is generally simple reformatting. The metadata in this case is also fairly simple, describing the business systems that want to subscribe to the event. Once the SEAI determines that a system is potentially interested in an event, it searches its metadata to determine how to format the event as an action or request that can be consumed by the original transactional system. In general, the interpretation of a given rule against the event will result in zero or one operations against the subscribing system.
Business analytical feedback loops are potentially much more complex. The analytical output is likely to take the form of a DSS-style document that was designed for human consumption. The SEAI still uses metadata to determine which systems are interested in data within the document. However, the document could contain multiple, distinct information components that a system might want to know about. Multiple complex filter-and-transformation rules may need to be applied in SEAI to convert the source document into a collection of events. This potential collection of events is then processed against the defined set of business rules to determine what operations, if any, need to be invoked on which real-time transactional systems.
Note: This approach requires that there be some sort of direct system (non-human) interface that the SEAI can invoke. If no interface exists or if the interface is implemented in an inaccessible technology, some type of adapter may be needed to integrate with the system. The overall operational latency of the data-flow into the DSS implementation needs to be considered, as well; it is important that any time-sensitive updates from the transactional system are applied to the DSS and the events are interpreted before the event notification becomes obsolete.
Although this article talks in terms of two distinct mechanisms, it is perfectly reasonable and feasible to apply both mechanisms to the same application. In this case, the rules controlling both mechanisms must be carefully designed so that actions taken by one mechanism do not interfere with actions taken by the other.
Finally, this article relies on an analytical tool to perform the data analysis and uses the SEAI to "bridge the gap" between analysis and corrective action. The concept is broader than this specific design, however. Similar mechanisms easily can be designed and developed between any two systems that share a common related business concept with different business language or business semantics.
All the general discussion can be better understood in the context of a detailed example. The example we'll use is based on a retail sales system tracking actual sales data against inventory levels. The retailer might have originally created an operational data store to consolidate sales data from both eCommerce and brick-and-mortar stores and compare them to historical sales data, a typical reporting structure for a DSS. After living with the reporting system for a while, the retailer wants to take the next step: using an SEAI to evolve the data from the reporting system needing manual intervention into an automated response.
For this example, the reporting system is providing a sales forecast report typical of a DSS. The report output might contain a series of entries by product. For example:
Event Type=Retail Demand Forecast
- Product ID 010
- Product Price 10.00
- Markdown 10% End Date = 1/1/06
- Forecast Demand 110% of plan
These entries would be compared to a collection of rules organized by the event type, such as:
Event Type=Retail Demand Forecast
- If demand >=110% && demand < 115% then service = adjust pricing information; action = markdown effective date += 10
- Else If demand > 115% then service = adjust pricing information; action = markdown effective date += 20
- Else If demand >= 90% && demand < 95% then service = adjust pricing information; action = markdown effective date += 5
The event document given in the example would trigger rule A, which would result in the effective date of a scheduled markdown being pushed back for 10 days. This is an example of a DSS report used to implement a feedback loop that adjusts the effective dates of planned price mark-downs.
Note: The rules may involve simple calculations with percentages applied to the parameters to arrive at the effective dates. Also, they may take on more complex forms such as invocation of a more robust statistical price-optimization engine service. The rules engine and its functional components are beyond the scope of this discussion. Also note that the format and the specification of the metadata layer (whether it consists of WSDL and a UDDI registry or something else) that provides the list of service definitions and operational end-points for the subscribing systems is also beyond the scope of this discussion.
Implementation of SEAI
The following diagram shows a view of an SEAI implementation.
In this diagram, Message 1 represents an event generated by the analytical tool. If the payload represents a BAM-style event, it can be consumed easily. However, if the payload is a DSS-style document, Message 2 represents a call to the rule interpreter to convert the document into a collection of events (Message 3). For each event in the collection, a call is made to the rules interpreter (Message 4) to see whether the event should be translated into additional service calls. Internally, the rules interpreter also uses the metadata component to identify which systems are subscribing to which events and how the event should be formatted for each transactional system (Message 5). Message 6 represents the zero or more operations going back to the transactional systems.
Depending on the environment, the outbound service calls might be mediated by external adapters, point-to-point interfaces, or by an Enterprise Service Bus (ESB). The process controller in this case is only responsible for message transport and sequencing; all of the metadata and rules interpretation is included within the rules interpreter. A typical realization of this design might use a business process engine for the process controller, a session façade for the rules interpreter invocation, and a strategy pattern behind the façade. The event-specific logic should be configured by metadata, but "one-of" custom components can always be used to meet unique event requirements.
The following class diagram represents one possible realization of an SEAI.
Note that there are certainly other approaches to designing and implementing the type of interpretive system described in this paper. The example above is intended to illustrate the author's thought on the detailed design of an interpreter and not to provide a canonical pattern for its implementation.
Traditional uses of analytical tools tend to focus on output that will be presented to a human for interpretation. The purpose of the SEAI concept is to provide a mechanism to use the multi-dimensional analysis features of best-of-breed analytical tools to create automated responses in terms of existing system attributes and operations. The SEAI component is designed from the ground-up to be an Event-Driven Architecture (EDA), which makes it easy to use its services in enterprise SOA ecosystems to connect them in a loosely coupled asynchronous manner. Taken as a whole, these concepts allow:
- Extension of existing systems with minimal organic code impact
- Application of existing investments in analytical tools to create advanced self-correcting ("closed loop") systems without writing complex custom code to perform the data analysis
- Creation of complex system interactions that are driven by configuration and not coding
- Compliance with emerging SOA ecosystems based on ESB products
About the Authors
Surekha Durvasula is an enterprise architect who has had 10 years in designing and architecting a variety of enterprise applications in the financial services sector and the retail industry vertical. Most of her work has been in the area of distributed N-tiered architecture including J2EE component architecture. Her more recent focus has been on Service Oriented Architecture and Business Process Management. Her efforts as an Enterprise Architect involve not only architecting new applications and business services but also in leveraging the principles of SOA to extend the life of existing Enterprise Information Systems.
Dave Lindema is an Architect with BEA Professional Services and has over 20 years of total industry experience.
Bob Hensle has more that 19 years of IT industry experience. He has
designed a variety of applications on multiple technologies for many
companies including General Motors, HP, Level 3 Communications, Federal
Express, Galileo International, and MCI. For the past eight years he has
been a consultant with BEA Systems and most recently has been focused on
Service Oriented Architectures built using the BEA WebLogic and
AquaLogic product families.
Page 2 of 2