One of the big advantages of Web services is their ability to wrap existing system interfaces and make them more easily accessible. This is particularly useful for companies that have significant investments in tools that provide excellent functionality but were not designed for their ease of interface. Coupling Web services with a service-oriented architecture broadens the scope of these tools. Many companies are actively seeking to service-enable key systems so that they can be used across the enterprise as a part of a SOA ecosystem. In this paper, we’re seeking to add another type of key system to the list, the analytical tool. These tools have become extremely common, and many companies are using them in conjunction with Extract, Transformation, and Load (ETL) tools to turn detailed transactional data into key information assets. Most of these analytical solutions are used to create reports that are reviewed by humans who then determine what, if any, action should be taken.
Analytical tools have proven to be very successful, providing businesses with invaluable information about business performance. However, having the events interpreted by human actors still has its share of possible problems:
- Cost of hiring and training people to review the reports
- Time for a person to review, understand, and act upon the reports
- Human error in interpreting the reports
Service-enabling an analytical tool allows these information assets to be used programmatically to extend existing operational systems. We’re proposing a specific type of component (a Service-Enabled Analytical Interpreter—SEAI) to service-enable an analytical tool. Using this new component, we then demonstrate how to use it to build self-correcting business systems by implementing Business Analytical Feedback Loops and Business Exception Correction mechanisms.
This article presents several key concepts:
- A business analytical feedback loop uses an external component (SEAI) to logically extend an attribute or process. This functionality extends an existing use case and provides data or additional functionality to enhance the normal processing of the system. The new feature might provide an alternate flow for an existing use case. One clear characteristic of a feedback loop is that it represents a “fine-tuning” of system behavior using existing business concepts.
- A business exception correction mechanism uses an external component to detect exceptional business conditions. The condition is represented as an event and triggers behavior that handles the exceptional flow using services and data owned by the existing application.
- A Service-Enabled Analytical Interpreter (SEAI) provides a remote service interface that subscribes to the analytical tool to transform a variety of logical analytical events that are interpreted by the SEAI and converted into behavior that is supported by the existing system. This allows companies to apply the multi-dimensional analytics capabilities of analytical tools programmatically.
Applications
There are undoubtedly many applications for an SEAI, but we’re focusing on the two we’ve just defined: business analytical feedback loops and business exception correction mechanisms. In this section, we’ll explain how an SEAI could be used to implement these concepts.
Business Analytics Feedback Loop
The following picture presents a logical view of a business analytical feedback loop.
The feedback loop shows the logical flow of order placement data (either being published by the existing system or extracted using a traditional ETL tool) from the purchase order system and the flow of sales information from the Point of Sale (POS) system into the data mart. The analytical tool creates a report comparing actual sales trends to predicted demand. The report shows demand tracking ahead of the predicted level. A “Demand Update Event” is generated by the analytical tool and published to the SEAI. The SEAI uses metadata to determine what systems are potentially interested in the event, and applies rules to the event to determine what action, if any, to take based on the revised actual demand numbers. In this case, a threshold rule is triggered that indicates the start date for any existing markdowns for the product should be pushed out.The SEAI interprets the rule and adjusts the existing markdown’s start date based on the revised demand profile.
This example shows several key aspects of a business analytical feedback loop. First, the analytical functionality provides the same kinds of information that one would expect from a traditional Decision Support System (DSS). Secondly, the SEAI can use a combination of metadata and rules to establish the relationships between analytical events, all known subscribing systems, and the specific rules that need to be evaluated to determine what actions (if any) should be taken. Finally, the overall function of a feedback loop is to “fine-tune” the existing behavior of the existing Pricing system based on actual inventory and sales data without requiring human intervention.
Business Exception Correction Mechanism
The following diagram presents an example of a business exception correction mechanism.
The diagram shows data flowing from the warehouse inventory system to the data mart in response to store inventory requests. The SEAI rules compare the current inventory levels to planned demand. This comparison shows demand running ahead of plan and triggers a “Low Inventory Level” alert. The action associated with the rule causes the SEAI to create a new purchase order request in the purchase order system. Fulfilling this request will bring actual inventory levels into compliance with the revised demand estimate and allow the system to continue normal functioning without depleting existing inventory.
These mechanisms are distinct but both share some key concepts. Both use some sort of secondary (non-transactional) data store. Both use metadata and rules to establish relationships and determine actions, but the results are different. The Business Exception Correction Mechanism creates an automated response that leverages an existing function to prevent an actual error (inventory out-of-stock). The Business Analytics Feedback Loop “fine tunes” existing attributes in response to non-exceptional business conditions thereby improving operational business efficiency and profitability. The behavior of both systems is event-driven and message-based.
Detailed Discussion
At a high level, the flow for both business analytical feedback loops and business exception correction mechanisms are very similar. However, the technical details reveal how different the two concepts really are.
Both systems start with data flows from real-time transactional systems into some sort of central data repository. We’ve used the term data mart in this paper to indicate a potentially focused collection of specific data artifacts designed to support the functions discussed in this paper, but the term is not intended to require any specific organization of data or level of detail. In fact, depending on the data requirements for the analytical tool, the repository could be an Operational Data Store (ODS), a Data Warehouse (DW), a Data Mart (DM), or something completely different. We’ve used a fairly neutral term to focus on the analytical output and how it is used rather than the type(s) of data repository. In general, we assume that most of the data flows would be implemented using a traditional ETL tool. The key would be to house data in a repository that does not effect the performance metrics of a real-time transactional system.
In both cases, the analytical tool creates output to which the SEAI reacts. However, the type of output and the way in which the SEAI must process the information is very different.
In a business exception correction mechanism, the analytical tool behaves like a Business Activity Monitoring solution (BAM), generating an event to which the SEAI subscribes. The SEAI does not need to filter or transform the event to act upon it because the event is specific to a pre-defined business condition. The SEAI may need to transform the event during its processing, but this type of transformation is generally simple reformatting. The metadata in this case is also fairly simple, describing the business systems that want to subscribe to the event. Once the SEAI determines that a system is potentially interested in an event, it searches its metadata to determine how to format the event as an action or request that can be consumed by the original transactional system. In general, the interpretation of a given rule against the event will result in zero or one operations against the subscribing system.
Business analytical feedback loops are potentially much more complex. The analytical output is likely to take the form of a DSS-style document that was designed for human consumption. The SEAI still uses metadata to determine which systems are interested in data within the document. However, the document could contain multiple, distinct information components that a system might want to know about. Multiple complex filter-and-transformation rules may need to be applied in SEAI to convert the source document into a collection of events. This potential collection of events is then processed against the defined set of business rules to determine what operations, if any, need to be invoked on which real-time transactional systems.
Note: This approach requires that there be some sort of direct system (non-human) interface that the SEAI can invoke. If no interface exists or if the interface is implemented in an inaccessible technology, some type of adapter may be needed to integrate with the system. The overall operational latency of the data-flow into the DSS implementation needs to be considered, as well; it is important that any time-sensitive updates from the transactional system are applied to the DSS and the events are interpreted before the event notification becomes obsolete.
Although this article talks in terms of two distinct mechanisms, it is perfectly reasonable and feasible to apply both mechanisms to the same application. In this case, the rules controlling both mechanisms must be carefully designed so that actions taken by one mechanism do not interfere with actions taken by the other.
Finally, this article relies on an analytical tool to perform the data analysis and uses the SEAI to “bridge the gap” between analysis and corrective action. The concept is broader than this specific design, however. Similar mechanisms easily can be designed and developed between any two systems that share a common related business concept with different business language or business semantics.
Example
All the general discussion can be better understood in the context of a detailed example. The example we’ll use is based on a retail sales system tracking actual sales data against inventory levels. The retailer might have originally created an operational data store to consolidate sales data from both eCommerce and brick-and-mortar stores and compare them to historical sales data, a typical reporting structure for a DSS. After living with the reporting system for a while, the retailer wants to take the next step: using an SEAI to evolve the data from the reporting system needing manual intervention into an automated response.
For this example, the reporting system is providing a sales forecast report typical of a DSS. The report output might contain a series of entries by product. For example:
Event Type=Retail Demand Forecast
- Product ID 010
- Product Price 10.00
- Markdown 10% End Date = 1/1/06
- Forecast Demand 110% of plan
These entries would be compared to a collection of rules organized by the event type, such as:
Event Type=Retail Demand Forecast
- If demand >=110% && demand < 115% then service = adjust pricing information; action = markdown effective date += 10
- Else If demand > 115% then service = adjust pricing information; action = markdown effective date += 20
- Else If demand >= 90% && demand < 95% then service = adjust pricing information; action = markdown effective date += 5
The event document given in the example would trigger rule A, which would result in the effective date of a scheduled markdown being pushed back for 10 days. This is an example of a DSS report used to implement a feedback loop that adjusts the effective dates of planned price mark-downs.
Note: The rules may involve simple calculations with percentages applied to the parameters to arrive at the effective dates. Also, they may take on more complex forms such as invocation of a more robust statistical price-optimization engine service. The rules engine and its functional components are beyond the scope of this discussion. Also note that the format and the specification of the metadata layer (whether it consists of WSDL and a UDDI registry or something else) that provides the list of service definitions and operational end-points for the subscribing systems is also beyond the scope of this discussion.
Implementation of SEAI
The following diagram shows a view of an SEAI implementation.
In this diagram, Message 1 represents an event generated by the analytical tool. If the payload represents a BAM-style event, it can be consumed easily. However, if the payload is a DSS-style document, Message 2 represents a call to the rule interpreter to convert the document into a collection of events (Message 3). For each event in the collection, a call is made to the rules interpreter (Message 4) to see whether the event should be translated into additional service calls. Internally, the rules interpreter also uses the metadata component to identify which systems are subscribing to which events and how the event should be formatted for each transactional system (Message 5). Message 6 represents the zero or more operations going back to the transactional systems.
Depending on the environment, the outbound service calls might be mediated by external adapters, point-to-point interfaces, or by an Enterprise Service Bus (ESB). The process controller in this case is only responsible for message transport and sequencing; all of the metadata and rules interpretation is included within the rules interpreter. A typical realization of this design might use a business process engine for the process controller, a session façade for the rules interpreter invocation, and a strategy pattern behind the façade. The event-specific logic should be configured by metadata, but “one-of” custom components can always be used to meet unique event requirements.
The following class diagram represents one possible realization of an SEAI.
Note that there are certainly other approaches to designing and implementing the type of interpretive system described in this paper. The example above is intended to illustrate the author’s thought on the detailed design of an interpreter and not to provide a canonical pattern for its implementation.
Conclusion
Traditional uses of analytical tools tend to focus on output that will be presented to a human for interpretation. The purpose of the SEAI concept is to provide a mechanism to use the multi-dimensional analysis features of best-of-breed analytical tools to create automated responses in terms of existing system attributes and operations. The SEAI component is designed from the ground-up to be an Event-Driven Architecture (EDA), which makes it easy to use its services in enterprise SOA ecosystems to connect them in a loosely coupled asynchronous manner. Taken as a whole, these concepts allow:
- Extension of existing systems with minimal organic code impact
- Application of existing investments in analytical tools to create advanced self-correcting (“closed loop”) systems without writing complex custom code to perform the data analysis
- Creation of complex system interactions that are driven by configuration and not coding
- Compliance with emerging SOA ecosystems based on ESB products
About the Authors
Surekha Durvasula is an enterprise architect who has had 10 years in
designing and architecting a variety of enterprise applications in the
financial services sector and the retail industry vertical. Most of her
work has been in the area of distributed N-tiered architecture including
J2EE component architecture. Her more recent focus has been on Service
Oriented Architecture and Business Process Management. Her efforts as an
Enterprise Architect involve not only architecting new applications and
business services but also in leveraging the principles of SOA to extend
the life of existing Enterprise Information Systems.
Dave Lindema is an Architect with BEA Professional Services and has over 20 years of total industry experience.
Bob Hensle has more that 19 years of IT industry experience. He has
designed a variety of applications on multiple technologies for many
companies including General Motors, HP, Level 3 Communications, Federal
Express, Galileo International, and MCI. For the past eight years he has
been a consultant with BEA Systems and most recently has been focused on
Service Oriented Architectures built using the BEA WebLogic and
AquaLogic product families.
