This article is the fourth in a series about the Event Web (EW), a thin layer on top of the World Wide Web that continuously monitors information sources and responds appropriately as conditions change. The EW, based on service-oriented architecture (SOA) and event-driven architecture (EDA), can also serve as a platform for Sense and Respond (S&R) systems, a new type of application that continuously responds to critical conditions.
This installment discusses EDA in relation to traditional publish-subscribe (pub-sub) systems.
As described in a previous article, software architectures consist of:
- Components,
- Compositional operators (for plugging components together), and
- Contracts
The contract or specification associated with a component tells systems designers everything they need to know about the component. This article shows that EDA is much more powerful than the traditional pub-sub architecture because of the flexibility and dynamic nature of contracts between subscriber (client) and publisher (server). The goal is to explore the richness of EDA contracts and explain the design consequences of this richness.
EDA Components
As discussed in the previous article, EDA has two components:
- agents and
- streams
Agents come in three types:
- sensors,
- event-processing agents (EPAs), and
- responders
Sensors, such as temperature sensors, monitor the environment and generate messages (events) that describe some aspect of the environment’s state, such as temperature. Event-processing agents receive multiple streams of messages, process them, and generate message streams in turn. Responders, such as the controller of an airplane rudder, receive message streams and modify the state of the environment (by moving the rudder, for instance).
In animals, sensors are eyes and other sensory organs, the nervous system is a network of EPAs, and muscles are responders.
Streams are sequences of messages. EDA has two types of streams, event streams and control streams. A message in an event stream contains information, such as temperature, about the system state. A message in a control stream contains commands to specify, or update, subscriptions that specify the types of event messages that should be sent to the subscriber.
In animals, the sequences of messages from the eyes to the central nervous system are event streams. These messages contain information that the eye sensed about the environment. A sequence of messages that the central nervous system sends to the eye, telling the eye what to watch out for, is a control stream. In a jungle, the eye may be commanded to look out for snake-like objects, while in the ocean the eye may be told to look for jelly fish-like objects.
Control streams between nodes of the network are crucial to the efficacy of an EDA application. Contracts between agents specify the relationship between the event and control streams that flow between them. The flexibility of the relationship between event and control streams makes EDA powerful. The following section explores these contracts in detail.
Relationship Between Event and Control Streams
First, look at contracts between a single subscriber (client) and a single publisher (server). The subscriber sends a message containing a subscription to the publisher; the subscription specifies the conditions under which the publisher will send a message to the subscriber. The subscriber may send subsequent messages updating the subscription.
The stream of messages from the subscriber to the publisher is a control stream. The stream tells the publisher what to look out for. Subscriptions are specified in terms of a formula or model or in terms of examples for use in machine learning. Subscriptions may include service-level agreements (SLAs) that specify, for example, the maximum delay between the occurrence of a situation and the delivery of a message to the subscriber identifying the situation. Clients find services for EDA just as they do for SOA: through brokers.
Models and machine learning techniques can be sophisticated.
Simplistic subscriptions result in publishers generating irrelevant messages. Subscribers then must devote computational cycles to extract the information they need from event streams. Model-based subscriptions allow publishers to carry out more computation—executing models—to ensure that only appropriate messages are generated. This reduces the volume of messages from publisher to subscriber and reduces the computational load on subscribers. Thus, the use of sophisticated models in subscriptions allows for the balancing of computational load across multiple processes and a reduction in communication volume.
In many cases, a subscription is of the form: “Tell me when something unusual happens.” The subscription may specify a model of normal behavior. Any violation of the model is deemed anomalous and results in an event message being sent. Alternatively, the subscriber may specify a formula that directly identifies anomalous behavior.
The two kinds of subscriptions are not fundamentally different. In some cases, specifying what is normal is easier than specifying exceptional situations; in other cases, specifying anomalous situations is more convenient.
The next section presents some examples of the first type of specification: a model specifies normality. Later, you’ll see examples of the second type, in which anomalous behavior is specified directly.
A Model Specifies Normality
Consider a security application in which an organization wants to be alerted if its infrastructure is being attacked. Specifying normal behavior (by providing a large number of examples of normal situations, for instance) is often easier than specifying situations that indicate attacks. A violation of normal behavior may not be the consequence of an attack but nevertheless is a situation that needs to be investigated.
In a commodity-trading application, a model may specify relationships between prices of a commodity at different places, prices of substitutable commodities, costs of shipment between different places, exchange rates, and so forth. A variation from the model may signify a threat or opportunity that is worthy of exploration.
Next, consider examples in which anomalous behavior is specified directly. The previous article presented a simple example of a publisher in the form of a temperature sensor. The subscription includes a forecast—a model—of temperature over the next 24 hours. The sensor sends a message containing the measured temperature when the measured temperature deviates from the forecast temperature by more than a threshold value.
A subscription in a stock-trading example contains a model of value-at-risk of the portfolio and a threshold for this value. The publisher sends an event to the subscriber when this threshold is exceeded. A subscription in an accounting application contains a model—a plan—of expenditures and revenues. The publisher sends an event when actual expenditures or revenues deviate from the plan.
When reality deviates from a model, a subscriber either updates the model by sending control messages to the publisher or simply receives messages notifying it of prescribed types of deviation. In the former kind of subscription, human intelligence (usually) is required to investigate the deviation and determine whether it is significant. An example of the latter kind of subscription would be one message received when revenues are below plan and another message received when expenditures exceed the plan—and the subscriber may invoke a computational process to respond to the message. In all these applications, a subscriber can conclude that the subscriber’s model fits reality until the subscriber receives a message indicating the contrary.
EDA Model in Use
Consider a couple of simple examples that illustrate the ubiquity of the EDA model. Do you know of any organization in which a vice president (VP) tells the CEO about an unusual situation—say a manufacturing plant burns to the ground—only if the CEO asks?
Effective organizations use (usually implicit) contracts that specify what should be communicated and when. The VP is responsible for telling the CEO that a plant has burned; the CEO is not responsible for extracting that information from the VP. Pharmaceutical companies are responsible for telling the FDA when drug trials result in deaths; the FDA is not required to poll the pharmaceuticals to obtain this information. The shared model—the shared expectation—about what is to be communicated is an implicit subscription.
Communication within any organization—say the healthcare system—is based on shared expectations between people and institutions. These shared models, or shared expectations, are critical to the success of the organization. What the pharmaceuticals tell the FDA and when, what the FDA tells doctors and when, what doctors tell patients and when, all depend on shared expectations.
In one type of system, the person who needs information is responsible for acquiring it: the patient is responsible for asking the doctor about drug interactions; the doctor is responsible for checking with the FDA, and the onus is on the FDA to find out facts about drugs from pharmaceuticals. In a different type of system, the organization that obtains information is responsible for propagating the information to organizations that need it (in other words, to subscribers) in a timely fashion: pharmaceuticals tell the FDA, which tells doctors, who tell patients. Thus, the time between the occurrence of a material event and the appropriate response is reduced.
EDA vs. SOA and Traditional Pub-Sub Systems
In SOAs, the person who needs information is responsible for asking the person who has the information: The client invokes a service on a server, and the server has no responsibility other than to respond to service invocations. In EDA, the client who needs information is responsible for updating subscriptions (models, forecasts, and plans) at the server with the information, and the server is responsible for continuously propagating relevant information. Three aspects of EDA contracts make EDA more powerful than SOA:
- The duration of the contract between client and server is long-term rather than a one-time invocation.
- The client has the responsibility to update subscriptions (models) so that the client gets information only to each moment from the publisher.
- The publisher has SLAs to provide relevant (as determined by the subscription), timely information to the subscriber.
Some might argue that EDA and the Event Web are nothing but well-known push technology. These skeptics miss the key point: Traditional push technologies and pub-sub systems exploit only a tiny fraction of the power of EDA for four reasons:
- Sophisticated subscriptions. Traditional technologies use very simple subscriptions such as subscriptions for “news about Iraq,” whereas EDA and the EW use a rich variety of subscription types that include sophisticated model-based subscriptions.
A traditional subscription is a filter on a single event (for example, “Is this news story about Iraq or not?”). By contrast, subscriptions in EDA and the EW can be correlations across histories of multiple event streams. Further, control streams—the continuous updating of subscriptions—are not a feature of traditional push systems.
- Evolving networks. Traditional push architectures have a hub-and-spoke form: Streams are pushed to the hub that does simple filtering on messages and then pushes filtered messages to subscribers. The EW is a network that evolves organically. Just as the Web is a network of Web-hosting servers, with servers invoking services on other servers and then offering value-added services, the EW is a network of Event-Web-hosting servers, with new servers added to the network over time, and with new streams connecting servers. Filtering, aggregation, correlation, and model computation take place throughout the network, not merely at a single hub.
- Structured events. Traditional technologies push events, such as news stories, that are unstructured. Events in EDA and EW, on the other hand, can be highly structured with valuable metadata.
- Ubiquitous standard. Traditional technologies used proprietary protocols to push events to subscribers, whereas the EW is a thin layer on a ubiquitous standard—Web services.
Traditional pub-sub systems exploit only a tiny fraction of the power of EDA because they use only simple subscriptions rather than sophisticated models. An example of a simple subscription is: “Send all messages dealing with ‘Sales’ to a subscriber.” Designing good EDA systems requires creativity because of the many ways in which models can be distributed and shared between agents. Upcoming articles in this series will discuss designs of EDA systems.
