This article presents an EAI integration scenario with a case study. The case study presented is a real-time project that was executed for a major European client. The name of the client has been reserved for confidentiality purposes. Seebeyond eGate software was chosen as the integration tool for this project.
The article starts with a brief overview of the fundamentals of the EAI initiative but expects the audience to have basic understanding of the EAI paradigm.
Brief Introduction to EAI
Today’s business world is infinitely more complex than it was a long time ago. Modern companies have a large number of applications that take care of running the business. Such diverse applications weren’t a problem initially because they were meant to provide stand-alone, independent, and automated functions. The result of this diversity was a collection of stovepipe applications rather than a unified network of linked systems. But now, the companies are realizing the utmost need to integrate these independent data silos to leverage the information stored in them across the various vertical and horizontal domains as well as surmount the ever-increasing costs of building new applications.
And this is where an EAI solution comes into the picture.
EAI is a collection of processes, software and hardware tools, methodologies, and technologies. When implemented together, they have the aim of consolidating, connecting, and organizing all the businesses computer applications, data, and business processes (both legacy and new) into a seamlessly interfaced framework of system components that allow real-time exchange, management, and easy reformulation of the company’s mission-critical information and knowledge. It is an unrestricted sharing of data throughout the networked applications or data sources in an enterprise.
When designing an Enterprise Application Integration (EAI) solution, it is important to recognize that there are different levels of integration, each with its own requirements and considerations. Successful implementation of consistent, scalable, reliable, incremental, cost-effective EAI solutions depends on the standards and methodologies that we define for these levels. It must be determined how we need to share information:
- Within an application
- Between applications within an enterprise
- Between enterprises
- Directly with customers
Moreover, the solution should be based upon a multi-tiered, non-redundant, coarse-grained architecture that is enforced across the enterprise. This requires the total set of functions involved in the end-to-end integrated solution be provided by distinct layers, each providing unique and non-overlapping services as shown in Figure 1.
Figure 1: Multi-Tiered EAI Architecture
This architecture provides sufficient high-level consistency for interoperability and a certain degree of local freedom. For example, the architecture supports semantic diversity (different interpretations of the same data) and permits use of diverse technical tools and techniques. It also provides the basis for organizational responsibility and ownership of each layer. For example, business data persists at the application component layer and not in the middleware/EAI layer. Hence, this data is the responsibility of the application owners and not the middleware solution developers.
Business Case Introduction
Royal Wallace, a UK-based transportation company, is a global leader in the rail equipment and servicing industry. Its wide-range of products includes passenger rail vehicles and total transit systems. It also manufactures locomotives, freight cars, bogies, and provides rail-control solutions.
Because of the structure of its business, the Royal Wallace company has a presence in several countries across Europe and in North America, including the USA and Canada. And, each of these country-specific sites has its own dedicated legacy system such as SAP-based systems in UK and Germany, Oracle financials in Canada, and so on (see Figure 2).
Figure 2:As-Is Scenario
As can be seen in Figure 2, in the as-is scenario, these legacy systems are acting as independent data silos with absolutely no kind of data-sharing happening between them. If the German SAP system wants to procure something from any of its vendors, it does so by using some kind of manual process at its end. The same scenario is replicated for all the other systems, too. And, this is leading to a lot of data duplication taking place. For example, if two different systems are using the same vendor (say, Vendor A) for procuring white board markers, this vendor (Vendor A) information is getting stored at both the legacy systems. Also, this kind of scenario makes it impossible to leverage existing information across multiple systems.
Royal Wallace understands that, to survive the competition in today’s fast paced economy and to reduce its time-to-market, it needs to put an integration solution in place as soon as possible. The integration model should integrate all the existing legacy systems irrespective of the platforms and operating systems that these systems are based upon. The model should take care of the various data formats that would pass through this integration scenario and apply the necessary transformation, translation, and data validation rules upon them. And last, but definitely not the least, this integration solution should be scalable enough such that any new system could be easily plugged into this solution in the future with minimal changes.
The integration model should provide:
- Integration between the various legacy systems
- Automated process for procurement of goods
- Data transformation and translation logic according to the prescribed rules
- Persistent storage mechanism for the data
- Validation of business data to a certain extent
- Connectivity to various drop zones and databases
Project Life Cycle
1. Design and Architecture
As seen in Figure 3, the EAI solution proposed would make use of the integration capabilities of the Seebeyond eGate software to seamlessly integrate the various individual legacy systems of the Royal Wallace company with the SAP-based purchasing order (PO) system.
SeeBeyond eGate Integrator is a fully J2EE certified and Web services-based distributed integration platform. It provides the core integration platform, comprehensive systems connectivity, guaranteed messaging, and robust transformation capabilities.
Figure 3: To-Be Scenario
In this scenario, the SAP-based PO system acts as a data hub for all the legacy systems such that all the procurement orders at Royal Wallace would be routed through this SAP based PO system. Also, Seebeyond is acting as the integration hub between the SAP-based PO system on one end and the legacy systems at the Royal Wallace company on the other, thereby enabling a bi-directional flow of data.
Process Flow
Figure 4 gives a comprehensive picture of the process-flow through the integration model.
Figure 4: Process Flow
Whenever a Royal Wallace employee needs to procure something, she logs on to an intranet application. The workflow in this application is managed by the SAP-based purchasing order (PO) system. The PO system, in turn, places the order with the vendor on behalf of the employee. The vendor acknowledges the receipt of the order to the PO system and delivers the goods to the concerned legacy system. Once the goods are delivered, the vendor sends an invoice for the same to the SAP PO system. The PO system, in turn, sends this invoice to the appropriate legacy system which then makes the necessary payment to the vendor. The legacy system also sends back a ‘payment done’ kind of acknowledge back to the PO system. This scenario is replicated for all the other legacy systems, too.
The EAI solution (Seebeyond eGate) is responsible for all the communication between the SAP PO system and the legacy systems. Various interfaces were developed as part of this solution to enable bi-directional flow of data. All the information pertaining to the various legacy systems and the SAP-based PO system (source and target) were captured as part of the functional specifications. These specifications covered the following topics:
- The platforms, databases, and operating systems for the source and target
- The various applications constituting these systems and the way they interacted with each other
- The input and output data formats for the source and target (EBCDIC, ASCII, Binary, and so forth)
- Frequency of data transfer and outage periods
- System connectivity details such as FTP drop zones
- The file structures that were being processed, if applicable
The technical specifications covered the following topics:
- Detailed technical design and architecture of the project
- All database-related details including the various servers, IP details, and the like
- Detailed configuration details of the various Seebeyond integration components, including connectors
- Detailed description of the error flow, including the codes
- Detailed description of the logging mechanism
- Input and output data structures and the transformations involved, if any
- Pseudo-code
Common Data Format (CDF)
The various systems at Royal Wallace company that were to be incorporated into the integration model were based on different platforms. And, the applications that were running on these systems used to exchange data in various different formats with the outside world. Most of these were proprietary formats that each of these applications had created as per their own requirements over the years; there was no single standard approach that had been followed. Some of them had fixed-length records, others were delimited; some followed ASCII character encoding, others were EBCDIC; and so on. And above all, as most of these formats were in existence over the past several years, the system owners were very reluctant to replace them for a common standard format/template.
So one of the major challenges was to come up with a solution that could somehow transform and translate these multiple data formats passing through the integration model, thereby enabling the various systems to communicate with the SAP-based PO system. Moreover, the solution had to be scalable enough to incorporate any new data formats in the future with minimal changes to the existing scenario.
The solution was to make use of the Common Data Format (CDF), as shown in Figure 5.
Figure 5: Common Data Format
The Common Data Format (CDF) was an event structure that was generated within the Seebeyond eGate environment. Because the various applications deployed on the legacy systems used the integration model to procure goods from vendors, the incoming data to the EAI environment from the various applications had some common data elements irrespective of the data formats that these applications used. For example, all the procurement requests had information about type and quantity of the goods to be procured. All these common elements were captured under multiple nodes in the CDF event structure in Seebeyond. This proved useful in several ways:
- The entire data structure for the whole integration model was defined using a single event definition in Seebeyond. Now, all the incoming data into EAI could be parsed and then mapped to the CDF event structure.
- Once the data was available in the CDF, it was very easy to create the event structure as per the target system requirement and then simply map the CDF data into that structure.
2. Project Build, Test, and Deployment
After an elaborate design an architecture phase, the project entered the build phase. It is during this period that the various integration components were developed by making use of the Seebeyond eGate software.
As seen in Figure 6, three different environments were used for this purpose.
Figure 6: Different Environments for Various Phases of the Project
Environment set-up
- Development Environment: The entire integration model in Seebeyond was developed here. This included the various adapters to connect to external applications, JMS-based messaging queues for persistent storage of data, configuring various connectors to connect to FTP drop zones and databases, and so forth. Once the development was completed, unit testing of the individual components was also performed on the development environment itself.
- Test Environment: Once the unit testing phase was over, the project model was migrated from the development environment to the test environment. The integration testing of the project model was performed in this environment. The test environment is a replica of the real production environment. Exhaustive testing of the model was performed here, along with the load testing wherein real-time data was fed to the model and the actual results that were received were captured against the expected results. In case of any issues, the model was moved back to the development environment, fixed for any outstanding issues, unit tested, and then moved back to the test environment. This whole cycle was repeated until the desired performance was reached and expected results achieved.
- Production Environment: It is this environment where the run-time instance of the model is deployed after rigorous testing is done and user acceptance achieved.
Seebeyond Transaction Flow Diagram
Figure 7 below shows the various Seebeyond components that were developed to enable the flow of data from the source to the target. Please note that because there is bi-directional flow of data between the SAP-based PO system and the legacy systems at the Royal Wallace company, the keywords ‘source’ and ‘target’ are used interchangeably for both ends.
Figure 7: Transaction Flow Diagram
The input files to the integration model were made available on the multiple FTP servers by the legacy systems at Royal Wallace as well as the SAP-based PO system. After the data in the file had been processed by Seebeyond, an output file was generated and transferred to the particular FTP location corresponding to each legacy system. Figure 7 shows the transaction flow for one such integration scenario.
Once the input file becomes available in the designated directory on the FTP server (source), it is picked up by eWay 1 (Seebeyond adapter) which is constantly pooling this directory. This eWay also performs any necessary validation checks on the structure of the file as well as the data content within it. It then publishes the data into JMS based IQ 1 (intelligent queue). eWay 2, which subscribes to JMS IQ 1, receives this data. It is in this eWay that the transformation of the data from the application-specific format to the independent Common Data Format (CDF) takes place.
Furthermore, any database-related logic that is required by the integration model is taken care of in this eWay. eWay 2 then publishes the CDF data event to the JMS IQ 2. eWay 3, subscribing to JMS IQ 2, receives this data. It takes care of transforming the data from the independent CDF format to the target application-specific message format. A few minor business validations in the input data are also done in this eWay on behalf of the target application. eWay 3 then publishes the data to the designated directory on the target FTP server in the form of a file. All the common services, such as exception handling and logging requirements, are provided by a separate schema that subscribes to a JMS-based IQ from this integration model.
The transaction flow explained above is applicable to all the interfaces that were designed as part of the integration model to enable bi-directional communication between the SAP based PO system and the legacy systems of the Royal Wallace company.
Conclusion
The EAI solution implemented at Royal Wallace addressed all the integration requirements of the company. It provided for an automated procurement process in place of a manual one and seamlessly integrated all the legacy applications at the company with minimum fuss. The changes required in the existing scenario were kept to a minimum. Moreover, because the Common Data Format (CDF) provides a lot of flexibility to the model, any new application can be easily incorporated with a plug-and-play kind of adaptability.
References
White Papers are posted at the following locations:
Useful Links
About the Author
Tarun Gupta is an EAI Consultant at Covansys India Pvt. Ltd. He is a post graduate from the Indian Institute of Information Technology, Bangalore.
Tarun has more than five years of experience in the integration domain. He has used various tools such as WebMethods, Seebeyond, Vitria, and TIBCO to provide solutions in manufacturing and telecomm domains. He can be contacted at tarun87@hotmail.com.
