Modellierung und Bewertung von Event-basierten Systemen
Final Report Abstract
The first major contribution of the project was the design and development of a representative workload for evaluating the performance and scalability of event-based middleware. We concentrated on commercial and open-source Message-Oriented Middleware (MOM) platforms based on the Java Message Service (JMS) interface which is the most popular vendor-agnostic interface for MOM platforms. The workload we proposed is based on a novel scenario in the suppy-chain management domain modeling the flow of goods and information in the supply chain of a supermarket company. The workload was used as a basis to build the first industry-standard benchmark for MOM, SPECjms2007, which was released by the Standard Performance Evaluation Corporation (SPEC) at the end of 2007. Producing official benchmark results and using them to compare alternative MOM platforms is just one usage scenario for SPECjms2007. Many users are using the benchmark to tune and optimize their platforms or to analyze the performance of specific MOM features. Others are using the benchmark in academic environments to evaluate the performance and scalability of novel methods and techniques for building high-performance MOM servers. In order to support such usage scenarios, we conducted an in-depth characterization of the benchmark workload and based on this characterization we developed a methodology for performance analysis of MOM platforms. The methodology can be used to define a workload configuration that stresses selected features of the MOM infrastructure in a way that resembles a given target customer workload. SPECjms2007 marked a milestone in benchmark development by providing, in addition to standard workload and metrics, a full-blown performance evaluation framework for in-depth platform analysis. Since its release, the benchmark has been increasing in popularity and has been taken up widely by performance engineers and researchers specializing in MOM performance. The second major contribution of the project was the development of a methodology for workload characterization and performance modeling of distributed event-based systems. As a first step, we developed a workload model based on monitoring data that captures the system routing behavior and resource consumption at a level of abstraction that allows us to use this information as input to performance models. The model does not make any assumptions about the algorithms used at the event routing and overlay network layers of the system, which makes it applicable to a wide range of systems. We proposed a workload characterization technique and based on operational analysis we developed a method to approximate the mean event delivery latency. Our workload characterization technique was applied successfully in the context of a novel event-based middleware for transport information monitoring developed at Cambridge University. For more accurate performance prediction, we proposed a modeling technique based on queueing Petri nets. A case study using the SIENA publish/subscribe system demonstrated the effectiveness and practicality of our approach. The major advantage of the latter lies in its generality. Our long term goal is to exploit predictive performance models generated on-the-fly as a means to implement autonomic resource management. As a further contribution of the project, we did some preliminary work on online performance prediction in the context of enterprise Grid environments. We developed a framework for designing autonomic QoS-aware Grid resource managers that have the capability to predict the performance of the Grid components they manage and allocate resources in such a way that service level agreements are honored. While the approach was validated in the context of a Grid computing scenario, it is not in any way limited to Grid environments and can be readily applied to other distributed systems including event-based systems. In parallel to the above activities, we have been working on a modeling environment for distributed systems called QPME (Queueing Petrinet Modeling Environment). Since its first release in January 2007, QPME has generated a lot of interest in the research community and has been distributed to more than 60 research organizations worldwide. Being a general purpose tool, QPME is used in a number of different areas beyond software performance engineering including computational biology, logistics planning and construction modeling.
Publications
- SPEC Open Systems Group (OSG). SPECjms2007 Run and Reporting Rules
S. Kounev et al.
- SPEC Open Systems Group (OSG). SPECjms2007User's Guide
S. Kounev et al.
- SPEC Open Systems Group (OSG). SPECjms2Q07 Design Document
S. Kounev et al.
- Message Types and Interfaces Between Components in SPECjms. Technical Report, No. DVS06-3, SPEC OSG Java Subcommittee, February 2006
K. Sachs and S. Kounev
- Workload Scenario for SPECjms - Supermarket Supply Chain. Technical Report, No. DVS06-2, SPEC OSG Java Subcommittee, January 2006
S. Kounev and K. Sachs
- Designing a Workload Scenario for Benchmarking Message-Oriented Middleware. In Proceedings of the 2007 SPEC Benchmark Workshop, Austin, Texas, January 21, 2007
K. Sachs, S. Kounev, M. Carter and A. Buchmann
- Workload Characterization of the SPECjms2007 Benchmark. In Formal Methods and Stochastic Models for Performance Evaluation, Proceedings of the 4th European Performance Engineering Workshop (EPEW-2007), Springer LNCS 4748/2007, Berlin, Germany, September 27-28, 2007
K. Sachs, S. Kounev, J. Bacon and A. Buchmann
- A Methodology for Performance Modeling of Distributed Event-Based Systems. In Proceedings of the 11th International Symposium on Object/ Component/Service-oriented Real-time Distributed Computing (ISORC-2008), Orlando, Florida, USA, May 5-7, 2008
S. Kounev, K. Sachs, J. Bacon and A. Buchmann
- Kaffeekunde - SPECjms misst Message-oriented Middleware. In IX Magazin, Heft 02/2008, Heise Zeitschriften Verlag, 2008
S. Kounev and K. Sachs
- SPECjms2007: A Novel Benchmark and Performance Analysis Framework for Message-Oriented Middleware. DEV2DEV Article, O'Reilly Publishing Group, March 2008
S. Kounev and K. Sachs
