An event-based near real-time data integration architecture
| aut.researcher | Naeem, Muhammad | |
| dc.contributor.author | Naeem, M | |
| dc.contributor.author | Dobbie, G | |
| dc.contributor.author | Weber, G | |
| dc.contributor.editor | Almeida, JPA | |
| dc.date.accessioned | 2012-04-26T02:00:04Z | |
| dc.date.available | 2012-04-26T02:00:04Z | |
| dc.date.copyright | 2008 | |
| dc.date.issued | 2008 | |
| dc.description.abstract | Extract-Transform-Load (ETL) tools feed data from operational databases into data warehouses. Traditionally, these ETL tools use batch processing and operate offline at regular time intervals, for example on a nightly or weekly basis. Naturally, users prefer to have up-to-date data to make their decisions, therefore there is a demand for real-time ETL tools. In this paper we investigate an event-based near real-time ETL layer for transferring and transforming data from the operational database to the data warehouse. One of our main concerns in this paper is master data management in the ETL layer. We present the architecture of a novel, general purpose, event-driven, and near real-time ETL layer that uses a Database Queue (DBQ), works on a push technology principle and directly supports content enrichment. We also observe that the system architecture is consistent with the information architecture of a classical Online Transaction Processing (OLTP) application, allowing us to distinguish between different kinds of data to increase the clarity of the design. Keywords: event-based architecture, content enrichment, master data, extract-transform-load, enterprise service bus. | |
| dc.identifier.citation | 12th Enterprise Distributed Object Computing Conference Workshops, Germany, pages 401 - 404 | |
| dc.identifier.doi | 10.1109/EDOCW.2008.14 | |
| dc.identifier.uri | https://hdl.handle.net/10292/4056 | |
| dc.publisher | IEEE Computer Society | |
| dc.rights | Copyright © 2009 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. | |
| dc.rights.accessrights | OpenAccess | |
| dc.title | An event-based near real-time data integration architecture | |
| dc.type | Conference Contribution | |
| pubs.organisational-data | /AUT | |
| pubs.organisational-data | /AUT/Design & Creative Technologies | |
| pubs.organisational-data | /AUT/Design & Creative Technologies/School of Computing & Mathematical Science |