They provide the context for the fact tables and hence for all the measurements. Dimension tables are the heart of the data warehouse. Many of these subsystems focus on dimension table processing. Careful and consistent use of these techniques is critical to building a successful dimensional data warehouse that is reliable, scalable and maintainable. There is considerable variation in source data structures and cleaning and conforming logic, but the delivery processing techniques are more defined and disciplined. The primary mission of the ETL system is the handoff of the dimension and fact tables in the delivery step. The five major subsystems in the cleaning and conforming step include: The key is to develop an ETL system capable of correcting, rejecting or loading data as is, and then highlighting, with easy-to-use structures, the modifications, standardizations, rules and assumptions of the underlying cleaning apparatus so the system is self-documenting.
It's essential to strike the proper balance between these conflicting goals. The ETL data cleaning process is often expected to fix dirty data, yet at the same time the data warehouse is expected to provide an accurate picture of the data as it was captured by the organization's production systems (see related article, “Data Stewardship 101: First Step to Quality and Consistency). Such diagnoses can eventually lead to business process reengineering initiatives to address the root causes of dirty data and to improve data quality over time. In addition, these subsystems should be architected to create metadata used to diagnose source-system problems.
The cleaning and conforming subsystems change data and enhance its value to the organization. The other activities, extracting and delivering data, are obviously important, but they simply move and load the data. These critical steps are where the ETL system adds value to the data.
0 kommentar(er)
