Organizing Data: Organizing Data in Data Management

Name: Organizing Data: Organizing Data in Data Management
Author: Bernie Kuan

Learn the concepts of data modeling and master data management.

Preview the first lesson free — get full access to all 3 lessons.

Course: On-Demand

In this lesson, we'll explore how data should be presented to data consumers and how building out a semantic layer is critical for organizing data for enterprises. In the world of data analytics, making collected data easily findable and accessible for analysts plays a huge role in how downstream data consumers can make use of that data. And because there are typically many analytic use cases that can be derived from the same set of fundamental data sets, like customer and sales transactions, data organizations also try to ensure that organized data for analytical use are interoperable and reusable. At the end of the day, an organization's overall objective is to ensure that there is a process in place for collected data to be consumed as quickly and accurately as possible. In order for that process to be reliable and scalable, collected data sets from various data sources need to be efficiently organized and managed. Whether in a large or small organization, the raw data from collected data sets are typically not immediately ready for use in analysis as is. Depending on the analytic objective and even the individual preferences of the analysts or developer, the language and level of detail of the raw collected data set could be insufficient. Data manipulation and data transformations will always be required to ensure that the raw data is ready for downstream analysis. Therefore, when organizing data, we need to give thought toward the semantics and naming conventions of how data is presented to data consumers. When an organization acquires and stores data, the data is not always going to be in a form that is presentable to the average business user. More often than not, additional work needs to be done to manipulate, transform and calculate the data to adequately present it in a form that's recognizable and usable to the business. A semantic layer is a business representation of the data using terminologies and naming conventions that the average data consumer can understand. This makes the data more accessible for individual users without the need for an interpreter or additional guidance. The difference in naming between the underlying data source and the business representation can be very similar and intuitive. For example, the underlying data may have the data point for customer name, written as CUST underscore name, or it could be less intuitive, such as mapping F5 underscore REV to sales amount. In larger organizations, a semantic layer needs to be properly implemented and governed in order to be effective. If collected data always has to be manipulated in order to be useful to business users, not only is there a lot of redundant effort involved, there's also the issue of consistency. Each analyst might implement data manipulation differently to curate a dataset fit for their specific analysis. The curated data produced may not be reusable by another person because it is presented in an unexpected format, lacks the information they need, or presents the data in a way that is confusing to someone else. Let's use, for example, a simple raw dataset of customer purchase transaction records in a table. The dataset may have the data columns for, customer name, purchased item description, purchase quantity, purchase price, purchase discount amount, and total sales amount. First, let's look at the potential issues related to naming standards. A data analyst might decide that they prefer all the names of the data points to be shorter and rename the columns to name, description, QTY, PRCE, and total. Another data analyst though, may insist on the naming conventions, CUST, underscore name, item, underscore DESC, quantity, price, and total sales. While these two individuals might be happy with their own presentation of the data, if we were a third analyst trying to build some analytics of our own, we might face a difficult decision of whether to build upon one of the given conventions or to create our own. Creating our own naming convention and calculating directly from the raw collected data source would require extra work on our part, and the need to manage that derivation if anything changes. However, choosing to build our analysis based on the curated data from one of the other two analysts is potentially risky, because their naming conventions could change at any time. We should also note that the two analysts actually left out the data point for the discount amount in their curated data set. Now there's a discrepancy across the data for quantity, price, and total sales. They don't add up. If we were to calculate total company revenue, do we create a new total sales data point by multiplying quantity and price? Or do we use the provided total sales data without really knowing why there is a discrepancy? Having to guess how to interpret such key business metrics is something that could have disastrous consequences. The solution to this problem is to have the organization enforce data standards and governance for the business semantic layer. While not all organizations can have large teams dedicated to creating and implementing data governance, all organizations need to ensure that the data sources analysts are calculating against our accurate, which means that there cannot be inconsistencies and interpretations. The organization would have to decide how all data should be presented to users consistently no matter how the raw source has organized it. For example, let's use the naming conversion for customer full name. Different data sources may have customer names presented in various ways, such as name, CUST underscore name, or only the separation of first name and last name. In this case, the data organization would need to perform data transformations so that the data values for name and CUST underscore name map to customer full name. The values from first name and last name would also need to be combined into one value and then mapped to customer full name. This is to ensure that all end users are presented with the same semantic view. This may sound straightforward when we're dealing with small amounts of data and a limited number of sources. However, creating and maintaining a consistent semantic view for the business becomes more important and complex as an organization increases the variety and volume of their data sets.

Beginner Provider : Bernie Kuan 3 Lessons · 19m in English

Course Description

Organizing data for one person’s consumption is not the same process as organizing data for an enterprise, which has many different data consumers, each with their own skill set, perspectives, and objectives. While data preparation activities are needed at both an individual and an enterprise level, organizations also need to invest in data management to properly take advantage of their collected data. Implementing and enforcing data standards, for example, is critical to ensure that all data consumers use the same data sources and the same business interpretation of the data. Ensure that the analytic outcomes produced by various users are consistent and based on the same data, otherwise confidence in analytic results can be impacted. Moreover, when data is constantly being accessed by many data consumers for different purposes, effective organizations aim to provide accurate, high-quality data to end-users.

In these lessons, we’ll discuss the importance of building out a data semantic layer. We’ll also explore the concepts of data modeling and master data management, the reconciling of disparate data into a single, comprehensive version of a company’s data.

What You'll Learn

Understand how organizations standardize their data so all data consumers use the same sources and business interpretation
Explore data semantic presentation and the importance of building out a data semantic layer
Learn various ways to logically model data through data modeling
Apply master data management to reconcile data from disparate sources into a single, comprehensive version of a company's data
Recognize how enterprise-level data organization differs from organizing data for one person's consumption

Key Takeaways

Organizing data for an enterprise with many data consumers is not the same process as organizing data for one person's consumption.
Implementing and enforcing data standards is critical so all data consumers use the same data sources and the same business interpretation of the data.
When analytic outcomes are not based on the same data, confidence in analytic results can be impacted.
Effective organizations aim to provide accurate, high-quality data to end-users who access data for different purposes.
Master data management reconciles disparate data into a single, comprehensive version of a company's data.

Frequently Asked Questions

What does this course cover?

This course covers organizing data in data management, including building out a data semantic layer, data modeling, and master data management for reconciling disparate data into a single, comprehensive version of a company's data. Its lessons are Data Semantic Presentation, Data Modeling, and Data Mastering.

What skills will I gain from this course?

The course develops skills in Data Architecture, Data Layers, Data Management, Data Modeling, Master Data Management, and Product Data Management.

Why does data standardization matter according to this course?

Implementing and enforcing data standards ensures all data consumers use the same data sources and the same business interpretation of the data, helping keep analytic outcomes consistent so confidence in analytic results is not impacted.

How does the course address combining data from different sources?

It explores master data management, the reconciling of disparate data into a single, comprehensive version of a company's data.