Organizing Data: Organizing Data for Access and Storage

Name: Organizing Data: Organizing Data for Access and Storage
Author: Bernie Kuan

Learn the formats in which data can be stored and where and how data can be stored.

Preview the first lesson free — get full access to all 3 lessons.

Course: On-Demand

As organizations collect and organize data for specific business objectives, that data can be stored and maintained in a variety of ways. In this lesson, we'll explore the various structures and formats in which that data can be stored. A large variety of data formats are available to organize data based on the original data collection method, the data content, and the purpose for the data. Generally, these data formats can be categorized into structured, semi-structured, and unstructured data. As its name implies, structured data is stored in a very specific, predefined format, where each data point has a predetermined place in a row and column paradigm. Structured data can be found in any relational database, like an Excel file. Within an Excel file, data values are clearly allocated to specific rows and columns. Let's look at collected data about the customer John, aged 27, whose address is 123 Main Street. In a structured data format, the data will be clearly allocated to a table where each column is a data point, and each row is a customer data record. Semi-structured data formats, on the other hand, are more versatile than structured data formats. Semi-structured data does not have to obey a specific model where data points must map directly to table columns and rows. However, the format still has some structure in that it contains varying tags or markers to separate individual data points into logical relationships. Furthermore, if we know the relationship governing those logical relationships, we can convert semi-structured data into structured data. Let's look at data in the comma-separated values format, or CSV, which can be stored in a free text file. In a CSV file format, several lines of data points are separated by commas, with each row representing a set of data points. To clarify, let's look at collected data where the customer is John, his address is 123 Main Street, and his age is 27. Here's what storing this information into a CSV structure would look like. The first line of comma-separated values represents the name of the data point, while the subsequent lines represent the values for an individual record. Because data in the CSV format is stored as text and can be freely manipulated, it is not strictly structured data. However, it is easy to see how a format structure such as this can easily translate into the common table representation of data. Each value, separated by commas, represents a table column, and each line of data represents a table row. Another common format is tab-separated value, or TSV. This format is similar to the CSV format, except tabs separate values instead of commas. In fact, it's feasible to create structured data formats with any specific value acting as the delimiter, which distinguishes one data point from another. Similarly, while a set of delimited data values can be distinguished by a new line, a user can also specify a different way to indicate the end of the data record. Next, let's look at another semi-structured data example that offers even more versatility, the JSON format. The JSON format uses a colon to indicate data relationships, and it can hold data in a hierarchical form with each level of the hierarchy enclosed by brackets. Our collected data for a customer named John, age 27, whose address is 123 Main Street, California, might look something like this. The first enclosed bracket indicates the data for customers. The next layer of brackets indicate that a set of customer data includes the data points for name, address, and age. We also see that the data point for address includes a set of data points for street and state. As you can see, semi-structured data, through the use of nested hierarchical data sets, provides much more flexibility in the variety of data that we can include in one place. In addition, a JSON format does not require a value for every data element. For example, another customer record could simply be a name without information about address or age. This would be allowed in a semi-structured design, but a structured data format would require a value for address and age, even if it's an empty placeholder. Just like the CSV file format, it's possible to map data from this JSON format into a structured table, provided we define and enforce the data relationships into a column and row orientation. For example, we may map the name of the data points from the JSON file into a tabular structure like this. There are many ways to represent semi-structured data, and the JSON format is just one of them. The last data format we'll discuss is unstructured data, which is data that does not have a predefined structure. In this format, the data points provided in the data don't necessarily have a relationship to one another. Unstructured data is typically text-heavy and requires a significant amount of work to convert into a structured format. Examples of unstructured data include text and PDF formats with mostly freeform text, as well as media formats such as photos and video. With such information, there is no predefined or obvious tagging of data points that can be organized into clear rows and columns.

Beginner Provider : Bernie Kuan 3 Lessons · 15m in English

Course Description

Before data can be leveraged for analytical use, it first needs to be stored and accessible. Just as there are many ways to collect data, you have multiple options for storing data. A variety of different data formats have been created based on the types of collected data and their intended purposes. In data analytics, the most common data formats are structured and semi-structured, which present data elements in a state that’s most ready for analysis. Besides a choice of data formats, organizations have a choice of data repositories as well: a ‘data warehouse’ or a ‘data lake’, each with its advantages and disadvantages. Organizations also can choose where the data will live: on-premise or in the cloud.

In these lessons, we’ll discuss the various formats in which data can be stored. We’ll also compare data warehouses and data lakes, two main types of data repositories. And we’ll look at the difference between on-premise and cloud data storage.

What You'll Learn

Identify commonly used data formats, including structured and semi-structured, and how they differ
Explain how data is stored and made accessible for analytical use
Compare data warehouses and data lakes as the two main types of data repositories
Distinguish between on-premise and cloud data storage options
Evaluate the advantages and disadvantages of different data storage choices

Key Takeaways

Before data can be leveraged for analytical use, it first needs to be stored and made accessible.
A variety of data formats exist based on the types of collected data and their intended purposes, with structured and semi-structured being the most common in data analytics.
Structured and semi-structured formats present data elements in a state that is most ready for analysis.
Organizations can choose between two main data repositories, a data warehouse or a data lake, each with its own advantages and disadvantages.
Organizations can also choose where data will live, either on-premise or in the cloud.

Frequently Asked Questions

What does this course cover?

It covers the various formats in which data can be stored, compares data warehouses and data lakes as the two main types of data repositories, and looks at the difference between on-premise and cloud data storage.

Which data formats does the course focus on?

It discusses commonly used data formats and how they differ, noting that in data analytics the most common are structured and semi-structured, which present data elements in a state most ready for analysis.

What lessons are included?

The course includes three lessons: Data Formats for Processing; Data Warehouse vs. Data Lake; and On-Premise vs. Cloud Data.

What skills will I gain from this course?

You will build skills in data classification, data management, data storage, data warehousing, data warehouse architectures, and structured storage.

Does the course cover where data is stored?

Yes. It explains that organizations can choose where data will live, on-premise or in the cloud, and looks at the difference between on-premise and cloud data storage.