Data Visualization and Mapping

Name: Data Visualization and Mapping
Author: Bill Hood

Learn to get your own data by data scraping, using RESTful APIs, and exploring geocentric data.

Preview the first lesson free — get full access to all 3 lessons.

Course: On-Demand

In this lesson, we're going to introduce. A new tool called Beautiful Soup. And what Beautiful Soup does is allow you to interrogate websites and extract data from websites. We'll start with Google.com and if you look at our code, we're going to import requests which allows you to just do straight URLs and your calls to URLs. And then we're going to import Beautiful. Soup very particular format from BS four, import Beautiful Soup, we are going to. Point BeautifulSoup towards Google.com and then we're going to print off that URL. And that's obviously going to be just this string right here. This headers argument is needed and is particular to your browser operating system combination. I run Mac and Chrome and so. This is the appropriate one for me. If you run Windows and Chrome, you. Would need a different header. All right, so you can dig that out simply by googling it and finding the string you need. So this is the string for me. We are then going to create a variable called Page. We're going to use this simple requests import that we brought in and we're. Going to go get that URL using these headers and then print out what it finds. So let's run that and we'll see. That this is just printing out Google. That tells us what we're going to go search. That could be Yahoo, that could be anything. And then response 200. For those of you not familiar with what that means, that's a website saying, copy that, I've got a response. What would you like to know? So this is the website basically responding that, yes, hello, I'm here. I'm there. Thank you very much. All right, now let's go ahead and. Introduce Beautiful Soup into the mix by. Simply creating a variable called Soup, pointing. It to the Beautiful Soup library and our page content from our recent request. So we've got the Page, the Google. Page in this variable right here, and. That is considered content in Beautiful Soup. And we're going to use our HTML parser. And you can see here quite a. Bit of data because it's returned basically. That entire web page. This is basically what Google looks like behind the scenes. And as you can see, it's quite. A lot of data. And what Beautiful Soup does is parse. That out now in a way that. We can play with and look for things and dig data out of this. Because this basically is returning Google.com, which. Allows us to then interrogate that, looking. For things such as link. And I think as we all know, if you've been around tech for a. While, it's considered an A link. And the reason is because it's designated with an A. So now we're going to create a variable called links. We're going to use Soup to find. All that, have the A in there. And let's run that. Now we assign the links. Now let's print those out. And here's all the links that it found and you can see it's considerably less data. It's only this one page. They all got this A in here. And they are all links. That's very cool. All right we can also just say. Find what find would do so that. Found all the links. This would find the first link. Let's run that. And there's our one link, the first one it found. And so what else can we do? So let's go down here real quick. And now notice that we basically just. Used this beautiful soup library to pull. All the content from that giant web page in. And now we can kind of slice and dice it and we can start off with just referencing what is the title and we can run that and. Find out that it's Google with the capital G. And we can see that, we understand that, we know that's true. That's really nice. We point it towards yahoo? We're going to get a different answer. Let's do see what the title is. And I think also makes sense because. It'S title and that's referring to this piece right here. We ask this what is this name right here? What's the container for it and it's title. We can also run and find out. What the string is associated with this and it's Google. This would also be links as we. Discovered a little earlier. And it's going to find the first one. And then here's Soup find all which. Would find all the links which really. Takes us full circle back to kind of where we started with soup and what we can do with soup. So let's move on and let's extract some data into a python list. We're going to start here and just. Recall that we're going to use the. Beautiful soup that we learned in the first part of this lesson. We're going to point to a new URL, whitehouse gov briefing room. And this is basically press releases from the White House. And what we can do here is we've got the same headers argument. All right, we've got to do that because I'm running Mac with Chrome. The page then becomes the combination of get to that URL using the headers. Appropriate to my environment and then we would print that out. You'll also recall that we can create a soup using our beautiful soup library. And that page content and then we. Can do things like find all the links. So that gets us caught up. Now what's different here is that we. Create a new python list. Remember lists are my favorite things but we're going to create a blank list. Called URLs and another blank list called links. And what we're going to do is. We'Re going to go through everything that soup finds. We're going to look for a keyword, the name of our president in this case Biden. And if we see his name in. That link, then we're going to get. The text of that link and we're going to get the Href of that link and we're going to put it in those lists. And then what this would allow us to do is to basically interrogate a website. And it could be any website we're. Using the White House. So we're going to look for these and then run them and see what we get back. In this case, you'll notice here we. Said print the links and so here's our links. And then here's the associated URL. And so you can see that what we've got here is for example, the. Actual URL for that one is about like right there. If we were to copy this out and go type it into a new browser tab, we'd be able to go to that link. All right, so that's Beautiful soup and a quick introduction to what you can do with Beautiful soup. And next we're going to for data scraping using an API.

Intermediate Provider : Bill Hood 3 Lessons · 17m in Arabic, German, English, Spanish, French, Portuguese, Chinese

Course Description

A reality of most data visualization projects is that you will usually face the task of obtaining or otherwise creating some of the data you need. In these lessons, we focus on two of the most common automated techniques for obtaining data and emphasize how you can always type in the data you need. You’ll learn that a website is all HTML, with many tags that can be used to identify and grab data. In our example, we will use the Beautiful Soup library to interrogate a website looking for a keyword and then pull all the web links associated with that keyword, making the links available programmatically.

Next, we will use a RESTful API to obtain data from an increasingly common approach to data exchange. Representational State Transfer, or REST, is an application program interface that rides on the internet and allows programmatic access to an informational repository. We’ll also explore several APIs and extract data into a model we can plot. Finally, we’ll combine several best practices in one exercise and explore data from a Pandas data frame with longitude, latitude, and magnitude of events that we can plot on a world map.

What You'll Learn

Obtain data by data scraping from websites using HTML tags
Use the Beautiful Soup library and the requests library to find a keyword and pull associated web links programmatically
Obtain data through RESTful APIs to access an informational repository over the internet
Extract data from several APIs into a model you can plot
Plot longitude, latitude, and magnitude data from a Pandas data frame onto a world map
Recognize geocentric data

Key Takeaways

Most data visualization projects require obtaining or creating some of the data yourself, and data can always be typed in manually when needed.
A website is all HTML, with many tags that can be used to identify and grab data.
Beautiful Soup can interrogate a website for a keyword and make the associated web links available programmatically.
REST (Representational State Transfer) is an application program interface that rides on the internet and allows programmatic access to an informational repository.
Data from a Pandas data frame containing longitude, latitude, and magnitude of events can be plotted on a world map.

Frequently Asked Questions

What data-gathering techniques does this course cover?

It focuses on two common automated techniques for obtaining data: data scraping from websites using the Beautiful Soup library, and obtaining data through RESTful APIs. It also emphasizes that you can always type in the data you need.

What libraries and tools will I work with?

You will use the Beautiful Soup library and the requests library to scrape websites, RESTful APIs to obtain data, and a Pandas data frame to plot data on a world map.

What will I be able to do by the end of the course?

You will be able to obtain data by scraping and by RESTful APIs, understand Beautiful Soup and the requests library, recognize geocentric data, and plot longitude, latitude, and magnitude of events on a world map.

What lessons are included in this course?

The course includes three lessons: Data Scraping Using the Web, Data Scraping Using an API, and Plotting Data to Geography Maps.

What skills does this course help build?

It builds skills in data mapping, data visualization, datamaps, information visualization, scientific visualization, and visualization.