Python: Evaluating Your Model

Name: Python: Evaluating Your Model
Author: Briana Brownell

Learn to evaluate data models

Preview the first lesson free — get full access to all 9 lessons.

Course: On-Demand

Hi, I'm Brianna Brownell and in these lessons we'll cover methods of model evaluation including overfitting and underfitting as well as explainability, bias and fairness and ethical considerations. In this lesson we're going to cover the basics of how to evaluate the performance of a machine learning model. There are a number of important metrics that we need to consider when doing a thorough evaluation of a machine learning model. In this case we're specifically looking at prediction models where we have data that can indicate whether our predictions were correct or not. Now it's tempting to use a simple method for evaluation by just looking at the overall accuracy of the model. We can define accuracy as the number of true positives and true negatives out of all of the data whose outcomes were predicted. So basically anytime the model makes a correct prediction we count that as an indicator for accuracy. The issue though is that this considers both true positives and true negatives to be equally good and false positives and false negatives as equally bad. The challenge is that sometimes the accuracy may be extremely high but in fact our model is performing poorly. We also might want to look at some of the other measures because we may have a case such as a very unbalanced data set where our overall accuracy might be extremely good but our model is actually performing poorly. So imagine you have a data set of examples of fraud and since fraud is so rare or at least we hope it's rare we may have something like 99% accuracy but the issue might be that we're actually just guessing that none of the cases are fraudulent and not identifying any of the real cases of fraud. For that reason we want to consider a few other scores that we can use to evaluate our model. The first one is precision. The precision is the number of true positives out of the true positives and false positives. So the higher the precision the better job we've done to eliminate false positives. So this is the measure that we want to do in the cases where we want to optimize for removing the false positives. The recall is the opposite. This measure takes the true positives over the true positives and false negatives. And so the recall attempts to minimize the false negatives. So you can imagine there are very different kinds of use cases where minimizing either false positives or false negatives would be important. For example in the case of fraud we would rather identify a case that might be fraud but is most likely legitimate and use a second layer of analysis to determine whether the transaction is actually fraudulent. We also have an additional score called the F1 score and this is a measure that combines the precision and recall in such a way that we can get a balance for false positives and false negatives. Because of the potential for having an unbalanced data set doing the strict accuracy doesn't give this advantage. So as you can see when considering how to evaluate a machine learning model used for prediction there are a number of measures that we can use to evaluate our model. The specific measures that we use will depend on the use case and the larger system that is put around it. However calculating these scores are generally standard so that we can get a better idea of how the model is performing. In the next lesson we'll talk about overfitting which is a common challenge when building machine learning models. Thanks for watching.

Beginner Provider : Briana Brownell 9 Lessons · 34m in English

Course Description

Get to know the essential aspects of evaluating machine learning models. Learn how to evaluate the performance of machine learning models using techniques such as accuracy, precision, and recall. This will help you to understand how well your model is performing and identify areas for improvement.

The course also covers overfitting and underfitting, which are common issues that can occur when building machine learning models. You will learn how to identify and prevent overfitting, which occurs when a model is too complex and performs poorly on unseen data, and underfitting, which occurs when a model is too simple and does not capture the complexity of the data. Additionally, the course covers explainability, which is the ability to explain the predictions of machine learning models to stakeholders. You will learn about techniques such as feature importance, which can be used to make the predictions of a model more transparent and understandable.

The module also covers bias and fairness in machine learning models and how to identify and mitigate bias in machine learning models. Furthermore, you will learn about deployment and maintenance of machine learning models, including how to deploy machine learning models into production and monitor their performance over time. You will also learn about model drift, which is the phenomenon of a model's performance degrading over time as the underlying distribution of the data changes. Additionally, you will learn about interpreting and communicating results, including using techniques such as data visualization and storytelling. You will also learn about ethical considerations in the use of machine learning models, including issues such as privacy, transparency, and accountability.

What You'll Learn

Evaluate machine learning model performance using accuracy, precision, and recall
Identify and prevent overfitting and underfitting
Improve model transparency and explainability through techniques such as feature importance
Recognize and mitigate bias to support fairness in machine learning models
Deploy and maintain machine learning models in production and monitor for model drift
Interpret and communicate results using data visualization and storytelling

Key Takeaways

Model performance can be evaluated with techniques such as accuracy, precision, and recall to gauge how well a model performs and find areas for improvement.
Overfitting occurs when a model is too complex and performs poorly on unseen data, while underfitting occurs when a model is too simple and does not capture the complexity of the data.
Explainability techniques such as feature importance make a model's predictions more transparent and understandable to stakeholders.
Model drift is the phenomenon of a model's performance degrading over time as the underlying distribution of the data changes.
Ethical use of machine learning models involves considerations such as privacy, transparency, and accountability.

Frequently Asked Questions

What does this course cover?

It covers evaluating machine learning model performance using accuracy, precision, and recall, along with overfitting and underfitting, explainability, bias and fairness, deployment and maintenance, model drift, interpreting and communicating results, and ethical considerations.

What will I be able to do after taking this course?

You will be able to evaluate machine learning model performance, identify overfitting and underfitting, improve machine learning model transparency, and recognize and mitigate bias.

How does the course address transparency in machine learning?

It covers explainability, the ability to explain a model's predictions to stakeholders, including techniques such as feature importance that make predictions more transparent and understandable.

What does the course teach about putting models into production?

It covers deployment and maintenance of machine learning models, including how to deploy models into production and monitor their performance over time, as well as model drift, where performance degrades as the underlying data distribution changes.

What skills does this course help build?

It supports skills in Machine Learning, Machine Learning Algorithms, Machine Learning Methods, Machine Learning Model Training, and Machine Learning Model Monitoring and Evaluation, and relates to the SAS Certified Specialist: Machine Learning.