Estimating ML-Models Financial Impact
- The course is aimed at teaching how to estimate the expected financial results of a given ML model.
- Learn principles of projects valuation which are also relevant for model implementation projects Learn the difference between NPV, IRR and PI
- Learn how to plot benefit curves which are similar to ROC curves but represent expected financial benefits Learn about different ways the decisions are made based on model predictions
- Learn about reasons and consequences of model risk Learn how to account for unexpected decrease in model quality with the help of confidence intervals
- Learn how to evaluate A/B test results with the help of hypotheses testing Learn how to properly design A/B tests
- Learn about potential biases within historical data which can mislead your financial estimates Learn how metalearning can help to restore unobserved events
- Project valuation: valuation metrics, planning and rulesModel development and deployment is always a project. During this week, we will discuss project valuation. We will consider the general financial metrics such as Net Present Value, Internal Rate of Return, and others.
- Model quality and decision making. Benefit curveDuring the second week, we will focus on decision-making based on model predictions and the relationship between model quality and financial benefit. We will discuss how to plot benefit curves using different threshold decisions and optimize the financial results using threshold tuning.
- Estimating model risk discountsWhen a model is being used in the production environment for a long time, its quality can deteriorate. During this week, we will learn to calculate confidence intervals for model quality estimates and estimate potential negative financial effects.
- A/B testing and financial result verificationA/B testing is a great way to verify our expectations concerning the financial effects. During this week, we will discuss the principles of A/B testing, its design, as well as the ways to assess its outcomes.
- Unobservable model errors, metalearningImagine that our historical data is biased, and we cannot obtain any other data. During this week, we will discuss how to restore unobservable events by using such methods as reject inference and metalearning.
- Interim assessment (2 module)The assessment is based on the certificate for the online course. Score (10-point scale) Criterion 10 95 - 100 % 9 85 - 94 % 8 75-84% 7 65-74% 6 55-64% 5 45-54% 4 35-44% 3 25-34% 2 15-24% 1 less than 15%
- Cornwall, J. R., Vang, D. O., & Hartman, J. M. (2019). Entrepreneurial Financial Management : An Applied Approach (Vol. Fifth Edition). New York: Routledge. Retrieved from http://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=edsebk&AN=2237944
- Bernard Marr, & Matt Ward. (2019). Artificial Intelligence in Practice : How 50 Successful Companies Used AI and Machine Learning to Solve Problems. Wiley.