This has included establishing the right use case, considering three critical business questions for ML, and prepping the data to ensure success. In this final article, I shed light on three additional areas that complete your comprehensive machine learning roadmap: model selection, evaluation, and establishing a center of excellence (CoE) to drive ML at scale.
At the most basic level, an ML model finds patterns in your data set, such as identifying objects in image recognition or forecasting demand for more effective inventory management. But with many ML models to choose from, how do you select one that is appropriate for the business problem and will deliver results that are relevant and valuable? For example, explainability will be a deal-breaker for industries with high regulatory and compliance standards, such as healthcare or finance. For other businesses, a more complex model that leads to better performance is preferable, even if it is a black box (with limited explainability).
The intricate nuances of selecting a model are beyond the scope of this article and include a wide number of criteria. For example, if you are looking for a nominal or categorical output, you might want to use a classification model such as Random Forests (an amalgamation of decision trees). However, if your ‘target’ is a discrete variable (i.e., estimating the price of a house based on features), the problem is best approached with regression learning algorithms such as logistic regression or generalized linear models. Additionally, the final model may often be a conglomeration of more than one distinct model or algorithm, particularly when building ML apps at scale.
No model is perfect, which is why evaluating your selection based on real-world business examples is key to achieving the best results for your organization. Data scientists use a number of criteria to evaluate and iterate their model, such as accuracy, recall, F1 score and area under ROC curve. For most ML models, your predictions can be classified as a True Positive (my model predicted a yes, and the actual result was a yes), False Positive (my model predicted a yes, but in reality, it was a no), and similarly with True/False Negatives. Let’s consider two examples to explain applicability for different use cases.
Weighting to False Positives
Weighting to False Positives would be appropriate for business problems like airport security or driver safety. Here, a False Positive means you screen someone as a threat who in reality is not. While this may cost you additional seconds of further analysis, it poses no tangible risk. However, if you weighted to a False Negative, a true threat may not be flagged and allowed to pass freely without additional screening. In this case, it comes with a high cost with heavy consequences.
Weighting to False Negatives
Conversely, weighting to False Negatives is appropriate for instances like automating loan eligibility. In this example, a False Negative could mean that an eligible loan candidate who would have paid back their loan was not given one because they were flagged as a risk: a missed opportunity, but one with no significant impact. Alternatively, weighting to False Positives means that you believe a loan candidate will pay you back when they are actually unable to. In this scenario, you would lose out financially.
It is important to note that evaluation is just the beginning of building a successful model. Every business must continue to run the iterative process of evaluating, tuning, and running again in order to achieve the best results.
The importance of machine learning ops
Once you have begun the iterative process of evaluating your model, and established it as successful for a single use case, how can you scale across the entire organization? With such fast technological growth over the last three years, we see many organizations struggling to take ML from a conceptual model to an enterprise-level deployment. That’s why the machine learning ops (MLOps) market is rapidly becoming one of the most important components of data science and is already being leveraged by the biggest tech companies in the world. This was a major point of discussion at this year’s MLconf San Francisco, with business leaders from Spotify, Airbnb, Walmart, and other experts discussing the importance of an Artificial Intelligence (AI) or ML center of excellence.
As ML production level scales in your business, MLOps are critical for pipeline automation, monitoring, lifecycle management and governance. Your CoE should serve as an ‘open source’ of ML learnings, shared ontologies, and centrally maintained libraries. This prevents silos developing across your different business units and ensures democratization of learning and maximum ROI. Indeed, in the future, those organizations that have these centralized ML and AI repositories will separate the winners from the aspirants.
I hope you enjoyed this article series on machine learning and found it an insightful addition to your own journey to technological excellence. At Cuesta, we turn ‘what if’ into ‘what’s next’ and help you create ML solutions that can achieve real ROI at scale. If you want to discuss any of these topics in more depth, please reach out to me for an informal chat – I would love to hear your insights!