Model Validation

One of the most important steps in machine learning is model validation. It's essentially a test run to determine how well the model will perform in the real world. Should the model display unsatisfactory performance during this process, programmers will need to take what they've learned from the test and use it to tune the model's hyperparameters and adjust its training data. 

‍

What is Model Validation? 

Model validation occurs immediately after a machine learning model has been fully trained. It typically presents the machine learning model with data it has never directly encountered before. The core idea is that if the model is properly trained, it should be able to generalize to the new data. 

The model validation process can be divided into one of two categories based on the source of its test data: 

• In-sample validation uses testing data from the same data set that was used to train and build the original model.
• Out-of-sample validation pulls testing data from an entirely new data set. 

Additionally, there are many different model validation techniques — each one applies the test data in a different way.  

Why is Model Validation Important? 

Model validation is important for the same reason as any performance or quality assurance test. It determines whether or not a machine learning model does what it's intended to do outside the confines of its training environment. An unvalidated machine learning model is essentially an unknown quantity — there's no real way of knowing if it's able to accurately and effectively generalize on unseen data. 

Additionally, model validation helps programmers optimize and fine-tune their machine learning model, while also identifying potential problems before the model moves ahead to final testing. It also allows development teams to compare the performance of different models, as well as models trained on different data sets, in order to identify which ones would be most effective at fulfilling their goals. Lastly, model validation may be carried out by a third party in the event that the model has to adhere to certain regulatory requirements. 

It's important to differentiate between model validation and model testing, as well — the test set is reserved for the final, optimized model. 

Types of ML Model Validation

There are several different types of machine learning model, and each one has multiple validation requirements depending on purpose, use case, and dataset. 

• Supervised learning models predict outcomes based on data analysis. 
• Unsupervised learning models identify patterns in unlabelled data.
• Hybrid learning models combine one or more machine learning techniques for optimal predictive performance. It's especially important to validate a hybrid learning model to ensure it's not underfitting or overfitting its data. 
• Deep learning models are more powerful and sophisticated than most other machine learning models, typically consisting of multiple neural networks — learning models that can make decisions without the need for predetermined datasets or parameters. Validation is crucial for deep learning models in order to ensure they can accurately achieve their stated purpose. 

The different types of machine learning model validation include:

• Train-test-split: Training data is split into two parts, usually either 80-20 or 70-30. One part is used for training, the other is applied in model validation.
• K-fold cross: The data set used to train the model is divided into a set number of equally sized subsets, represented by the variable k, and the model is trained 'k' times, using one fold for validation and another for training.  Finally, performance is averaged out over 'k' iterations. There are several different variations of this technique, including Leave-One-Out-Cross-Validation and stratified k-fold cross validation. 
• Time-based splits: Specifically leveraged in scenarios where timestamps are relevant to training data. Data is split so that the validation sets occur chronologically after the training set. 
• Random forest models: A specialized type of machine learning model that leverages multiple decision trees to randomly select samples from a data set, frequently used as validation tools for other machine learning models. 
• Support vector machines: A support vector machine is a machine learning model capable of maximizing the margin between differently-classed data points, making it an excellent tool for data validation.