Customizing keras-tuner for flexibility and maintainability
Introduction
This is a continuation of Hyperparameter tuning with keras-tuner full tutorial. If you do not have basic understanding of hyperparameter optimization and keras-tuner, it is recommended that you read previous blog or do basic research.
Today we will customize build() and fit() method from keras tuner as well as other codes to make our project more maintainable and scalable. Main reason for customizing build() and fit() is for improved flexibility and to implement complex models.
- All code will be provided in this github repo.
Let’s get right into coding!
We will import all necessary libraries, internal code, and specify callbacks, metrics, inputs, and loss function which will be passed to our custom HyperModel. Hypermodel is a class that allow us to use hp arguments to define hyperparameters.
After initializing hypermodel, reassign directory name get_project_name()(where hpo trials will be saved) if already exists since if you run with already existing directory name keras tuner does not run properly. Another way to avoid this is to add overwrite=True parameter when instantiating kt.RandomSearch.
RandomSearch class
- inherits from Tuner class(which also inherits from BaseTuner class): class that manages building, training, evaluation, and saving of keras models.
- contain RandomSearchOracle class as attribute: inherits from oracle class which is base class for all search algorithms, it evaluates results given by Tuner class and generate new hyperparameter value to search next.
Starting the searchtuner.search(), first builds a model with hyperparameters given by oracle in build(self,hp) function then built model gets passed to fit() function along with dataset, metrics, callbacks, etc… that has been passed at search method. This process is repeated at each search trial which new hp.
Benefit of custom build function is separation of model and keras tuner, this reduce dependency therefore model can easily be run standalone or with other hpo libraries with ease.
fit method train, update_metrics, save history, reset metrics for each epoch and evaluate on validation set if exists.
- line 2: create history dictionary for all metrics. {loss:[], val_loss:[],etc…}
- line 5~6: instantiate optimizer_name and lr with values in
hpthat has been passed from build() function. - line 15~21: for each batch train with custom training loop and for each epoch we save metrics and reset for validation or next train epoch.
- line 23~30: if validation data is given evaluate, save metrics, and reset.
Benefit of custom training loop is its ability to work with complex models and you could also change any smallest details like how to print training/validation process.
All methods of custom HyperModel class is not displayed so recommend taking a look at github repo.
Now once we are done searching we retrain our model with full dataset. Even though we could retrieve best model from tuner class it is not best practice because hpo methods like hyperband may not train model with full training dataset, best model retrieved is version that’s not trained on full training dataset.
we simply grab best_hyperprameters from tuner class, build our custom hypermodel with best_hps and call fit method just like in keras.
Conclusion
This is basically it, we’ve just customized both build and fit method of hypermodel in keras-tuner for enhanced maintainability and flexibility.
These few days I’ve been thinking about how important it is to write code that is maintainable, clean, and well-structured both for your future self and teammates because I truly believe that by simplifying code read, we could use energy saved by trying to understand complex and non-maintainable code to create creative algorithms and products.