Support Vector Machines (SVM) can be used as a binary classifier for a wide range of data and problems. For text, it is often used to determine automatically if a document belongs to a given topic category. Automatic and reliable classification of documents can be an invaluable tool for industries that monitor problem reports, accident reports, maintenance logs, and corrective action reports that span different specialties.

Classifying aviation safety reporting system (ASRS) documents submitted by pilots, air traffic controllers, mechanics, airport workers, and many others involved with air travel is a problem since the large number of documents filed each month makes it difficult for humans to hand-classify the reports, and it becomes difficult to identify potential hazards and recurring problems. Supervised learning using SVM is a binary classifier that requires a training data set and certain hyper-parameters as inputs to build a model. The model is used to divide data whose class is not known such as explicitly defined training and validation data sets, or a large data set that will be broken up into an N-fold cross-validation by the program for inputs. The training set and hyper-parameters are fed into an SMV optimization problem, and the resulting support vectors and Lagrange multipliers are the outputs. The validation set is then used to check the quality of the support vectors and measure the error of prediction. The hyperparameters are most often determined by the data or simple trial-and-error searches. These methods rarely find optimum hyper-parameters that minimize the error of the classifier.

Mariana is an algorithm that efficiently optimizes the hyper-parameters for SVMs for regression and classification. It currently uses simulated annealing for optimization, but can be extended to use a variety of stochastic optimization techniques such as Markov Chain Monte Carlo (MCMC), Sequential Monte Carlo, and genetic algorithms.

The system automates the search for the optimum hyper-parameters by a variety of stochastic optimization techniques. The methods differ in their approach, but they all follow a similar procedure. Given a training and validation test set, the system randomly selects a set of hyper-parameters, builds a model from the training data, tests the model’s performance using the validation set, and compares performance to previous performance to determine if the current set of hyperparameters is better than the previous. The hyper-parameters are then recorded. This is repeated until no noticeable improvements in performance are determined, or at a predefined stopping point.

This work was done by Ashok Srivastava and Joseph Castle of Ames Research Center. This software is available for use. To request a copy, please visit https://software.nasa.gov/software/ARC-16070-1 .