SVM’s and Our Data Source

With Eric Miller

History and Background

• Developed by Vapnik and Chervonenkis in 1964 then revised in 1992 to incorporate Non-linear classifiers
• Support Vector Machines (SVM) employs supervised learning using past data to train models for new cases
• SVM models are used to form clusters of two distinct data groups
• SVM establishes a hyperplane to maximize the margin between data groups
• The hyperplane acts as a line separating the clusters
• The goal is to ensure the greatest separation between neighboring members of each cluster

Data Source

• Our team is utilizing data from the Medical Information Mart for Intensive Care (MIMIC)
• We are focusing on the MIMIC III dataset.
• The dataset was pre-processed to isolate the most statistically significant features

Methods - Kernal Function

• Support Vector Machines operate on both linear and non-linear data by using a kernel function or “kernel trick” to manipulate non-linear data into a linear space for classification. The general formula for a kernel function is as follows, where \(X_i, X_j\) is a tuple.

\[ K(X_i, X_j) = \Phi(X_i)\Phi(X_j) \]

Methods - Hyperplane

Once the data is linearly separable, we can define our maximally marginal hyperplane. A general formula for the hyperplane is as follows \[ W \times X + b = 0 \]

This formula has 2 components of import. The first is a \(W\), a weight vector and \(b\) a scalar bias.

Since \(W\) is a simple weighting vector it would be in the form of the below

\[ W = \{w_1, w_2, \dots, w_n \} \]

Methods - Support Vectors

The tuples that lie the closest to the margins of the maximal marginal hyperplane are the actual support vectors.

Using the formulas on the previous slide, if our support vectors where at \(y_i = 1\) and \(y_i = -1\) our hyperplane margins would be defined as

\[ H_1: W_0 + W_{1}X_{1} + W_{2}X_{2} + \dots + W_{n}X_{n} \ge 1 \]

and

\[ H_2: W_0 + W_{1}X_{1} + W_{2}X_{2} + \dots + W_{n}X_{n} \le -1 \]

SVM Vizualization

What was wrong with our model

• Class imbalance !!!
  • 86% of the time, patients surived
  • 14% of the time, patients died
• Random sampling could exacerbate this problem

Fixing the model

We attempted 2 strategies to fix the

• Oversampling of minority case
  • Ensure that minority case was a larger percentage of the training set via selection with replacement
• Downsampling majority case
  • Take a large percentage of the minority cases, select and EQUIVALENT number of majority cases for a 50/50 split

Tuning the new model

• tested using a rbf (radial bias kernel)
• switched to nu-classification
• tuned for nu and gamma
  • using grid search strategy and a bunch of packages

Tuning the new model…more problems

• Tune SVM is VERY VERY SLOW
  • does not leverage mulitiple cpu cores
  • operates sequentially
• doParallel and foreach to the rescue
• still slow but much quicker at the same time

Tuning the new model and paying more for it

Tuning Results

• nu-classification won out
• Primary Model
  • Gamma = 0.02
  • Nu = 0.75
• Challenger model
  • Gamma = 0.17
  • Nu = 0.87