Preliminary Support Vector Machine Report 8/15/09

by Dr. John Quinn, Tree Star Application Scientist

Summary
Support Vector Machines (SVMs) were tested as a potential classifier for flow cytometric data using
the SIV data provided by Joern Schmitz. Two types of SVMs were tested, one with a polynomial
basis function, and another with a radial basis function. Several calibration experiments were
performed using lymphocytes identification as a metric. In these calibration experiments appropriate
sizes for training vectors, design of training data, and degrees for the polynomial basis function were
determined. The SVMs were then used to identify the six populations at the bottom of the gating
hierarchy, the populations expressing either CD4 or CD8 along with a set of three cytokines. Training
data labels and “correct” classification were determined using the match ratio statistic.

Both of the SVMs proved able to learn the expert classification pattern. The average match ratio for
the radial basis function (RBF) SVMs for the CD4 panel was 0.937 ± 0.011 and was 0.942 ± 0.013 for
the polynomial basis function (PBF) SVMs. For the CD8 panel the average match ratios were 0.896
±0.008 (RBF) and 0.939 ± 0.015 (PBF).

The next step in this analysis would be to examine the results further and determine how the SVMs
performed on the difficult data, for example events with match ratio averages among experts below
0.833, and then to contrast the reasons for machine classification versus expert classification. This
experiment provides us an SVM benchmark that can next be compared against Artificial Neural
Networks (ANNs) and other pattern recognition tools.

Background
Support Vector Machines (SVMs) are deterministic algorithms designed to identify the vectors within
the training data that define the boundaries between classes of data. Non-linear boundary problems are addressed using support vector machines by including a basis function that maps the input data into a transformed space that allows a linear discriminant to separate the classes.

In this experiment we have chosen to use SVMs with polynomial basis functions (PBF), the most common SVM, and with radial basis functions, a choice that seems appropriate for flow cytometric data that is thought to be distributed in a Gaussian or radial manner in many cases.

Procedure
The SVMs were implemented using Matlab® (Natick, MA) software, using a set of SVMs produced at
Ohio State University by XXXX.

Data were submitted by Joern Schmitz. For this preliminary study SIV run 1 was used exclusively.
The data set consists of 9 files; three unstimulated controls identified here as A01 through A03, three
samples treated with peptide identified here as B01 through B03, and three positive control samples
treated with PMA, identified here as C01 through C09. Match ratios were calculated in FlowJo andexported to text files that were then read into Matlab. Data with FSC and SSC values above the
measurable threshold were removed in Matlab by removing all events in the maximum bin. Antibody
dependant parameters were transferred to a logarithmic scale and then all data were normalized to a
unit scale.

‘Positive’ training events were determined using the average score among experts as a metric. Events
with scores of 1.000 were events considered positive by all experts and were used as exemplars.
Events with average scores of less than 0.5 were considered ‘negative’. The value 0.5 was chosen
because the match ratio equation deems these events as negative. Choosing events that were
universally classified as negative was not selected as it was experimentally observed that the training
data set became over simplified and the resulting decision boundaries were unintuitive. In this work,
positive is used to mean events classified as part of a population, or within a gate, and negative is used to mean events outside of the population or gate.

To select certain experimental criteria, we began the experiment by attempting to classify the
lymphocyte population using the FSC, SSC, and CD3 parameters. Classification was performed using
training data segregated from each file while the testing was performed on the remainder of the data.
The number of events to use in a training set was determined experimentally then by classifying the
lymphocytes using training sets of 100, 500, 1000, 2000, and 5000 events. Results are shown in Figure 1 below. It was observed that the match ratio scores declined noticeably between 100 and 500 samples but then increased to near even performance thereafter. One thousand events were chosen for all training sets thereafter to avoid the less stable sets.

Additionally training the data with a set taken from a single file was compared to using a training set
that was a composite of multiple files. As expected, the results from multiple files were better so this
procedure was used. Data not shown.

With the experimental conditions determined, the next task performed was to train SVMs to classify
the six populations lowest on the gating hierarchy, populations that were positive for either CD4 or
CD8 along with one of three possible cytokines TNF, IL2, or IFN. For each of these populations the
SVM was trained using the parameters FSC-A, SSC-A, CD3, either CD4 or CD8 depending on the
subset, and whichever cytokine we were looking to identify the events that bound it, for a total of 5
dimensional analyses.

Data were all preprocessed as described above, and classified using each SVM.

Results
Shown below are tables listing the match ratio scores for first the CD4+ subsets (Table 1) and then the
CD8+ subsets (Table 2). Following that are plots of each training set used for the six populations and
an example of the classification result for each of the two SVMs. The classification result in all cases
is for sample C03, the sample with the most disagreement between experts, lowest match ratio scores,
and thus the sample presumed to be most difficult to classify.

 

            Table 1:  Match ratio scores for the CD4+ subsets

 

 

            Table 2: match ratio scores for the CD8+ subsets

 

 

Training Data	RBF Result	PBF Result

Figure 2: Plots illustrating a training set and classification results

 

Conclusions

 

Both of the SVMs proved able to learn the expert classification pattern.  The average match ratio for the radial basis function (RBF) SVMs for the CD4 panel was 0.937 ± 0.011 and was 0.942 ± 0.013 for the polynomial basis function (PBF) SVMs.  For the CD8 panel the average match ratios were 0.896 ±0.008 (RBF) and 0.939 ± 0.015 (PBF).  This indicates that the machines achieved on average above 90% of the possible match points possible.  This experiment can serve as a benchmark for SVMs for future comparison to alternative pattern recognition tools such as ANNs.

 

We can observe in the plots above that there is significant overlap in much of the training data.  These plots all show all events (except those excluded as being above the measurable FSC or SSC threshold), and so it is not surprising that some events that are positive for a cytokine may be a negative example as they were excluded in some prior gating step.  What this does demonstrate is that this is a non-trivial, non-linear classification problem.

 

The next step in this analysis would be to examine the results further and determine how the SVMs performed on the difficult data, for example events with match ratio averages among experts below 0.833, and then to contrast the reasons for machine classification versus expert classification.  It is possible that the machine classification can be demonstrated to be more consistent and more rationally determined.  It would also be interesting to identify the cells that the machine differed in the classification of from either the experts or the other SVM to understand which SVM performed better and what was the basis of the decisions to better understand the overall project better.