Study: Alternative Artificial Neural Networks (ANN) and use of the Waikato Environment for Knowledge Analysis (WEKA) for implementation

by Dr. John Quinn, Tree Star Application Scientist

For this particular study, the AIM was to test artificial neural networks as a classification tool on a data set that we will use in the design of FlowDX and to evaluate the usefulness of the Waikato Environment for Knowledge Analysis (WEKA) and a design and testing tool.

Summary of results and importance of the ANN and WEKA study:

The Waikato Environment for Knowledge Analysis (WEKA) was tested to evaluate the ability of several Artificial Neural Networks (ANNs) to correctly classify flow cytometric data and to evaluate the platform itself for usefulness in continued evaluations. WEKA is a popular and free collection of machine learning algorithms for data mining tasks, and thus a strong candidate for larger use within this project for developing algorithms.

The result of this evaluation was that four of the five ANNs classified the design data set with greater than 85% success. The metric used was ten fold cross validation. We could thus conclude that ANNs will be a valuable tool to proceed forward with, and that certain types of ANNs, particularly the immune based networks, that we had not previously considered may be of importance in our work.

The overall assessment of the WEKA platform is that it implements the ANNs in a quick and easy to use manner, allowed for testing some complicated ANNs without the need to program them, and is a nice tool for setting up statistical comparisons. The WEKA platform also introduced us to some new and useful networks or modifications to traditional networks. However, it was found lacking, as it requires CSV inputs that were time consuming to assemble and did not “batch” process, thus requiring each classification study to be assembled and run individually. WEKA also does not handle large files and so it required testing by classifying only small samples of the data. Larger and sortable data files could also be used to improve accuracy, as they would reflect the ratio of classes in the data. Additionally, trained networks could not be saved, and so a trained network could not be applied to alternative data. Finally, the output was not accessible on a per cell basis; so output stats like match ratio could not be constructed.

Data Description

Treestar received SIV data from Dr. Joern Schmitz of BIDMC at Harvard. Ten non-mutually exclusive populations are to be identified in these peripheral blood mononuclear cells (PBMCs). Shown hierarchically, those populations are:

Lymphocytes

CD3+ Lymphocytes (T-cells)

i. CD4+ T-cells (activated T-cells)

1. Activated T-cells & IFN+

2. Activated T-cells & IL2+

3. Activated T-cells & TNF+

ii. CD8+ T-cells (antigen specific T-cells)

1. Antigen Spec T-cells & IFN+

2. Antigen Spec T-cells & IL2+

3. Antigen Spec T-cells & TNF+

The total data set included 9 files, 6 controls and a set of 3 files that were expected to stain positively for three cytokines, tumor necrosis factor alpha (TNF), interferon gamma, (IFN) and interleukin 2 (IL2). Each of the files was classified by a panel of experts. The controls did not contain many cells identified by the experts to be positive for the cytokines, so the three positive data files were used for testing. It is expected that the cytokine identification results should be similar the CD8+ and CD4+ branches, only CD4+ were tested here. For simplicity in the training and reporting the populations have been abbreviated as:

Lymph

Tcell

Act Tcell

ANNs

WEKA provides the user with three types of immune based ANNs, each of which with variants, nine types of Learning Vector Quantization (LVQ) networks, and four perceptrons. After removing from consideration all unsupervised methods, and choosing the latest variant of each network that has been improved over time, we are left with 5 networks to test:

*Immune based*	LVQ networks	*Multilayer Perceptron*
1. AIRS2	1. Optimized-LVQ3	1. Back propagation Multiperceptron
2. Immunos2	2. Multi-pass - LVQ

All of these tools involve creating a weighted, directed graph that maps the data inputs to classification outputs. The nodes of the graph are activation functions that are used to assess similarity between the input and the node, while the connections are weights. In each system, the nodes and weights are modified to create a graph that will do its job better, specifically to take the training inputs and produce the correct outputs. The immune based ANNs create and mutate more nodes that are similar to the training data while removing dissimilar nodes. In this manner the network can evolve to become increasingly more fine in its ability to distinguish types of inputs. The LVQ networks adjust the nodes to map the data space and cluster the input data per node, then use a second layer to assign clusters to the desired classification. The multilayer perceptrons map the inputs to a data space where a linear classifier can be used to separate classes.

Procedure

For each of the six populations, from each of the three files, a “positive” population was identified by randomly choosing 500 cells from among the data space that had been identified by all of the experts as “positive” during a gating exercise, and were thus considered “consensus positive”. It was experimentally determined that WEKA was not able to handle more than two thousand events without analysis interruptions so 500 cells were chosen per population. These cells were then exported from FlowJo as CSV. For each of the six populations a set of “negative” cells was selected for ANN evaluation by choosing 500 cells randomly from outside the “positive” data space created by the experts. The data were combined using Excel to place a negative population with each positive population for training and testing.

The data were presented to each ANN and classified. ANN configuration was chosen by rule of thumb knowledge, with the defaults accepted when choices were debatable. Experimentation was used to attempt to improve the multiperceptron performance after it became obvious that its performance was at a success rate near random classification. Cross validation was chosen as the evaluation method due to WEKA lacking event by event result access, and cross validation being an accepted error gauge. Results were then compiled in Excel. All parameters were shown to the ANNs.

Results

Treestar tested the efficiency of multiple ANNs at classifying multi-parametric data as being included in an expert defined population, or being “positive” in the vernacular, in a cross validation environment. The ANNs were generally successful at this especially considering the limits imposed by WEKA and the variability human experts have demonstrated in similar experiments. Thus Treestar has benefited by identifying several types of networks that appear useful to incorporate into automatic classification software. We have also gained the knowledge that the multiperceptron, a very basic and common form of ANN, should be tested further and potentially eliminated from discussion.

The following tables show the misclassification rate per ANN for each of the three samples, followed by an average misclassification rate per sample across populations. The last table is an average across all samples per ANN.

Sample C1	Misclassification rate
	Lymphs	Tcell	Act Tcell	TNF	IL2	IFN	AVE
AIRS2	1.9%	4.1%	5.0%	21.3%	14.0%	6.6%	8.8%
Immunos2	9.3%	13.0%	30.7%	14.7%	10.8%	9.3%	14.6%
O-LVQ3	1.0%	10.6%	22.9%	12.8%	10.3%	4.4%	10.3%
M-pass LVQ	0.3%	7.0%	17.6%	12.2%	7.6%	3.8%	8.1%
Multiperceptron	46.1%	47.8%	49.1%	34.2%	46.6%	24.4%	41.4%

Sample C2	Misclassification rate
	Lymphs	Tcell	Act Tcell	TNF	IL2	IFN	AVE
AIRS2	1.6%	2.8%	6.2%	24.9%	19.6%	0.0%	9.2%
Immunos2	9.7%	11.5%	29.5%	17.7%	13.3%	9.3%	15.2%
O-LVQ3	2.5%	8.2%	20.8%	16.0%	9.5%	5.2%	10.4%
M-pass LVQ	1.2%	4.2%	15.6%	16.3%	7.6%	3.8%	8.1%
Multiperceptron	47.1%	49.8%	48.2%	42.2%	34.7%	36.5%	43.1%

Sample C3	Misclassification rate
	Lymphs	Tcell	Act Tcell	TNF	IL2	IFN	AVE
AIRS2	1.6%	4.9%	2.3%	15.9%	13.2%	8.3%	7.7%
Immunos2	9.7%	12.0%	24.6%	14.6%	9.7%	18.9%	14.9%
O-LVQ3	2.5%	10.4%	18.3%	13.6%	7.1%	5.0%	9.5%
M-pass LVQ	1.2%	6.3%	16.0%	13.2%	7.4%	3.8%	8.0%
Multiperceptron	47.1%	49.9%	47.8%	37.1%	33.4%	19.4%	39.1%

ANN	AVE.	±	Stdev
AIRS2	8.6%	±	7.7%
Immunos2	14.9%	±	6.9%
O-LVQ3	10.1%	±	6.4%
M-pass LVQ	8.1%	±	5.7%
Multiperceptron	41.2%	±	9.1%

Of particular gain in using the WEKA environment as a starting point was exposure to modification of the LVQ algorithms, such as a multi-pass formulation that performed very well, and the AIRS class of algorithms. These are algorithms to take forward in our research. The graphical and numerical results put out by WEKA were also nice.

In terms of evaluating WEKA as a operational tool going forward, the data handling was found to be tedious as WEKA requires CSV files for input. The potential classification metrics are limited to traditional measures, so mach ratios could not be considered nor could a file aside from the training file be classified by a given network. The limited size of data that WEKA could handle also limited the training/testing sets to the smallest possible number of events. Coupled with a more able medium for assembling training and testing data, better data sets could be assembled. One of the biggest disadvantages to the platform was its inability to batch any type of operation. It is likely possible to write WEKA scripts, but if I were to start writing scripts, Treestar would do so for a more versatile platform.