3.2.3 SIV - Results

2.2.3.1 SIV Use Case Experimental Design and Methods
2.2.3.2 SIV gating SOP
4.2.3 SIV Discussion

Analysis has been done by Tree Star application scientists and a set of summer interns in two rounds (round 1 was done with an oral description of the process; round 2 was done after the interns updated the specifically written SOP). Additionally several rounds of automated gating have been applied to this use case.

Figure 1. CD4+ T-cell IL-2 population gated by 7 different trained scientists from one flow sample shows the gating variability between individual people.

Figure 2. Comparison of the variability of manual analysis, based on level of experience and
training. Shown are the % of CD4+ T-cells with IL-2 responses by individual analyst. Experts
are Tree Star application scientists. Interns are high-school students trained to find these
populations, before and after an improved SOP for the gating.

Match Ratio is a comparison between a single classification result and the consensus results of a group. The closer the Match Ratio is to 1.0, the more closely an individual's gated population is to the consensus. Cells in the manually classified samples are weighted based on the frequency with which they were included by all the persons forming the consensus. The sum of the weights of all the cells in the sample is the total possible score. The consensus is similar to a probabilistic cluster. For a single classification act, each cell in a population is compared to the consensus probability of inclusion. If there is a match between a cell in the individual's gate and that cell in the consensus gate, then the weight of that cell is added to the individual's accumulating score. A sum of these weights is compared with the total possible score of the consensus group to produce a Match Ratio. If candidate agrees with expert average on every event, MR = 1. For every event missed, MR drops away from 1, with less weight given to events less well resolved by the experts. A detailed explanation of Match Ratio is available in this PDF.

Figure 3. When using a widely variable group, such as the experts, for the consensus, it is fairly easy to achieve a high Match Ratio score because there is not as much agreement in the consensus. When using a consensus with less variability, such as the interns, the Match Ratio scores spread out, as seen when evaluating each expert to the consensus of interns on the far right of this figure.

How do we know if there is too much variability in the consensus?
One way is to look at the biological outcome of the experiment and see if the same answer is found by all manual gaters included in the consensus. For this experiment, this meta-analysis requires the evaluation of the cytokine responses over time. The stimulation is adjusted for background by subtracting the % response in the unstimulated well from the % peptide responses, and the data is reported as this difference. These cytokine responses of the T-cells are viewed over the time-course of the study for each primate.

Figure 4. CD4+ T-cell IL-2 responses show an increase at time-point 2 for all primates, with the exception of one gater for Monkey 3. This data point is probaby errant and would be flagged on the basis that its Match Ratio score would be very low compared to the consensus. Flagged samples would need to be checked manually in the completed FlowDx analysis.

Therefore, even with the widely varying results shown in Figures 1-3, all experts agreed that IL-2 responses were up significantly at time-point 2 for all primates.

John Quinn of Tree Star, Inc. has done several rounds of analysis of this data using algorithmic methods of classifying the cells. His results are shown in several files.

Support Vector Machine Algorithm Preliminary Report
Automated Classifier Report