2.1.6.1.2 Collaborative Process -Draft 12/10/2009

The Collaborative Process Document more fully defines the protocol, whereby a researcher can compare two or more classifications of identical datasets to study the differences, biases and effectiveness of human and algorithmic classifiers. Diagram and text describing how a new client/scientist would get FlowDx customized for their Assay. From approaching Tree Star to a working automated analysis system in their hands. FlowDx Collaborative Process Diagram is the prototype workflow which needs to be updated to reflect the complexity on the research side such as purchasing department, QA, technicians, lab manager, legal department, and all personnel involved in the process. We should have two versions of this workflow; one for use cases during the construction phase and one for FlowDx clients.

The Collaborative Process Document describes the process of starting with a client with a clinical assay and finishing with FlowDx doing automated analysis for this client. This will also include the Questionaire based on MiFlowCyt for the potential clients to describe their Assay methods, analysis, QC and requirements for the FlowDx automated service.

Figure 1.
Functions of the Database/Repository are shown on the right.
Collaborative activities are shown from top to bottom.

Conduct a prescribed set of experiments on our identified use cases, illustrating the potential of this technique to affect clinical analysis. Apply range of automated classifications to use case data and evaluate outcomes.  Algorithms can be used alone or in combination  Compare to manual consensus gating with range of comparison metrics

Steps to be performed on each use case:

1. Have a group of experts manually gate a biologically relevant subset of the experimental data submitted by the collaborator to test the clarity of the experiment description and SOP for gating; and to form the consensus gating.  Is there is agreement of the manual expert gating results as determined by the submitting collaborator? (Should the manual gating group be Application Scientists or Technicians in the Researcher’s Lab?)
2. Manual gating should be repeated to include all experimental data.  If there is not agreement of the manual gating as determined by the comparison metrics, then the different approaches to the manual gating need to be discussed with the submitting researcher and the experts doing the manual gating.
3. Automated classification methods need to be tested on the biologically relevant subset of the experimental data as a preliminary test.  Determine how the classification metrics will calculated and reported.  Some possible choices are: 
• Average/Mean/Median of all samples and populations with the CV and range reported. 
• a table showing the comparison metric values for each sample and each population to enable the viewing of difficult populations or samples for classification .  
4. If comparison metric is within the range of the expert consensus, then can proceed to classify all the data for a use case.  If comparison metric values are outside those defined by the consensus, then adjust the classification methods, look for outlier samples or populations that are not easily classified and look for solutions.
5. Classifying the entire set of data for the use case for each classification method.  

Still to be included:  

Show an example to demonstrate what the results may look like.

•  Subset of biologically relevant samples
•  Group of experts manual gating and their match ratio scores vs. MFI, # events, % parent, corrected % parent (- unstim)
•  Show the comparison metric scores of the classification algorithms.