Research Design
1. To more fully define the experimental protocol, whereby a researcher can compare two or more classifications of identical datasets to study the differences, biases and effectiveness of human and algorithmic classifiers.
1.1. Define the FlowDx process that we want to achieve for a researcher to submit an experimental protocol (Assay protocol)
1.1.1. Submit an Assay with MiFlowCyt standards description including 1) Assay Overview, 2) Sample Details, 3) Instrument Details, and 4) Data Analysis Details to FlowDx, LLC.
1.1.2. The Assay would proceed through prescribed set of experiments described in step 2 below and in the accompanying diagram. Each time a report is sent to the researcher, there is consultation and agreement as to whether the next step should be taken.
1.1.2. The Assay would proceed through prescribed set of experiments described in step 2 below and in the accompanying diagram. Each time a report is sent to the researcher, there is consultation and agreement as to whether the next step should be taken.
2. Conduct a prescribed set of experiments on our identified use cases, illustrating the potential of this technique to affect clinical analysis. Apply range of automated classifications to use case data and evaluate outcomes. Algorithms can be used alone or in combination Compare to manual consensus gating with range of comparison metrics
Steps to be performed on each use case:
- Have a group of experts manually gate a biologically relevant subset of the experimental data submitted by the collaborator to test the clarity of the experiment description and SOP for gating; and to form the consensus gating. Is there is agreement of the manual expert gating results as determined by the submitting collaborator? (Should the manual gating group be Application Scientists or Technicians in the Researcher’s Lab?)
- Manual gating should be repeated to include all experimental data. If there is not agreement of the manual gating as determined by the comparison metrics, then the different approaches to the manual gating need to be discussed with the submitting researcher and the experts doing the manual gating.
- Automated classification methods need to be tested on the biologically relevant subset of the experimental data as a preliminary test. Determine how the classification metrics will calculated and reported. Some possible choices are:
- Average/Mean/Median of all samples and populations with the CV and range reported.
- a table showing the comparison metric values for each sample and each population to enable the viewing of difficult populations or samples for classification .
- If comparison metric is within the range of the expert consensus, then can proceed to classify all the data for a use case. If comparison metric values are outside those defined by the consensus, then adjust the classification methods, look for outlier samples or populations that are not easily classified and look for solutions.
- Classifying the entire set of data for the use case for each classification method.
3. Iteratively implement the tools to automate these experiments, improve the experimental capabilities and collaborate in new use cases.
3.1. Match Ratio – determine what a “good” score is. What is that based on? What is the Cut-off?
3.2. Determine the questions Tree Star needs to be able to ask of the data and analysis and results. (Defined by the Data Analysis Details in MiFlowCyt description of the Researcher’s Assay)
3.3. Determine the questions that the researcher will ask about the analysis and results. (Are there samples that are more difficult to classify? What classification method is the best? Which classifiers have higher false positives vs. false negatives, confusion matrix.)
3.4. Build the tools to be able to answer these questions.
4. To maintain prescribed standards of software quality, establishing metrics in uptime, throughput and robustness that may set the bar for future iterations.3.2. Determine the questions Tree Star needs to be able to ask of the data and analysis and results. (Defined by the Data Analysis Details in MiFlowCyt description of the Researcher’s Assay)
3.3. Determine the questions that the researcher will ask about the analysis and results. (Are there samples that are more difficult to classify? What classification method is the best? Which classifiers have higher false positives vs. false negatives, confusion matrix.)
3.4. Build the tools to be able to answer these questions.
4.1. Is this referring to the FlowDx automated solution itself? Could you define some of these and how they will be measured and reported?
1) Inception Phase = Pre-Grant
2) Elaboration Phase = Grant Phase I and Grant Phase II
3) Construction = Grant Phase II
4) Transition = Post Grant Funding, How we take FlowDx to market
Aspects that cover multiple
Phases
1. All Documents - storage, revision, backups, accessibility
2. Presentation/Publication – plans and links