I have a few questions regarding the evaluation in the competition:
(1) In the evaluation metrics it says: (a) AUC for prediction of pCR following NAC, (b) wCV for repeatability of quantitative DWI measures. Is it possible to compete just on one of the metrics?
(2) It says that in the test phase, we can submit 10 individual metrics and no resubmission are allowed. What do you mean by individual metrics? You wrote that the metrics are AUC or wCV. So, do you mean that we have 10 submissions to predict pCR, and we can choose for that any data we want? Can we have two submissions that use the same train data but with a different algorithm? Also, among the 10 submissions of the same team - do you choose the best AUC the team achieved or the latest one?
(3) Is it allowed to submit to the training phase till the end of the competition?
(4) Is there an example submission? There is a place holder in the submission guidelines but no real link.
Thanks, Simona.Posted by: simona-rc @ July 16, 2021, 1:34 p.m.
( Simona: this is just the email I sent. Afraid I forgot to post it here for others benefit).
I'm afraid the terminology is confusing sometimes! The “Metric’s” here are the experimental measures the challenge participants are submitting as proposed predictors of pCR. AUC is how we are evaluating those metrics. So, for example if doing histogram analysis of the ADCs in the tumor regions one might propose using 15th percentile or median as predictors of pCR – those would be 2 separate metrics counting towards your maximum of 10.
The wCV is a secondary analysis to help see how much variability there is in the measurements. It will not be used in ranking the submissions. We ask that people include it if it is relevant – i.e., if the metric is dependent on the DWI measurements. Since only the DWI got repeated in the test/retest scans wCV will just be 0 if a metric only depends on the DCE images.
I can’t guarantee the training submissions will stay open until the end of the trial, but I think it will.
Regarding the Forums: Sorry we had gotten a little lax on monitoring them. Will try to do better. But you were the 1st one to submit stuff there, so you didn’t miss anything.
We are working on an example submission. Should be available soon.
Thank you David!Posted by: simona-rc @ Aug. 8, 2021, 7:57 a.m.