Validation results

This is a validation report for model Carcinogenicity prediction with Ensemble of Classifier Chains.

General information

The model was validated with a 10-times repeated 10-fold cross-validation.

Performance measures

measurefull-namesynonymsdescriptiondetails
accuracycorrect predictions / all predictions
aucarea under (the roc) curveprobability that the classifier ranks a compound with class active higher than with class inactiveto compute auc, the predictions are ranked according to confidences given by the classifier for each prediction, i.e. first the compounds with high confidence for class active, than the compounds the classifier is unsure about, than the compounds with high confidence for class inactive
sensitivityrecall, true positive ratecorrectly predicted active compounds / all compounds that are really active
specificitytrue negative ratecorrectly predicted inactive compounds / all compounds that are really inactive
ppvpositive predictive valueprecision, selectivitycorrectly predicted active compounds / all compounds that are predicted as activeppv is the probability that a active prediction is correct
npvnegative predictive valuecorrectly predicted inactive compounds / all compounds that are predicted as inactiveppv is the probability that a inactive prediction is correct
subset-accuracynumber of test compounds with all endpoints predicted correctly / number of all test compounds
inside-adnumber of test compounds inside the applicability domain / number of all test compounds

Probability that a prediction is correct

When applying the model to an unseen compound, the performance measures ppv and npv give a probability estimate that the prediction is correct. The confidence of the prediction is taken into account to make the probability estimate more accurate. Therefore, ppv and npv have been computed for different confidence levels.

Average performance over all endpoints

The average measures have been computed as the mean of all single-endpoint measures, these measures are so-called 'macro'-measures (Exception: subset-accuracy is computed using all endpoints). Each endpoint is weighted equally.

accuracyaucsensitivityspecificityppvnpvsubset-accuracyinside-ad
0.6660.7490.6420.7040.6890.6630.4820.978

Single endpoint validation

activityoutcome-cpdbas-singlecellcall

The endpoint activityoutcome-cpdbas-singlecellcall is 706 x active, 799 x inactive and 3 x missing in the training dataset. In each cross-validation 178.4 (of all 1505 non-missing compounds) were predicted with high confidence (>66%), 583.9 with medium confidence (>33%) and 739.1 with low confidence (<33%).

model confidenceaccuracyaucsensitivityspecificityppvnpvinside-ad
all predictions (ignoring confidence)67.40273.56667.45167.45470.23364.60999.761
predictions with high confidence (>66%)87.97788.48591.08279.13992.84972.98299.823
predictions with medium confidence (>33%)72.30575.34572.05172.68372.38372.29499.583
predictions with low confidence (<33%)58.67161.61655.28962.18559.57958.03799.876

activityoutcome-cpdbas-rat

The endpoint activityoutcome-cpdbas-rat is 618 x active, 580 x inactive and 310 x missing in the training dataset. In each cross-validation 104.2 (of all 1198 non-missing compounds) were predicted with high confidence (>66%), 434.1 with medium confidence (>33%) and 655.1 with low confidence (<33%).

model confidenceaccuracyaucsensitivityspecificityppvnpvinside-ad
all predictions (ignoring confidence)65.171.20157.29772.57466.38364.37699.62
predictions with high confidence (>66%)86.86287.88786.90686.99595.06770.6299.84
predictions with medium confidence (>33%)73.38773.69458.14885.02374.57972.9999.559
predictions with low confidence (<33%)56.23158.76549.95162.28855.27457.17199.622

activityoutcome-cpdbas-multicellcall

The endpoint activityoutcome-cpdbas-multicellcall is 540 x active, 580 x inactive and 388 x missing in the training dataset. In each cross-validation 148.5 (of all 1120 non-missing compounds) were predicted with high confidence (>66%), 436.5 with medium confidence (>33%) and 532 with low confidence (<33%).

model confidenceaccuracyaucsensitivityspecificityppvnpvinside-ad
all predictions (ignoring confidence)69.86476.56168.52871.33971.97267.88799.729
predictions with high confidence (>66%)92.00291.70592.90487.03396.55680.87399.443
predictions with medium confidence (>33%)74.42176.62870.70877.83671.19777.24599.568
predictions with low confidence (<33%)60.02862.97757.07562.7661.76758.29499.945

activityoutcome-cpdbas-mouse

The endpoint activityoutcome-cpdbas-mouse is 536 x active, 444 x inactive and 528 x missing in the training dataset. In each cross-validation 82 (of all 980 non-missing compounds) were predicted with high confidence (>66%), 391 with medium confidence (>33%) and 502.5 with low confidence (<33%).

model confidenceaccuracyaucsensitivityspecificityppvnpvinside-ad
all predictions (ignoring confidence)66.22972.28356.27574.67764.90167.17599.537
predictions with high confidence (>66%)83.96979.04165.66294.66985.82483.77898.448
predictions with medium confidence (>33%)74.8476.73661.98684.4374.84474.93799.327
predictions with low confidence (<33%)56.63759.4251.68661.48756.90256.32899.905

activityoutcome-cpdbas-mutagenicity

The endpoint activityoutcome-cpdbas-mutagenicity is 448 x active, 402 x inactive and 658 x missing in the training dataset. In each cross-validation 137.3 (of all 850 non-missing compounds) were predicted with high confidence (>66%), 354.5 with medium confidence (>33%) and 353.8 with low confidence (<33%).

model confidenceaccuracyaucsensitivityspecificityppvnpvinside-ad
all predictions (ignoring confidence)76.44984.07569.09983.38478.9274.81399.486
predictions with high confidence (>66%)91.87392.38691.32692.76294.77388.528100
predictions with medium confidence (>33%)84.26586.05975.3791.04485.88683.23799.101
predictions with low confidence (<33%)62.50868.37453.40271.83263.96962.00199.663

activityoutcome-cpdbas-hamster

The endpoint activityoutcome-cpdbas-hamster is 41 x active, 45 x inactive and 1422 x missing in the training dataset. In each cross-validation 18.5 (of all 86 non-missing compounds) were predicted with high confidence (>66%), 27.2 with medium confidence (>33%) and 35.2 with low confidence (<33%).

model confidenceaccuracyaucsensitivityspecificityppvnpvinside-ad
all predictions (ignoring confidence)69.65480.67675.12966.21972.62469.61194.305
predictions with high confidence (>66%)91.66186.58194.97780.6892.00992.63695.582
predictions with medium confidence (>33%)79.00785.64584.06275.47680.3881.43995.842
predictions with low confidence (<33%)50.34852.16144.64655.10246.63155.01991.912

activityoutcome-cpdbas-dog-primates

The endpoint activityoutcome-cpdbas-dog-primates is 17 x active, 15 x inactive and 1476 x missing in the training dataset. In each cross-validation 0.2 (of all 32 non-missing compounds) were predicted with high confidence (>66%), 5.4 with medium confidence (>33%) and 24 with low confidence (<33%).

model confidenceaccuracyaucsensitivityspecificityppvnpvinside-ad
all predictions (ignoring confidence)50.96659.93553.43752.24454.18252.28692.222
predictions with high confidence (>66%)0?0000100
predictions with medium confidence (>33%)64.0357571.15451.85269.75353.12595
predictions with low confidence (<33%)46.78458.15349.7755048.21451.15791.879