Validation results

This is a validation report for model Rat toxicity prediction with Predictive Clustering Trees.

General information

The model was validated with a 10-times repeated 10-fold cross-validation.

Performance measures

measurefull-namesynonymsdescriptiondetails
accuracycorrect predictions / all predictions
aucarea under (the roc) curveprobability that the classifier ranks a compound with class active higher than with class inactiveto compute auc, the predictions are ranked according to confidences given by the classifier for each prediction, i.e. first the compounds with high confidence for class active, than the compounds the classifier is unsure about, than the compounds with high confidence for class inactive
sensitivityrecall, true positive ratecorrectly predicted active compounds / all compounds that are really active
specificitytrue negative ratecorrectly predicted inactive compounds / all compounds that are really inactive
ppvpositive predictive valueprecision, selectivitycorrectly predicted active compounds / all compounds that are predicted as activeppv is the probability that a active prediction is correct
npvnegative predictive valuecorrectly predicted inactive compounds / all compounds that are predicted as inactiveppv is the probability that a inactive prediction is correct
subset-accuracynumber of test compounds with all endpoints predicted correctly / number of all test compounds
inside-adnumber of test compounds inside the applicability domain / number of all test compounds

Probability that a prediction is correct

When applying the model to an unseen compound, the performance measures ppv and npv give a probability estimate that the prediction is correct. The confidence of the prediction is taken into account to make the probability estimate more accurate. Therefore, ppv and npv have been computed for different confidence levels.

Average performance over all endpoints

The average measures have been computed as the mean of all single-endpoint measures, these measures are so-called 'macro'-measures (Exception: subset-accuracy is computed using all endpoints). Each endpoint is weighted equally.

accuracyaucsensitivityspecificityppvnpvsubset-accuracyinside-ad
0.6070.6160.570.5940.5860.610.2850.989

Single endpoint validation

adrenal-gland-weight-increased

The endpoint adrenal-gland-weight-increased is 54 x active, 42 x inactive and 803 x missing in the training dataset. In each cross-validation 58.7 (of all 96 non-missing compounds) were predicted with high confidence (>66%), 17.1 with medium confidence (>33%) and 19.2 with low confidence (<33%).

model confidenceaccuracyaucsensitivityspecificityppvnpvinside-ad
all predictions (ignoring confidence)54.57654.8736.84170.94648.10659.02699
predictions with high confidence (>66%)56.85553.63928.90776.72142.29763.81499.557
predictions with medium confidence (>33%)53.29249.6864359.40943.61758.05698.56
predictions with low confidence (<33%)54.2443.99454.16753.24160.10345.83397.877

body-weight-decreased

The endpoint body-weight-decreased is 175 x active, 186 x inactive and 538 x missing in the training dataset. In each cross-validation 178.6 (of all 361 non-missing compounds) were predicted with high confidence (>66%), 78.8 with medium confidence (>33%) and 102.6 with low confidence (<33%).

model confidenceaccuracyaucsensitivityspecificityppvnpvinside-ad
all predictions (ignoring confidence)59.33462.40865.30753.50259.75459.39699.721
predictions with high confidence (>66%)64.00564.22870.43157.01864.3663.26699.643
predictions with medium confidence (>33%)58.462.97655.32861.6860.93956.84399.714
predictions with low confidence (<33%)52.96553.7765.03441.79951.5254.94799.792

bone-marrow

The endpoint bone-marrow is 25 x active, 18 x inactive and 856 x missing in the training dataset. In each cross-validation 31.8 (of all 43 non-missing compounds) were predicted with high confidence (>66%), 5.1 with medium confidence (>33%) and 4.9 with low confidence (<33%).

model confidenceaccuracyaucsensitivityspecificityppvnpvinside-ad
all predictions (ignoring confidence)57.34166.19736.90576.05256.13563.35697.878
predictions with high confidence (>66%)62.1264.41432.69484.29459.18465.48198.401
predictions with medium confidence (>33%)39.18958.33342.539.1335.71445.23896.053
predictions with low confidence (<33%)47.9177552.38146.0325047.61994.444

brain

The endpoint brain is 61 x active, 32 x inactive and 806 x missing in the training dataset. In each cross-validation 69.1 (of all 93 non-missing compounds) were predicted with high confidence (>66%), 11.4 with medium confidence (>33%) and 11.3 with low confidence (<33%).

model confidenceaccuracyaucsensitivityspecificityppvnpvinside-ad
all predictions (ignoring confidence)68.77867.55137.91286.37859.67872.61598.812
predictions with high confidence (>66%)73.57367.66730.07694.25670.11175.06199.438
predictions with medium confidence (>33%)59.0252.60451.28267.59355.71463.02198.068
predictions with low confidence (<33%)52.87454.86152.60451.70140.58364.34196.19

clinchem-hepatotox

The endpoint clinchem-hepatotox is 41 x active, 49 x inactive and 809 x missing in the training dataset. In each cross-validation 60.8 (of all 90 non-missing compounds) were predicted with high confidence (>66%), 13.4 with medium confidence (>33%) and 14.7 with low confidence (<33%).

model confidenceaccuracyaucsensitivityspecificityppvnpvinside-ad
all predictions (ignoring confidence)56.99261.87868.44846.71160.93354.53998.527
predictions with high confidence (>66%)61.06561.56472.93748.08267.47852.51998.771
predictions with medium confidence (>33%)52.41246.02355.66749.30652.24452.778100
predictions with low confidence (<33%)45.68939.74455.28539.72234.24262.60296.753

clinchem-nephrotox

The endpoint clinchem-nephrotox is 117 x active, 93 x inactive and 689 x missing in the training dataset. In each cross-validation 112.7 (of all 210 non-missing compounds) were predicted with high confidence (>66%), 45.2 with medium confidence (>33%) and 51 with low confidence (<33%).

model confidenceaccuracyaucsensitivityspecificityppvnpvinside-ad
all predictions (ignoring confidence)54.71955.46538.61767.85448.96358.34899.536
predictions with high confidence (>66%)57.35155.76732.53875.96648.73960.61499.763
predictions with medium confidence (>33%)53.90251.91137.30370.07353.78854.49399.532
predictions with low confidence (<33%)49.349.43454.46343.75847.84351.21499.378

cns

The endpoint cns is 144 x active, 164 x inactive and 591 x missing in the training dataset. In each cross-validation 151.6 (of all 308 non-missing compounds) were predicted with high confidence (>66%), 69 with medium confidence (>33%) and 86.3 with low confidence (<33%).

model confidenceaccuracyaucsensitivityspecificityppvnpvinside-ad
all predictions (ignoring confidence)58.63960.82571.45245.33859.37559.16999.65
predictions with high confidence (>66%)63.98864.09377.61349.17263.15767.67999.393
predictions with medium confidence (>33%)50.90151.35465.89335.54652.00949.00499.857
predictions with low confidence (<33%)55.1855.86867.11844.15858.29455.425100

female-reproductive-organ

The endpoint female-reproductive-organ is 59 x active, 47 x inactive and 793 x missing in the training dataset. In each cross-validation 70.3 (of all 106 non-missing compounds) were predicted with high confidence (>66%), 15.4 with medium confidence (>33%) and 19.4 with low confidence (<33%).

model confidenceaccuracyaucsensitivityspecificityppvnpvinside-ad
all predictions (ignoring confidence)60.51965.35750.168.86757.22962.74499.145
predictions with high confidence (>66%)65.29465.91348.91776.37757.43769.11999.168
predictions with medium confidence (>33%)53.82563.39349.83760.60653.40155.74798.101
predictions with low confidence (<33%)44.93140.25851.76235.05755.87129.48799.412

haematology-anaemia

The endpoint haematology-anaemia is 65 x active, 70 x inactive and 764 x missing in the training dataset. In each cross-validation 88.3 (of all 135 non-missing compounds) were predicted with high confidence (>66%), 21.8 with medium confidence (>33%) and 23.7 with low confidence (<33%).

model confidenceaccuracyaucsensitivityspecificityppvnpvinside-ad
all predictions (ignoring confidence)60.90664.5565.95657.05962.18561.94899.142
predictions with high confidence (>66%)63.72965.33865.04462.44763.94166.55998.994
predictions with medium confidence (>33%)56.56751.67759.84851.55662.83147.52999.128
predictions with low confidence (<33%)54.78759.39972.74639.5256.82957.51699.537

haematology-cellular-hemostasis

The endpoint haematology-cellular-hemostasis is 54 x active, 51 x inactive and 794 x missing in the training dataset. In each cross-validation 67.1 (of all 105 non-missing compounds) were predicted with high confidence (>66%), 17.5 with medium confidence (>33%) and 18.8 with low confidence (<33%).

model confidenceaccuracyaucsensitivityspecificityppvnpvinside-ad
all predictions (ignoring confidence)60.45662.08256.53665.1860.44761.80498.363
predictions with high confidence (>66%)62.49262.88157.65672.25862.94766.1998.295
predictions with medium confidence (>33%)60.80264.69364.54860.45562.67863.70497.542
predictions with low confidence (<33%)53.60153.19855.83347.39646.79758.36499.588

haematology-plasmatic-hemostasis

The endpoint haematology-plasmatic-hemostasis is 33 x active, 44 x inactive and 822 x missing in the training dataset. In each cross-validation 54.8 (of all 77 non-missing compounds) were predicted with high confidence (>66%), 10.7 with medium confidence (>33%) and 10.4 with low confidence (<33%).

model confidenceaccuracyaucsensitivityspecificityppvnpvinside-ad
all predictions (ignoring confidence)65.15867.88278.6450.61669.08263.90198.752
predictions with high confidence (>66%)67.93665.14681.42951.77471.08565.56298.392
predictions with medium confidence (>33%)64.07759.21170.21353.60465.76161.667100
predictions with low confidence (<33%)52.59353.7566.66736.53854.52951.149100

haematopoiesis

The endpoint haematopoiesis is 31 x active, 44 x inactive and 824 x missing in the training dataset. In each cross-validation 49.1 (of all 75 non-missing compounds) were predicted with high confidence (>66%), 12.7 with medium confidence (>33%) and 12.2 with low confidence (<33%).

model confidenceaccuracyaucsensitivityspecificityppvnpvinside-ad
all predictions (ignoring confidence)57.79157.32670.71639.05262.150.13198.805
predictions with high confidence (>66%)6160.41774.36145.33666.73257.22998.766
predictions with medium confidence (>33%)53.94165.7416644.84158.33351.80299.55
predictions with low confidence (<33%)47.34332.557.66728.40949.38334.28699.517

intestine

The endpoint intestine is 28 x active, 43 x inactive and 828 x missing in the training dataset. In each cross-validation 56.1 (of all 71 non-missing compounds) were predicted with high confidence (>66%), 8 with medium confidence (>33%) and 5.7 with low confidence (<33%).

model confidenceaccuracyaucsensitivityspecificityppvnpvinside-ad
all predictions (ignoring confidence)64.77868.59376.89549.97871.99458.62598.228
predictions with high confidence (>66%)67.98267.94680.62349.51274.33259.17498.035
predictions with medium confidence (>33%)55.66763.02156.86359.37563.23553.704100
predictions with low confidence (<33%)48.1885052.38143.18256.17339.13100

kidney

The endpoint kidney is 110 x active, 157 x inactive and 632 x missing in the training dataset. In each cross-validation 166.7 (of all 267 non-missing compounds) were predicted with high confidence (>66%), 48.3 with medium confidence (>33%) and 50.7 with low confidence (<33%).

model confidenceaccuracyaucsensitivityspecificityppvnpvinside-ad
all predictions (ignoring confidence)63.48462.33379.20841.10365.90158.75599.512
predictions with high confidence (>66%)66.97763.4983.57342.35968.13564.69599.16
predictions with medium confidence (>33%)57.96353.87972.24140.29162.9450.892100
predictions with low confidence (<33%)58.19755.11173.85739.4761.96752.418100

kidney-weight-increased

The endpoint kidney-weight-increased is 142 x active, 159 x inactive and 598 x missing in the training dataset. In each cross-validation 151.9 (of all 301 non-missing compounds) were predicted with high confidence (>66%), 65.2 with medium confidence (>33%) and 82.9 with low confidence (<33%).

model confidenceaccuracyaucsensitivityspecificityppvnpvinside-ad
all predictions (ignoring confidence)56.00858.04668.98342.15957.12855.19599.642
predictions with high confidence (>66%)58.80558.64670.82544.73861.32856.7399.51
predictions with medium confidence (>33%)55.72956.34472.34340.4754.15760.69499.888
predictions with low confidence (<33%)51.88953.05563.37640.2653.2652.71999.8

liver

The endpoint liver is 138 x active, 252 x inactive and 509 x missing in the training dataset. In each cross-validation 224.9 (of all 390 non-missing compounds) were predicted with high confidence (>66%), 83 with medium confidence (>33%) and 81.1 with low confidence (<33%).

model confidenceaccuracyaucsensitivityspecificityppvnpvinside-ad
all predictions (ignoring confidence)64.02161.18983.79129.29468.43550.02599.759
predictions with high confidence (>66%)68.90858.78588.95326.55872.49351.5799.671
predictions with medium confidence (>33%)61.51758.34978.25234.66865.19150.6199.757
predictions with low confidence (<33%)52.06252.72370.60430.28656.26145.22100

liver-weight-increased

The endpoint liver-weight-increased is 207 x active, 272 x inactive and 420 x missing in the training dataset. In each cross-validation 232.5 (of all 479 non-missing compounds) were predicted with high confidence (>66%), 116.4 with medium confidence (>33%) and 129.1 with low confidence (<33%).

model confidenceaccuracyaucsensitivityspecificityppvnpvinside-ad
all predictions (ignoring confidence)61.54862.74276.80741.47763.26157.9199.784
predictions with high confidence (>66%)67.04464.61182.72944.83867.99864.70199.727
predictions with medium confidence (>33%)61.24359.1378.96139.93863.58960.24799.9
predictions with low confidence (<33%)53.54853.0766.55436.96755.81749.04199.794

male-accessory-gland

The endpoint male-accessory-gland is 26 x active, 16 x inactive and 857 x missing in the training dataset. In each cross-validation 30.8 (of all 42 non-missing compounds) were predicted with high confidence (>66%), 4.7 with medium confidence (>33%) and 5.5 with low confidence (<33%).

model confidenceaccuracyaucsensitivityspecificityppvnpvinside-ad
all predictions (ignoring confidence)58.37959.1325.18179.76440.60863.3397.566
predictions with high confidence (>66%)62.23261.97220.09189.27246.46565.47598.474
predictions with medium confidence (>33%)41.0267531.57952.08336.84248.077100
predictions with low confidence (<33%)39.2863044.44436.20726.78657.592.222

male-reproductive-organ

The endpoint male-reproductive-organ is 42 x active, 32 x inactive and 825 x missing in the training dataset. In each cross-validation 48.7 (of all 74 non-missing compounds) were predicted with high confidence (>66%), 11 with medium confidence (>33%) and 13.3 with low confidence (<33%).

model confidenceaccuracyaucsensitivityspecificityppvnpvinside-ad
all predictions (ignoring confidence)56.6358.20742.57368.18348.1562.73998.677
predictions with high confidence (>66%)59.6559.23238.19976.5948.65767.62299.344
predictions with medium confidence (>33%)48.52242.04544.7925055.1840.6597.487
predictions with low confidence (<33%)53.68940.87356.88947.05947.64554.92496.711

male-reproductive-organ-sperm

The endpoint male-reproductive-organ-sperm is 34 x active, 23 x inactive and 842 x missing in the training dataset. In each cross-validation 44.2 (of all 57 non-missing compounds) were predicted with high confidence (>66%), 5.4 with medium confidence (>33%) and 6.4 with low confidence (<33%).

model confidenceaccuracyaucsensitivityspecificityppvnpvinside-ad
all predictions (ignoring confidence)62.27860.56935.9581.14753.79467.36298.176
predictions with high confidence (>66%)65.55560.05931.05890.41267.59365.28597.72
predictions with medium confidence (>33%)68.84162.559.3757552.63280.64597.872
predictions with low confidence (<33%)40.9725047.05938.88922.77865.94296.939

male-reproductive-organ-weight-decreased

The endpoint male-reproductive-organ-weight-decreased is 34 x active, 22 x inactive and 843 x missing in the training dataset. In each cross-validation 42.1 (of all 56 non-missing compounds) were predicted with high confidence (>66%), 6.7 with medium confidence (>33%) and 5.9 with low confidence (<33%).

model confidenceaccuracyaucsensitivityspecificityppvnpvinside-ad
all predictions (ignoring confidence)60.13163.42232.6278.23646.11167.45398.055
predictions with high confidence (>66%)66.06457.39126.20884.4546.35271.35298.512
predictions with medium confidence (>33%)47.22277.38133.33363.09546.49148.48597.959
predictions with low confidence (<33%)43.45258.33348.55140.38544.23143.7594.697

male-reproductive-organ-weight-increased

The endpoint male-reproductive-organ-weight-increased is 43 x active, 22 x inactive and 834 x missing in the training dataset. In each cross-validation 49 (of all 65 non-missing compounds) were predicted with high confidence (>66%), 7.6 with medium confidence (>33%) and 7.3 with low confidence (<33%).

model confidenceaccuracyaucsensitivityspecificityppvnpvinside-ad
all predictions (ignoring confidence)64.92859.87822.11988.43547.41467.65398.334
predictions with high confidence (>66%)68.02652.30610.02392.83938.27270.29298.257
predictions with medium confidence (>33%)62.05771.42935.89778.57155.55664.95797.917
predictions with low confidence (<33%)50.5158.3335053.76346.66756.2598.98

rbc

The endpoint rbc is 149 x active, 150 x inactive and 600 x missing in the training dataset. In each cross-validation 154.5 (of all 299 non-missing compounds) were predicted with high confidence (>66%), 70.2 with medium confidence (>33%) and 73.3 with low confidence (<33%).

model confidenceaccuracyaucsensitivityspecificityppvnpvinside-ad
all predictions (ignoring confidence)56.71459.39858.49255.67556.8757.39799.688
predictions with high confidence (>66%)60.14760.52158.5662.57160.34861.89199.665
predictions with medium confidence (>33%)56.32958.15661.04852.61855.86357.13899.833
predictions with low confidence (<33%)50.18949.44157.11341.82951.45547.31699.633

spleen

The endpoint spleen is 103 x active, 105 x inactive and 691 x missing in the training dataset. In each cross-validation 134.6 (of all 208 non-missing compounds) were predicted with high confidence (>66%), 35.3 with medium confidence (>33%) and 37 with low confidence (<33%).

model confidenceaccuracyaucsensitivityspecificityppvnpvinside-ad
all predictions (ignoring confidence)63.95966.21465.03662.50363.26564.73599.498
predictions with high confidence (>66%)66.93967.61462.58470.4464.33568.54999.452
predictions with medium confidence (>33%)57.26859.62769.94443.27263.72655.51399.656
predictions with low confidence (<33%)60.31258.73666.31851.83364.67956.76998.582

thymus

The endpoint thymus is 39 x active, 19 x inactive and 841 x missing in the training dataset. In each cross-validation 50 (of all 58 non-missing compounds) were predicted with high confidence (>66%), 3.5 with medium confidence (>33%) and 3.5 with low confidence (<33%).

model confidenceaccuracyaucsensitivityspecificityppvnpvinside-ad
all predictions (ignoring confidence)64.96562.54621.16189.94250.4968.0998.154
predictions with high confidence (>66%)67.40363.51120.37792.85260.56968.55898.213
predictions with medium confidence (>33%)56.54891.6671076.1916.66767.424100
predictions with low confidence (<33%)48.33366.667505034.37565.62593.75

thymus-weight-decreased

The endpoint thymus-weight-decreased is 49 x active, 36 x inactive and 814 x missing in the training dataset. In each cross-validation 57.8 (of all 85 non-missing compounds) were predicted with high confidence (>66%), 12.4 with medium confidence (>33%) and 14.5 with low confidence (<33%).

model confidenceaccuracyaucsensitivityspecificityppvnpvinside-ad
all predictions (ignoring confidence)54.40256.77536.29469.69845.0761.35699.6
predictions with high confidence (>66%)56.80958.9529.78376.84942.14364.06699.69
predictions with medium confidence (>33%)42.562.28139.45654.38652.84640.42699.265
predictions with low confidence (<33%)52.80649.24256.06151.1750.62957.812100

thyroid-gland

The endpoint thyroid-gland is 18 x active, 45 x inactive and 836 x missing in the training dataset. In each cross-validation 56.7 (of all 63 non-missing compounds) were predicted with high confidence (>66%), 2.7 with medium confidence (>33%) and 2.6 with low confidence (<33%).

model confidenceaccuracyaucsensitivityspecificityppvnpvinside-ad
all predictions (ignoring confidence)69.9575797.7663.0370.71242.85798.524
predictions with high confidence (>66%)72.03355.05398.6051.56972.98833.33398.409
predictions with medium confidence (>33%)58.6962581.2511.11162.525100
predictions with low confidence (<33%)34.12737.57513.33328.94760100

wbc

The endpoint wbc is 90 x active, 113 x inactive and 696 x missing in the training dataset. In each cross-validation 126.7 (of all 203 non-missing compounds) were predicted with high confidence (>66%), 35.6 with medium confidence (>33%) and 39.7 with low confidence (<33%).

model confidenceaccuracyaucsensitivityspecificityppvnpvinside-ad
all predictions (ignoring confidence)61.5863.50170.10551.57463.74359.54399.557
predictions with high confidence (>66%)64.83463.99773.45654.41768.3861.4399.353
predictions with medium confidence (>33%)60.36357.91462.54152.10657.159.675100
predictions with low confidence (<33%)51.2353.48760.24942.82453.3949.085100