Validation results

This is a validation report for model Rat toxicity prediction with Ensemble of Classifier Chains.

General information

The model was validated with a 10-times repeated 10-fold cross-validation.

Performance measures

measurefull-namesynonymsdescriptiondetails
accuracycorrect predictions / all predictions
aucarea under (the roc) curveprobability that the classifier ranks a compound with class active higher than with class inactiveto compute auc, the predictions are ranked according to confidences given by the classifier for each prediction, i.e. first the compounds with high confidence for class active, than the compounds the classifier is unsure about, than the compounds with high confidence for class inactive
sensitivityrecall, true positive ratecorrectly predicted active compounds / all compounds that are really active
specificitytrue negative ratecorrectly predicted inactive compounds / all compounds that are really inactive
ppvpositive predictive valueprecision, selectivitycorrectly predicted active compounds / all compounds that are predicted as activeppv is the probability that a active prediction is correct
npvnegative predictive valuecorrectly predicted inactive compounds / all compounds that are predicted as inactiveppv is the probability that a inactive prediction is correct
subset-accuracynumber of test compounds with all endpoints predicted correctly / number of all test compounds
inside-adnumber of test compounds inside the applicability domain / number of all test compounds

Probability that a prediction is correct

When applying the model to an unseen compound, the performance measures ppv and npv give a probability estimate that the prediction is correct. The confidence of the prediction is taken into account to make the probability estimate more accurate. Therefore, ppv and npv have been computed for different confidence levels.

Average performance over all endpoints

The average measures have been computed as the mean of all single-endpoint measures, these measures are so-called 'macro'-measures (Exception: subset-accuracy is computed using all endpoints). Each endpoint is weighted equally.

accuracyaucsensitivityspecificityppvnpvsubset-accuracyinside-ad
0.6350.6790.6170.6070.6080.6490.3330.964

Single endpoint validation

liver-weight-increased

The endpoint liver-weight-increased is 225 x active, 292 x inactive and 505 x missing in the training dataset. In each cross-validation 48.7 (of all 517 non-missing compounds) were predicted with high confidence (>66%), 177.2 with medium confidence (>33%) and 288.2 with low confidence (<33%).

model confidenceaccuracyaucsensitivityspecificityppvnpvinside-ad
all predictions (ignoring confidence)63.34767.72276.37746.79665.06560.25399.419
predictions with high confidence (>66%)87.04189.51196.8776.6381.2793.889100
predictions with medium confidence (>33%)69.57266.67986.70641.37671.20265.59299.274
predictions with low confidence (<33%)55.52157.03965.41344.39558.11551.79999.384

body-weight-decreased

The endpoint body-weight-decreased is 214 x active, 202 x inactive and 606 x missing in the training dataset. In each cross-validation 49 (of all 416 non-missing compounds) were predicted with high confidence (>66%), 161.9 with medium confidence (>33%) and 202.1 with low confidence (<33%).

model confidenceaccuracyaucsensitivityspecificityppvnpvinside-ad
all predictions (ignoring confidence)66.02271.87564.77967.47265.36967.05999.179
predictions with high confidence (>66%)85.39779.85469.81590.84980.06386.49100
predictions with medium confidence (>33%)71.23873.55470.85873.12874.23370.27598.969
predictions with low confidence (<33%)57.21560.86258.48756.7355.76259.42299.207

liver

The endpoint liver is 146 x active, 259 x inactive and 617 x missing in the training dataset. In each cross-validation 44.3 (of all 405 non-missing compounds) were predicted with high confidence (>66%), 187.1 with medium confidence (>33%) and 170.1 with low confidence (<33%).

model confidenceaccuracyaucsensitivityspecificityppvnpvinside-ad
all predictions (ignoring confidence)65.69563.41188.43125.48667.85256.65699.143
predictions with high confidence (>66%)76.88370.53692.17537.53478.85665.93399.39
predictions with medium confidence (>33%)69.93858.75594.30820.50270.32866.95399.483
predictions with low confidence (<33%)57.42258.73980.33526.9760.40550.21198.776

kidney-weight-increased

The endpoint kidney-weight-increased is 160 x active, 169 x inactive and 693 x missing in the training dataset. In each cross-validation 25.2 (of all 329 non-missing compounds) were predicted with high confidence (>66%), 107.4 with medium confidence (>33%) and 190.2 with low confidence (<33%).

model confidenceaccuracyaucsensitivityspecificityppvnpvinside-ad
all predictions (ignoring confidence)60.85265.61365.38457.0160.98861.55898.054
predictions with high confidence (>66%)88.70287.56787.36687.72580.30392.90197.337
predictions with medium confidence (>33%)63.6868.78370.67854.80865.05561.98797.91
predictions with low confidence (<33%)56.35858.31960.59653.35956.96557.41498.203

cns

The endpoint cns is 151 x active, 171 x inactive and 700 x missing in the training dataset. In each cross-validation 11.1 (of all 322 non-missing compounds) were predicted with high confidence (>66%), 94.6 with medium confidence (>33%) and 214.1 with low confidence (<33%).

model confidenceaccuracyaucsensitivityspecificityppvnpvinside-ad
all predictions (ignoring confidence)63.6569.18572.00855.31464.04764.34799.327
predictions with high confidence (>66%)84.39096.29682.25868.05698.14898.571
predictions with medium confidence (>33%)72.81175.17585.8159.23772.73975.49898.866
predictions with low confidence (<33%)58.5762.48265.33752.26660.33657.74399.606

rbc

The endpoint rbc is 155 x active, 160 x inactive and 707 x missing in the training dataset. In each cross-validation 14.4 (of all 315 non-missing compounds) were predicted with high confidence (>66%), 79.6 with medium confidence (>33%) and 216.7 with low confidence (<33%).

model confidenceaccuracyaucsensitivityspecificityppvnpvinside-ad
all predictions (ignoring confidence)60.32366.44466.01756.0261.22561.22898.547
predictions with high confidence (>66%)89.97992.66785.49496.596.25985.565100
predictions with medium confidence (>33%)71.45477.59977.34367.67472.83774.03698.452
predictions with low confidence (<33%)54.28556.560.37749.51955.37354.51698.483

kidney

The endpoint kidney is 118 x active, 163 x inactive and 741 x missing in the training dataset. In each cross-validation 20.7 (of all 281 non-missing compounds) were predicted with high confidence (>66%), 120.4 with medium confidence (>33%) and 135.8 with low confidence (<33%).

model confidenceaccuracyaucsensitivityspecificityppvnpvinside-ad
all predictions (ignoring confidence)64.89669.69782.19241.61865.75363.48798.513
predictions with high confidence (>66%)71.87988.6582.12463.78365.94580.90999.593
predictions with medium confidence (>33%)74.07270.10189.2246.07476.14768.50898.364
predictions with low confidence (<33%)55.44460.68673.35436.2555.1153.29698.552

clinchem-nephrotox

The endpoint clinchem-nephrotox is 122 x active, 98 x inactive and 802 x missing in the training dataset. In each cross-validation 7.2 (of all 220 non-missing compounds) were predicted with high confidence (>66%), 67.5 with medium confidence (>33%) and 141.5 with low confidence (<33%).

model confidenceaccuracyaucsensitivityspecificityppvnpvinside-ad
all predictions (ignoring confidence)61.90264.05143.65577.45460.09963.51198.17
predictions with high confidence (>66%)85.13151.7865596.49185.71484.146100
predictions with medium confidence (>33%)66.42666.26644.91785.24573.99664.86396.482
predictions with low confidence (<33%)58.29457.74141.08172.40353.98261.00898.883

spleen

The endpoint spleen is 105 x active, 109 x inactive and 808 x missing in the training dataset. In each cross-validation 25.5 (of all 214 non-missing compounds) were predicted with high confidence (>66%), 67.4 with medium confidence (>33%) and 119.2 with low confidence (<33%).

model confidenceaccuracyaucsensitivityspecificityppvnpvinside-ad
all predictions (ignoring confidence)67.47472.84274.03961.51866.98369.57599.052
predictions with high confidence (>66%)82.52284.24773.73490.46189.92579.28100
predictions with medium confidence (>33%)74.85574.65182.52369.15571.61182.8499.8
predictions with low confidence (<33%)61.14563.67471.09651.40362.32962.40498.504

wbc

The endpoint wbc is 95 x active, 118 x inactive and 809 x missing in the training dataset. In each cross-validation 11.7 (of all 213 non-missing compounds) were predicted with high confidence (>66%), 72.4 with medium confidence (>33%) and 125.6 with low confidence (<33%).

model confidenceaccuracyaucsensitivityspecificityppvnpvinside-ad
all predictions (ignoring confidence)61.51965.68975.48945.86262.97761.25998.467
predictions with high confidence (>66%)89.78993.7510069.11887.432100100
predictions with medium confidence (>33%)65.39170.34678.84751.06367.40965.19598.828
predictions with low confidence (<33%)56.89259.03971.70641.60358.02357.67998.254

haematology-anaemia

The endpoint haematology-anaemia is 66 x active, 72 x inactive and 884 x missing in the training dataset. In each cross-validation 5.1 (of all 138 non-missing compounds) were predicted with high confidence (>66%), 41.4 with medium confidence (>33%) and 87.7 with low confidence (<33%).

model confidenceaccuracyaucsensitivityspecificityppvnpvinside-ad
all predictions (ignoring confidence)62.03665.86469.14955.5662.7361.90497.346
predictions with high confidence (>66%)79.58368.7569.5658680.95276.923100
predictions with medium confidence (>33%)70.01972.46783.97458.37768.89379.22296.011
predictions with low confidence (<33%)57.9359.09362.00452.82558.26357.14297.882

haematology-cellular-hemostasis

The endpoint haematology-cellular-hemostasis is 56 x active, 53 x inactive and 913 x missing in the training dataset. In each cross-validation 3.7 (of all 109 non-missing compounds) were predicted with high confidence (>66%), 29.5 with medium confidence (>33%) and 72.1 with low confidence (<33%).

model confidenceaccuracyaucsensitivityspecificityppvnpvinside-ad
all predictions (ignoring confidence)56.32759.73553.88760.85754.759.87396.369
predictions with high confidence (>66%)51.04250446573.33330.70291.429
predictions with medium confidence (>33%)65.33861.34554.91371.63162.80264.91892.972
predictions with low confidence (<33%)54.38554.47153.78254.97550.859.36297.44

female-reproductive-organ

The endpoint female-reproductive-organ is 59 x active, 48 x inactive and 915 x missing in the training dataset. In each cross-validation 13.6 (of all 107 non-missing compounds) were predicted with high confidence (>66%), 32.6 with medium confidence (>33%) and 54.8 with low confidence (<33%).

model confidenceaccuracyaucsensitivityspecificityppvnpvinside-ad
all predictions (ignoring confidence)66.47171.03561.1570.03161.75569.07994.59
predictions with high confidence (>66%)83.87466.6672010010082.991100
predictions with medium confidence (>33%)67.91862.61251.42576.78857.37772.25992.501
predictions with low confidence (<33%)61.27265.28768.21453.9559.2860.45395.055

adrenal-gland-weight-increased

The endpoint adrenal-gland-weight-increased is 59 x active, 44 x inactive and 919 x missing in the training dataset. In each cross-validation 12.6 (of all 103 non-missing compounds) were predicted with high confidence (>66%), 27.9 with medium confidence (>33%) and 58.5 with low confidence (<33%).

model confidenceaccuracyaucsensitivityspecificityppvnpvinside-ad
all predictions (ignoring confidence)56.82164.45641.69169.38551.45261.90996.386
predictions with high confidence (>66%)91.66785.18538.46297.79471.42993.478100
predictions with medium confidence (>33%)62.40167.00540.90380.37359.67367.0194.929
predictions with low confidence (<33%)47.7249.98541.76555.68948.01349.23395.954

brain

The endpoint brain is 63 x active, 33 x inactive and 926 x missing in the training dataset. In each cross-validation 26.4 (of all 96 non-missing compounds) were predicted with high confidence (>66%), 36.3 with medium confidence (>33%) and 30.4 with low confidence (<33%).

model confidenceaccuracyaucsensitivityspecificityppvnpvinside-ad
all predictions (ignoring confidence)75.23772.83349.74887.46966.20978.68996.446
predictions with high confidence (>66%)85.40436.725.71499.05837.586.2298.316
predictions with medium confidence (>33%)77.73175.62860.63186.59573.73781.33792.955
predictions with low confidence (<33%)67.19467.94556.92675.85366.20467.82299.575

clinchem-hepatotox

The endpoint clinchem-hepatotox is 42 x active, 49 x inactive and 931 x missing in the training dataset. In each cross-validation 1.5 (of all 91 non-missing compounds) were predicted with high confidence (>66%), 21.1 with medium confidence (>33%) and 65.3 with low confidence (<33%).

model confidenceaccuracyaucsensitivityspecificityppvnpvinside-ad
all predictions (ignoring confidence)61.84965.4975.63549.13163.00464.89696.468
predictions with high confidence (>66%)95.833?94.44410010075100
predictions with medium confidence (>33%)66.01175.99281.28256.66764.70375.73195.524
predictions with low confidence (<33%)59.80362.03774.17645.82760.75160.76197.019

thymus-weight-decreased

The endpoint thymus-weight-decreased is 52 x active, 36 x inactive and 934 x missing in the training dataset. In each cross-validation 6.5 (of all 88 non-missing compounds) were predicted with high confidence (>66%), 21.2 with medium confidence (>33%) and 56.4 with low confidence (<33%).

model confidenceaccuracyaucsensitivityspecificityppvnpvinside-ad
all predictions (ignoring confidence)48.74554.93623.58769.69634.81556.96995.221
predictions with high confidence (>66%)69.07486.3640100?69.07494.255
predictions with medium confidence (>33%)61.65744.1174.25585.87.40768.49998.052
predictions with low confidence (<33%)41.69945.13727.92558.22338.31346.9394.698

haematology-plasmatic-hemostasis

The endpoint haematology-plasmatic-hemostasis is 33 x active, 46 x inactive and 943 x missing in the training dataset. In each cross-validation 3 (of all 79 non-missing compounds) were predicted with high confidence (>66%), 32.4 with medium confidence (>33%) and 40.2 with low confidence (<33%).

model confidenceaccuracyaucsensitivityspecificityppvnpvinside-ad
all predictions (ignoring confidence)63.38367.45780.16744.60565.79664.30494.96
predictions with high confidence (>66%)71.73966.66710058.33350100100
predictions with medium confidence (>33%)69.10169.79690.88939.34266.97978.78896.854
predictions with low confidence (<33%)58.0463.92171.42343.92366.08248.68993.05

male-reproductive-organ

The endpoint male-reproductive-organ is 43 x active, 35 x inactive and 944 x missing in the training dataset. In each cross-validation 8.9 (of all 78 non-missing compounds) were predicted with high confidence (>66%), 26.2 with medium confidence (>33%) and 40.7 with low confidence (<33%).

model confidenceaccuracyaucsensitivityspecificityppvnpvinside-ad
all predictions (ignoring confidence)56.07265.50147.74862.57847.97961.55297.039
predictions with high confidence (>66%)78.27435.08827.77898.9589076.235100
predictions with medium confidence (>33%)72.06772.59164.48174.49265.57474.69598.333
predictions with low confidence (<33%)41.3642.46943.39439.35635.13346.6596.118

haematopoiesis

The endpoint haematopoiesis is 31 x active, 46 x inactive and 945 x missing in the training dataset. In each cross-validation 10.1 (of all 77 non-missing compounds) were predicted with high confidence (>66%), 21.2 with medium confidence (>33%) and 44 with low confidence (<33%).

model confidenceaccuracyaucsensitivityspecificityppvnpvinside-ad
all predictions (ignoring confidence)69.43278.27985.48747.66170.79668.27997.837
predictions with high confidence (>66%)89.94733.333100089.947?100
predictions with medium confidence (>33%)76.9473.68493.16236.45877.89273.333100
predictions with low confidence (<33%)59.55367.96871.73650.24654.72766.49296.516

intestine

The endpoint intestine is 29 x active, 44 x inactive and 949 x missing in the training dataset. In each cross-validation 14.3 (of all 73 non-missing compounds) were predicted with high confidence (>66%), 22.2 with medium confidence (>33%) and 33.5 with low confidence (<33%).

model confidenceaccuracyaucsensitivityspecificityppvnpvinside-ad
all predictions (ignoring confidence)66.30473.04681.01946.41669.37662.88595.759
predictions with high confidence (>66%)87.87191.82710068.84182.80210094.283
predictions with medium confidence (>33%)72.34473.95892.09438.25171.5179.16799.176
predictions with low confidence (<33%)50.95756.57760.57542.77156.3347.29494.79

male-reproductive-organ-weight-increased

The endpoint male-reproductive-organ-weight-increased is 44 x active, 23 x inactive and 955 x missing in the training dataset. In each cross-validation 5.3 (of all 67 non-missing compounds) were predicted with high confidence (>66%), 32 with medium confidence (>33%) and 27 with low confidence (<33%).

model confidenceaccuracyaucsensitivityspecificityppvnpvinside-ad
all predictions (ignoring confidence)69.48675.0325.30193.28269.04870.24895.249
predictions with high confidence (>66%)97.7781000100?97.778100
predictions with medium confidence (>33%)79.2672.95520.76310010078.22296.667
predictions with low confidence (<33%)56.22956.03626.4498157.57655.29190.539

thyroid-gland

The endpoint thyroid-gland is 20 x active, 47 x inactive and 955 x missing in the training dataset. In each cross-validation 12.5 (of all 67 non-missing compounds) were predicted with high confidence (>66%), 25.5 with medium confidence (>33%) and 26 with low confidence (<33%).

model confidenceaccuracyaucsensitivityspecificityppvnpvinside-ad
all predictions (ignoring confidence)68.44962.99890.95816.48172.50538.69395.642
predictions with high confidence (>66%)84.30344.69798.065085.884088.843
predictions with medium confidence (>33%)69.14565.10497.18910.34569.56158.33397.242
predictions with low confidence (<33%)61.66455.54583.1126.98468.33337.60295.709

male-reproductive-organ-sperm

The endpoint male-reproductive-organ-sperm is 36 x active, 25 x inactive and 961 x missing in the training dataset. In each cross-validation 8.5 (of all 61 non-missing compounds) were predicted with high confidence (>66%), 19.6 with medium confidence (>33%) and 30.4 with low confidence (<33%).

model confidenceaccuracyaucsensitivityspecificityppvnpvinside-ad
all predictions (ignoring confidence)68.2775.43251.94879.60860.80673.62895.268
predictions with high confidence (>66%)88.7190.90954.54510010085.714100
predictions with medium confidence (>33%)78.63189.02455.73899.26597.573.43397.94
predictions with low confidence (<33%)55.09955.49643.25363.87640.58668.07892.013

male-reproductive-organ-weight-decreased

The endpoint male-reproductive-organ-weight-decreased is 37 x active, 24 x inactive and 961 x missing in the training dataset. In each cross-validation 10.3 (of all 61 non-missing compounds) were predicted with high confidence (>66%), 18.6 with medium confidence (>33%) and 29.1 with low confidence (<33%).

model confidenceaccuracyaucsensitivityspecificityppvnpvinside-ad
all predictions (ignoring confidence)66.14473.63453.07475.82555.76575.43395.558
predictions with high confidence (>66%)91.52475097.304094.05899.286
predictions with medium confidence (>33%)68.49969.59850.79476.29644.81580.16494.767
predictions with low confidence (<33%)58.74959.67756.47761.14762.59357.37295.446

thymus

The endpoint thymus is 39 x active, 20 x inactive and 963 x missing in the training dataset. In each cross-validation 10 (of all 59 non-missing compounds) were predicted with high confidence (>66%), 15.5 with medium confidence (>33%) and 28.8 with low confidence (<33%).

model confidenceaccuracyaucsensitivityspecificityppvnpvinside-ad
all predictions (ignoring confidence)64.89166.19824.53885.84947.63970.12492.271
predictions with high confidence (>66%)87.392500100?87.39299.274
predictions with medium confidence (>33%)70.15973.0168.97410010068.00895.543
predictions with low confidence (<33%)54.57648.81834.01872.07344.10961.57288.314

bone-marrow

The endpoint bone-marrow is 25 x active, 19 x inactive and 978 x missing in the training dataset. In each cross-validation 1.4 (of all 44 non-missing compounds) were predicted with high confidence (>66%), 13.6 with medium confidence (>33%) and 25.2 with low confidence (<33%).

model confidenceaccuracyaucsensitivityspecificityppvnpvinside-ad
all predictions (ignoring confidence)59.47566.79445.23371.25653.565.24590.661
predictions with high confidence (>66%)19.23100100?19.23196.154
predictions with medium confidence (>33%)76.6271.95544.14486.11162.82180.22685.214
predictions with low confidence (<33%)51.48763.8345.83360.29950.10155.61492.615

male-accessory-gland

The endpoint male-accessory-gland is 27 x active, 17 x inactive and 978 x missing in the training dataset. In each cross-validation 6.2 (of all 44 non-missing compounds) were predicted with high confidence (>66%), 12.7 with medium confidence (>33%) and 19.9 with low confidence (<33%).

model confidenceaccuracyaucsensitivityspecificityppvnpvinside-ad
all predictions (ignoring confidence)62.77268.33737.17176.56140.9972.91288.432
predictions with high confidence (>66%)87.31937.59.09110010087.037100
predictions with medium confidence (>33%)72.5957142.88691.9757571.48498.81
predictions with low confidence (<33%)47.11847.34434.75254.92927.09163.53579.979