Benchmark

Fiducio against netcal, torch-uncertainty and dirichletcal on synthetic voxels. Generated by benchmarks/run.py report; do not edit by hand.

Versions: dirichletcal 0.5.3, fiducio 0.4.0, netcal 1.4.0, torch-uncertainty 0.13.0, torch 2.15.0+cpu. Hardware: Intel(R) Xeon(R) W-2225 CPU @ 4.10GHz, 8 cores, 4 torch threads.

Reading these results

• Synthetic data only. Labels are drawn from softmax(z); the model sees a_i * z (plus a class bias in bias), with a_i varying per case. A scenario with a known oracle lets the temperature scalers be checked against a Bayes oracle (a lower bound); with jitter=0, TS recovers T* = a up to finite-sample error. It says nothing about nnU-Net logits.
• One metric implementation. NLL, ECE and ACE come from fiducio.metrics (15 uniform bins, pooled over voxels) for every method. ACE here is the unweighted mean over non-empty bins, not the adaptive ACE of other work. Cross-check against netcal's ECE (4 classes only: its binary ECE measures P(class 1), a different quantity) and scikit-learn's log loss on the same probabilities: max |ΔECE| = 9.33e-06, max |ΔNLL| = 3.08e-05.
• Differences are paired: same seed, same cases. The interval is 1.96 standard errors over seeds (a normal approximation; with 10 seeds the exact t multiplier is 2.26); each seed redraws all cases, so it covers case-to-case variance.
• Input contract. Fiducio and torch-uncertainty receive logits; netcal and dirichletcal receive softmax(z) in float64. dirichletcal runs on JAX, which computes in float32 unless x64 is enabled (probabilities differ by a few 1e-6 between the two modes).
• netcal pitfall. With float32 probabilities TemperatureScaling.fit leaves T = 1 without a warning; the adapter casts to float64. Its T multiplies the logits, Fiducio's divides them.
• Defaults. Each library runs with its default optimiser budget; no per-library tuning.

Calibration

The figure shows NLL excess and ECE; the tables also report NLL, ECE and ECE of the last class (the rare one in rare), mean ± 95% CI over seeds. Δ NLL is paired against the matching Fiducio method.

over

C	method	NLL	ECE	ECE last class	Δ NLL vs Fiducio
2	oracle	0.4482 ± 0.0008	0.0025 ± 0.0005	0.0034 ± 0.0004
2	identity	0.6047 ± 0.0084	0.1142 ± 0.0022	0.1142 ± 0.0022
2	fiducio TS	0.4513 ± 0.0011	0.0093 ± 0.0024	0.0093 ± 0.0024
2	fiducio VS	0.4513 ± 0.0011	0.0093 ± 0.0025	0.0093 ± 0.0024
2	fiducio MS	0.4513 ± 0.0011	0.0093 ± 0.0025	0.0093 ± 0.0024
2	fiducio DC	0.4513 ± 0.0011	0.0092 ± 0.0025	0.0093 ± 0.0024
2	netcal TS	0.4513 ± 0.0011	0.0092 ± 0.0024	0.0093 ± 0.0024	0.00002 ± 0.00003
2	torch-uncertainty TS	0.4513 ± 0.0011	0.0093 ± 0.0024	0.0093 ± 0.0024	0.00001 ± 0.00001
2	torch-uncertainty VS	0.4513 ± 0.0011	0.0092 ± 0.0025	0.0093 ± 0.0024	0.00001 ± 0.00002
2	torch-uncertainty MS	0.4513 ± 0.0011	0.0094 ± 0.0025	0.0094 ± 0.0024	0.00001 ± 0.00001
2	dirichletcal DC	0.4513 ± 0.0011	0.0092 ± 0.0025	0.0093 ± 0.0024	0.00000 ± 0.00000
4	oracle	0.9250 ± 0.0013	0.0036 ± 0.0006	0.0036 ± 0.0004
4	identity	1.2523 ± 0.0174	0.2000 ± 0.0035	0.1012 ± 0.0021
4	fiducio TS	0.9315 ± 0.0022	0.0114 ± 0.0033	0.0066 ± 0.0016
4	fiducio VS	0.9315 ± 0.0022	0.0114 ± 0.0033	0.0067 ± 0.0017
4	fiducio MS	0.9315 ± 0.0022	0.0116 ± 0.0033	0.0068 ± 0.0017
4	fiducio DC	0.9315 ± 0.0022	0.0115 ± 0.0033	0.0068 ± 0.0017
4	netcal TS	0.9315 ± 0.0022	0.0114 ± 0.0033	0.0066 ± 0.0016	0.00000 ± 0.00000
4	torch-uncertainty TS	0.9315 ± 0.0022	0.0115 ± 0.0033	0.0066 ± 0.0016	0.00001 ± 0.00002
4	torch-uncertainty VS	0.9315 ± 0.0022	0.0114 ± 0.0033	0.0068 ± 0.0017	0.00001 ± 0.00002
4	torch-uncertainty MS	0.9316 ± 0.0022	0.0116 ± 0.0033	0.0068 ± 0.0017	0.00001 ± 0.00001
4	dirichletcal DC	0.9315 ± 0.0022	0.0115 ± 0.0033	0.0068 ± 0.0017	0.00000 ± 0.00001

under

C	method	NLL	ECE	ECE last class	Δ NLL vs Fiducio
2	oracle	0.4482 ± 0.0008	0.0025 ± 0.0005	0.0034 ± 0.0004
2	identity	0.4897 ± 0.0024	0.1009 ± 0.0034	0.0996 ± 0.0033
2	fiducio TS	0.4513 ± 0.0011	0.0092 ± 0.0024	0.0093 ± 0.0024
2	fiducio VS	0.4513 ± 0.0011	0.0093 ± 0.0025	0.0093 ± 0.0024
2	fiducio MS	0.4513 ± 0.0011	0.0093 ± 0.0025	0.0093 ± 0.0024
2	fiducio DC	0.4513 ± 0.0011	0.0093 ± 0.0024	0.0093 ± 0.0024
2	netcal TS	0.4513 ± 0.0011	0.0092 ± 0.0024	0.0093 ± 0.0024	0.00000 ± 0.00000
2	torch-uncertainty TS	0.4513 ± 0.0011	0.0092 ± 0.0024	0.0093 ± 0.0024	-0.00001 ± 0.00001
2	torch-uncertainty VS	0.4513 ± 0.0011	0.0092 ± 0.0024	0.0092 ± 0.0023	-0.00001 ± 0.00001
2	torch-uncertainty MS	0.4513 ± 0.0011	0.0093 ± 0.0024	0.0093 ± 0.0024	-0.00001 ± 0.00001
2	dirichletcal DC	0.4513 ± 0.0011	0.0093 ± 0.0024	0.0093 ± 0.0024	0.00000 ± 0.00000
4	oracle	0.9250 ± 0.0013	0.0036 ± 0.0006	0.0036 ± 0.0004
4	identity	1.0123 ± 0.0036	0.1569 ± 0.0040	0.0793 ± 0.0022
4	fiducio TS	0.9315 ± 0.0022	0.0114 ± 0.0033	0.0066 ± 0.0016
4	fiducio VS	0.9315 ± 0.0022	0.0114 ± 0.0033	0.0067 ± 0.0017
4	fiducio MS	0.9315 ± 0.0022	0.0116 ± 0.0033	0.0068 ± 0.0017
4	fiducio DC	0.9315 ± 0.0022	0.0115 ± 0.0034	0.0067 ± 0.0017
4	netcal TS	0.9315 ± 0.0022	0.0114 ± 0.0033	0.0066 ± 0.0016	0.00000 ± 0.00000
4	torch-uncertainty TS	0.9314 ± 0.0022	0.0114 ± 0.0034	0.0065 ± 0.0017	-0.00001 ± 0.00001
4	torch-uncertainty VS	0.9315 ± 0.0022	0.0114 ± 0.0033	0.0068 ± 0.0017	-0.00001 ± 0.00002
4	torch-uncertainty MS	0.9315 ± 0.0022	0.0115 ± 0.0034	0.0067 ± 0.0017	-0.00001 ± 0.00002
4	dirichletcal DC	0.9315 ± 0.0022	0.0115 ± 0.0033	0.0067 ± 0.0017	0.00000 ± 0.00000

bias

C	method	NLL	ECE	ECE last class	Δ NLL vs Fiducio
2	oracle	0.4482 ± 0.0008	0.0025 ± 0.0005	0.0034 ± 0.0004
2	identity	0.6135 ± 0.0083	0.1165 ± 0.0021	0.1177 ± 0.0020
2	fiducio TS	0.4540 ± 0.0011	0.0088 ± 0.0025	0.0293 ± 0.0012
2	fiducio VS	0.4513 ± 0.0011	0.0093 ± 0.0025	0.0093 ± 0.0024
2	fiducio MS	0.4513 ± 0.0011	0.0093 ± 0.0025	0.0093 ± 0.0024
2	fiducio DC	0.4513 ± 0.0011	0.0092 ± 0.0025	0.0093 ± 0.0024
2	netcal TS	0.4540 ± 0.0011	0.0088 ± 0.0024	0.0293 ± 0.0012	0.00003 ± 0.00003
2	torch-uncertainty TS	0.4540 ± 0.0011	0.0088 ± 0.0025	0.0294 ± 0.0012	0.00001 ± 0.00001
2	torch-uncertainty VS	0.4540 ± 0.0011	0.0088 ± 0.0025	0.0292 ± 0.0012	0.00271 ± 0.00019
2	torch-uncertainty MS	0.4513 ± 0.0011	0.0094 ± 0.0025	0.0093 ± 0.0024	0.00001 ± 0.00001
2	dirichletcal DC	0.4513 ± 0.0011	0.0092 ± 0.0025	0.0093 ± 0.0024	0.00000 ± 0.00000
4	oracle	0.9250 ± 0.0013	0.0036 ± 0.0006	0.0036 ± 0.0004
4	identity	1.3064 ± 0.0167	0.2118 ± 0.0033	0.1234 ± 0.0014
4	fiducio TS	0.9473 ± 0.0021	0.0107 ± 0.0031	0.0514 ± 0.0010
4	fiducio VS	0.9315 ± 0.0022	0.0114 ± 0.0033	0.0067 ± 0.0017
4	fiducio MS	0.9315 ± 0.0022	0.0116 ± 0.0033	0.0068 ± 0.0017
4	fiducio DC	0.9315 ± 0.0022	0.0115 ± 0.0033	0.0067 ± 0.0017
4	netcal TS	0.9473 ± 0.0021	0.0106 ± 0.0030	0.0514 ± 0.0010	0.00000 ± 0.00000
4	torch-uncertainty TS	0.9473 ± 0.0021	0.0107 ± 0.0030	0.0514 ± 0.0010	0.00001 ± 0.00002
4	torch-uncertainty VS	0.9426 ± 0.0022	0.0102 ± 0.0036	0.0334 ± 0.0012	0.01114 ± 0.00058
4	torch-uncertainty MS	0.9315 ± 0.0022	0.0115 ± 0.0033	0.0068 ± 0.0017	0.00000 ± 0.00001
4	dirichletcal DC	0.9315 ± 0.0022	0.0115 ± 0.0033	0.0067 ± 0.0017	0.00000 ± 0.00000

rare

C	method	NLL	ECE	ECE last class	Δ NLL vs Fiducio
2	oracle	0.0404 ± 0.0005	0.0007 ± 0.0001	0.0007 ± 0.0001
2	identity	0.0756 ± 0.0021	0.0091 ± 0.0002	0.0091 ± 0.0002
2	fiducio TS	0.0417 ± 0.0004	0.0016 ± 0.0004	0.0016 ± 0.0004
2	fiducio VS	0.0414 ± 0.0005	0.0014 ± 0.0003	0.0014 ± 0.0003
2	fiducio MS	0.0414 ± 0.0005	0.0014 ± 0.0003	0.0014 ± 0.0003
2	fiducio DC	0.0416 ± 0.0005	0.0016 ± 0.0004	0.0017 ± 0.0004
2	netcal TS	0.0417 ± 0.0004	0.0016 ± 0.0004	0.0016 ± 0.0004	0.00000 ± 0.00000
2	torch-uncertainty TS	0.0417 ± 0.0004	0.0016 ± 0.0004	0.0016 ± 0.0004	0.00000 ± 0.00000
2	torch-uncertainty VS	0.0417 ± 0.0004	0.0016 ± 0.0004	0.0017 ± 0.0004	0.00022 ± 0.00006
2	torch-uncertainty MS	0.0414 ± 0.0005	0.0014 ± 0.0003	0.0014 ± 0.0003	0.00000 ± 0.00000
2	dirichletcal DC	0.0416 ± 0.0005	0.0016 ± 0.0004	0.0017 ± 0.0004	0.00000 ± 0.00000
4	oracle	0.7666 ± 0.0017	0.0035 ± 0.0005	0.0007 ± 0.0001
4	identity	1.0527 ± 0.0132	0.1729 ± 0.0027	0.0118 ± 0.0002
4	fiducio TS	0.7727 ± 0.0024	0.0102 ± 0.0029	0.0012 ± 0.0006
4	fiducio VS	0.7726 ± 0.0023	0.0105 ± 0.0029	0.0012 ± 0.0004
4	fiducio MS	0.7727 ± 0.0023	0.0104 ± 0.0029	0.0012 ± 0.0004
4	fiducio DC	0.7727 ± 0.0023	0.0101 ± 0.0030	0.0013 ± 0.0004
4	netcal TS	0.7727 ± 0.0024	0.0103 ± 0.0029	0.0012 ± 0.0006	0.00000 ± 0.00000
4	torch-uncertainty TS	0.7727 ± 0.0024	0.0103 ± 0.0029	0.0012 ± 0.0006	0.00001 ± 0.00001
4	torch-uncertainty VS	0.7727 ± 0.0024	0.0103 ± 0.0029	0.0012 ± 0.0005	0.00004 ± 0.00004
4	torch-uncertainty MS	0.7727 ± 0.0023	0.0105 ± 0.0029	0.0013 ± 0.0005	0.00003 ± 0.00002
4	dirichletcal DC	0.7727 ± 0.0023	0.0101 ± 0.0030	0.0013 ± 0.0004	0.00000 ± 0.00000

Δ NLL compares each method with the matching Fiducio calibrator. In bias, Fiducio's vector scaling fits a per-class bias (regularised by mu_reg) while torch-uncertainty's VectorScaler has none, so its Δ NLL in that scenario measures capacity, not implementation quality.

Downstream (Fiducio only)

Same paired seeds as the calibration axis, 4 classes, scenarios with a per-class bias (bias) and a rare class (rare), and identity as the reference decision. None of the competitors' tested methods is class-conditional, so this axis is Fiducio only.

Per-class ECE uses the class-specific metric (class_index), and the volume bias of a class is its predicted voxel share minus its label share (signed: negative means under-segmentation). Δ Dice is paired against identity over seeds. argmax changed is the share of test voxels whose argmax differs from the uncalibrated one.

bias

method	NLL	ECE	ECE c0	ECE c1	ECE c2	ECE c3	Dice	Δ Dice	argmax changed	bias c0	bias c1	bias c2	bias c3
identity	1.3064 ± 0.0167	0.2118 ± 0.0033	0.1013 ± 0.0018	0.1063 ± 0.0020	0.1063 ± 0.0013	0.1234 ± 0.0014	0.6047 ± 0.0010	0.0000 ± 0.0000	0.0000 ± 0.0000	-0.0255 ± 0.0009	0.0189 ± 0.0009	-0.0654 ± 0.0013	0.0721 ± 0.0020
fiducio TS	0.9473 ± 0.0021	0.0107 ± 0.0031	0.0186 ± 0.0012	0.0148 ± 0.0014	0.0476 ± 0.0010	0.0514 ± 0.0010	0.6047 ± 0.0010	0.0000 ± 0.0000	0.0000 ± 0.0000	-0.0255 ± 0.0009	0.0189 ± 0.0009	-0.0654 ± 0.0013	0.0721 ± 0.0020
fiducio VS	0.9315 ± 0.0022	0.0114 ± 0.0033	0.0062 ± 0.0015	0.0068 ± 0.0017	0.0062 ± 0.0015	0.0067 ± 0.0017	0.6156 ± 0.0007	0.0109 ± 0.0007	0.1128 ± 0.0025	0.0004 ± 0.0011	-0.0008 ± 0.0011	0.0005 ± 0.0012	-0.0001 ± 0.0013
fiducio MS	0.9315 ± 0.0022	0.0116 ± 0.0033	0.0061 ± 0.0014	0.0069 ± 0.0017	0.0061 ± 0.0015	0.0068 ± 0.0017	0.6156 ± 0.0007	0.0109 ± 0.0007	0.1125 ± 0.0026	0.0006 ± 0.0011	-0.0007 ± 0.0011	0.0003 ± 0.0013	-0.0002 ± 0.0014
fiducio DC	0.9315 ± 0.0022	0.0115 ± 0.0033	0.0061 ± 0.0014	0.0067 ± 0.0017	0.0061 ± 0.0015	0.0067 ± 0.0017	0.6156 ± 0.0007	0.0108 ± 0.0007	0.1132 ± 0.0028	0.0009 ± 0.0013	-0.0005 ± 0.0011	0.0005 ± 0.0016	-0.0008 ± 0.0013
fiducio CMSap	0.9431 ± 0.0030	0.0320 ± 0.0061	0.0172 ± 0.0015	0.0170 ± 0.0022	0.0353 ± 0.0009	0.0400 ± 0.0012	0.6047 ± 0.0010	0.0000 ± 0.0000	0.0000 ± 0.0000	-0.0255 ± 0.0009	0.0189 ± 0.0009	-0.0654 ± 0.0013	0.0721 ± 0.0020
fiducio CMSop	0.9466 ± 0.0028	0.0292 ± 0.0060	0.0204 ± 0.0013	0.0148 ± 0.0015	0.0443 ± 0.0011	0.0444 ± 0.0006	0.6047 ± 0.0010	0.0000 ± 0.0000	0.0000 ± 0.0000	-0.0255 ± 0.0009	0.0189 ± 0.0009	-0.0654 ± 0.0013	0.0721 ± 0.0020

rare

method	NLL	ECE	ECE c0	ECE c1	ECE c2	ECE c3	Dice	Δ Dice	argmax changed	bias c0	bias c1	bias c2	bias c3
identity	1.0527 ± 0.0132	0.1729 ± 0.0027	0.1119 ± 0.0019	0.1121 ± 0.0020	0.1121 ± 0.0015	0.0118 ± 0.0002	0.5394 ± 0.0016	0.0000 ± 0.0000	0.0000 ± 0.0000	0.0047 ± 0.0008	0.0035 ± 0.0011	0.0039 ± 0.0009	-0.0122 ± 0.0003
fiducio TS	0.7727 ± 0.0024	0.0102 ± 0.0029	0.0070 ± 0.0015	0.0072 ± 0.0019	0.0069 ± 0.0014	0.0012 ± 0.0006	0.5394 ± 0.0016	0.0000 ± 0.0000	0.0000 ± 0.0000	0.0047 ± 0.0008	0.0035 ± 0.0011	0.0039 ± 0.0009	-0.0122 ± 0.0003
fiducio VS	0.7726 ± 0.0023	0.0105 ± 0.0029	0.0073 ± 0.0015	0.0073 ± 0.0020	0.0070 ± 0.0014	0.0012 ± 0.0004	0.5346 ± 0.0021	-0.0048 ± 0.0012	0.0032 ± 0.0006	0.0048 ± 0.0011	0.0038 ± 0.0014	0.0041 ± 0.0012	-0.0127 ± 0.0003
fiducio MS	0.7727 ± 0.0023	0.0104 ± 0.0029	0.0073 ± 0.0016	0.0074 ± 0.0020	0.0070 ± 0.0015	0.0012 ± 0.0004	0.5359 ± 0.0022	-0.0035 ± 0.0013	0.0049 ± 0.0007	0.0048 ± 0.0012	0.0037 ± 0.0013	0.0040 ± 0.0013	-0.0125 ± 0.0003
fiducio DC	0.7727 ± 0.0023	0.0101 ± 0.0030	0.0072 ± 0.0015	0.0073 ± 0.0019	0.0069 ± 0.0014	0.0013 ± 0.0004	0.5359 ± 0.0020	-0.0035 ± 0.0011	0.0045 ± 0.0008	0.0050 ± 0.0014	0.0037 ± 0.0015	0.0039 ± 0.0015	-0.0126 ± 0.0003
fiducio CMSap	0.7742 ± 0.0025	0.0165 ± 0.0036	0.0114 ± 0.0025	0.0114 ± 0.0024	0.0112 ± 0.0022	0.0015 ± 0.0003	0.5394 ± 0.0016	0.0000 ± 0.0000	0.0000 ± 0.0000	0.0047 ± 0.0008	0.0035 ± 0.0011	0.0039 ± 0.0009	-0.0122 ± 0.0003
fiducio CMSop	0.7742 ± 0.0025	0.0168 ± 0.0036	0.0114 ± 0.0026	0.0115 ± 0.0024	0.0113 ± 0.0022	0.0016 ± 0.0003	0.5394 ± 0.0016	0.0000 ± 0.0000	0.0000 ± 0.0000	0.0047 ± 0.0008	0.0035 ± 0.0011	0.0039 ± 0.0009	-0.0122 ± 0.0003

TS, CMSap and CMSop preserve the argmax in exact arithmetic (CMSop the full class order), so their Dice and argmax change are zero by construction, up to floating-point ties; VS, MS and DC can change the decision, so Dice is informative for them. In these runs the preserving methods improve ECE but keep identity's volume bias, while VS, MS and DC change the decision and reduce it; in rare, the decision-changing methods also lower Dice slightly.

Robustness

Every (case, method, seed) is a row in results/robustness.csv and appears below; timeout, oom and error are results, not omissions. Status is counted over seeds; NLL and ECE are medians over the successful seeds. Each fit-and-transform is bounded by an in-process SIGALRM deadline (not a fresh process, unlike the scaling axis), so a timeout is recorded per row. An error is the adapter/library combination, not necessarily a library defect: torch-uncertainty's matrix scaler does not accept float64 through this adapter, and dirichletcal rejects a missing class.

saturated

logits scaled by 10⁴ so that softmax underflows to exact 0/1; netcal and dirichletcal receive those probabilities, Fiducio and torch-uncertainty the logits, and the (probs) variants exercise Fiducio's log(p) path.

method	status	NLL (median)	ECE (median)
identity	ok ×10	10.6982	0.3872
fiducio TS	ok ×10	10.6717	0.3871
fiducio VS	ok ×10	1.8475	0.3074
fiducio MS	ok ×10	1.6367	0.2179
fiducio DC	ok ×10	2.4269	0.3319
netcal TS	ok ×10	1.0930	0.0068
torch-uncertainty TS	ok ×10	10.7830	0.3902
torch-uncertainty VS	error ×9, ok ×1	19.9220	0.7210
torch-uncertainty MS	ok ×10	0.9384	0.0395
dirichletcal DC	ok ×10	10.6167	0.3845
fiducio VS (probs)	ok ×10	1.0939	0.0079
fiducio DC (probs)	ok ×10	1.0963	0.0078
netcal TS (raw dtype)	ok ×10	6.1737	0.3872

tiny

one case and 8 voxels per split.

method	status	NLL (median)	ECE (median)
identity	ok ×10	1.2124	0.3250
fiducio TS	ok ×10	0.9602	0.2547
fiducio VS	ok ×10	4.4635	0.4283
fiducio MS	ok ×10	5.8130	0.5474
fiducio DC	ok ×10	7.0441	0.5228
netcal TS	ok ×10	0.9668	0.2666
torch-uncertainty TS	ok ×10	0.9665	0.2665
torch-uncertainty VS	ok ×10	1.2940	0.3113
torch-uncertainty MS	ok ×10	12.0347	0.6171
dirichletcal DC	error ×5, ok ×5	15.7762	0.6766
fiducio VS (probs)	ok ×10	4.4798	0.4511
fiducio DC (probs)	ok ×10	7.0441	0.5228
netcal TS (raw dtype)	ok ×10	1.2124	0.3250

class_absent

class 3 removed from the calibration split; the test split keeps it.

method	status	NLL (median)	ECE (median)
identity	ok ×10	1.2409	0.2026
fiducio TS	ok ×10	0.9371	0.0407
fiducio VS	ok ×10	2.2343	0.1797
fiducio MS	ok ×10	2.2846	0.1820
fiducio DC	ok ×10	5.0782	0.1822
netcal TS	ok ×10	0.9371	0.0407
torch-uncertainty TS	ok ×10	0.9371	0.0407
torch-uncertainty VS	ok ×10	0.9930	0.0614
torch-uncertainty MS	ok ×10	4.0430	0.1822
dirichletcal DC	error ×10
fiducio VS (probs)	ok ×10	2.6578	0.1773
fiducio DC (probs)	ok ×10	5.0777	0.1822
netcal TS (raw dtype)	ok ×10	1.2409	0.2026

float32

inputs in float32; netcal TS (raw dtype) skips the adapter's float64 cast.

method	status	NLL (median)	ECE (median)
identity	ok ×10	1.2409	0.2026
fiducio TS	ok ×10	0.9363	0.0395
fiducio VS	ok ×10	0.9367	0.0376
fiducio MS	ok ×10	0.9387	0.0444
fiducio DC	ok ×10	0.9380	0.0379
netcal TS	ok ×10	0.9363	0.0395
torch-uncertainty TS	ok ×10	0.9363	0.0396
torch-uncertainty VS	ok ×10	0.9366	0.0401
torch-uncertainty MS	ok ×10	0.9386	0.0443
dirichletcal DC	ok ×10	0.9380	0.0382
fiducio VS (probs)	ok ×10	0.9369	0.0386
fiducio DC (probs)	ok ×10	0.9380	0.0382
netcal TS (raw dtype)	ok ×10	1.2409	0.2026

float64

inputs in float64.

method	status	NLL (median)	ECE (median)
identity	ok ×10	1.2409	0.2026
fiducio TS	ok ×10	0.9363	0.0395
fiducio VS	ok ×10	0.9367	0.0376
fiducio MS	ok ×10	0.9387	0.0444
fiducio DC	ok ×10	0.9380	0.0379
netcal TS	ok ×10	0.9363	0.0395
torch-uncertainty TS	ok ×10	0.9363	0.0395
torch-uncertainty VS	ok ×10	0.9367	0.0400
torch-uncertainty MS	error ×10
dirichletcal DC	ok ×10	0.9380	0.0382
fiducio VS (probs)	ok ×10	0.9369	0.0387
fiducio DC (probs)	ok ×10	0.9380	0.0379
netcal TS (raw dtype)	ok ×10	0.9363	0.0395

Status totals over every robustness row: error 34, ok 616.

With raw float32 probabilities netcal leaves T = 1: its median NLL is 1.2409, the same as identity (1.2409), against 0.9363 in float64. The main adapter avoids this by casting to float64.

Scaling

Scenario over, 4 classes, one process per row.

Fit times reflect each library's default optimiser budget: Fiducio's temperature scaling uses Adam with max_iter=200, which dominates its fit time at 1M voxels, while netcal's default is much cheaper.

The table reports medians over 3 repetitions; the figures show no spread.

N	method	status	fit (s)	transform (s)	peak RSS (MB)
100000	identity	ok	0.00	0.00	290.32
100000	fiducio TS	ok	1.33	0.00	374.58
100000	fiducio VS	ok	1.52	0.00	370.84
100000	fiducio MS	ok	1.53	0.00	378.94
100000	fiducio DC	ok	1.53	0.01	380.71
100000	netcal TS	ok	0.07	0.01	410.62
100000	torch-uncertainty TS	ok	1.03	0.00	589.90
100000	torch-uncertainty VS	ok	1.09	0.00	590.39
100000	torch-uncertainty MS	ok	1.11	0.00	594.98
100000	dirichletcal DC	ok	2.78	0.01	1479.97
1000000	identity	ok	0.00	0.02	392.63
1000000	fiducio TS	ok	13.45	0.03	525.79
1000000	fiducio VS	ok	13.89	0.03	490.89
1000000	fiducio MS	ok	14.20	0.04	501.19
1000000	fiducio DC	ok	27.04	0.06	520.62
1000000	netcal TS	ok	0.72	0.08	609.58
1000000	torch-uncertainty TS	ok	10.35	0.02	677.11
1000000	torch-uncertainty VS	ok	11.28	0.02	677.67
1000000	torch-uncertainty MS	ok	10.45	0.02	668.88
1000000	dirichletcal DC	ok	24.08	0.09	5631.00

Scaling (GPU)

Same scenario and process, on NVIDIA GeForce RTX 3090 (torch 2.15.0+cu132). Fiducio and torch-uncertainty run on the GPU; netcal and dirichletcal are numpy/JAX and stay on the CPU, so only the CUDA rows are comparable with each other, and only on this machine.

The graph shows CUDA rows only; the table also includes CPU baselines. Medians over 3 repetitions; no spread is shown.

N	method	device	status	fit (s)	transform (s)	peak RSS (MB)	peak VRAM (MB)
100000	identity	cpu	ok	0.00	0.0057	799.41
100000	fiducio TS	cuda	ok	0.29	0.0007	1393.97	12.21
100000	fiducio VS	cuda	ok	0.24	0.0007	1307.01	9.16
100000	fiducio MS	cuda	ok	0.23	0.0007	1404.30	14.24
100000	fiducio DC	cuda	ok	0.31	0.0010	1404.70	15.76
100000	netcal TS	cpu	ok	0.08	0.0083	940.59
100000	torch-uncertainty TS	cuda	ok	0.99	0.0004	1381.91	10.53
100000	torch-uncertainty VS	cuda	ok	0.97	0.0004	1382.82	10.52
100000	torch-uncertainty MS	cuda	ok	1.01	0.0004	1477.25	12.06
100000	dirichletcal DC	cpu	ok	2.76	0.0131	1859.47
1000000	identity	cpu	ok	0.00	0.0301	862.17
1000000	fiducio TS	cuda	ok	0.59	0.0025	1451.34	126.52
1000000	fiducio VS	cuda	ok	0.63	0.0025	1363.92	94.52
1000000	fiducio MS	cuda	ok	0.64	0.0026	1461.45	93.78
1000000	fiducio DC	cuda	ok	0.65	0.0027	1464.34	109.78
1000000	netcal TS	cpu	ok	0.64	0.0797	1318.34
1000000	torch-uncertainty TS	cuda	ok	8.36	0.0020	1431.36	71.01
1000000	torch-uncertainty VS	cuda	ok	8.34	0.0020	1427.14	71.01
1000000	torch-uncertainty MS	cuda	ok	8.39	0.0027	1527.36	71.00
1000000	dirichletcal DC	cpu	ok	23.38	0.0856	6062.28

Features

	Fiducio	netcal	torch-uncertainty	dirichletcal
Native (B, C, *spatial) input, 2D and 3D	yes	no (flatten to (N, C))	temperature only (other scalers need (N, C))	no (flatten to (N, C))
mask and ignore_index	yes	no (index before fit)	no mask (-100 ignored by the loss)	no (index before fit)
Logits as input	yes	no, probabilities only	yes	no, probabilities only
Save and reload a fitted calibrator	yes	yes (save_model)	manual (state_dict + set_model)	not evaluated
Per-class bias in vector scaling	yes	n/a	no	n/a
Class-conditional calibrators (CDC, CMSap, CMSop)	yes	none among the methods tested	none among the methods tested	none among the methods tested

"Flatten" means the adapter used here, not a limitation proven in the library; "not evaluated" means the capability was not tested for this page; "n/a" means no vector-scaling method was tested for that library.

Reproduce

cd benchmarks
pip install -r requirements.txt  # Python 3.12 or 3.13
python run.py calibration && python run.py scaling && python run.py downstream && python run.py robustness && python run.py report
# with a CUDA build of torch, for the GPU table:
python run.py scaling --device cuda --out results/scaling-gpu.csv && python run.py report