DATA → MODEL → ESTIMATE ⇄ EVALUATE → SIMULATE → REPORT
Parameter uncertainty
Where you are
Model development and selection selected the covariate model two_cpt_oral_cov. Before reporting its estimates or simulating from it, quantify how precisely they are known. ferx offers four routes:
- the covariance step: asymptotic standard errors from the curvature of the objective function;
- SIR: sampling importance resampling around that asymptotic distribution;
- the bootstrap: refitting the model to resampled datasets;
- Bayesian estimation: posterior intervals.
This chapter applies each to the covariate model and compares their intervals.
The data
cov <- ferx_example("two_cpt_oral_cov")
fit <- ferx_fit(cov$model, cov$data, verbose = FALSE)Minimal runnable call: the covariance step
The covariance step runs by default (covariance = TRUE in this model file). Its results are the se, rse_pct and 95% interval columns of fit$estimates:
fit$covariance_status
#> [1] "computed"
fit$estimates[, c("estimate", "se", "rse_pct", "lower_95", "upper_95")]
#> estimate se rse_pct lower_95 upper_95
#> TVCL 4.94779482 0.359479607 7.265451 4.24321479 5.65237485
#> TVV1 48.58746315 2.952179258 6.076010 42.80119181 54.37373450
#> TVQ 9.57692886 0.407841779 4.258586 8.77755898 10.37629875
#> TVV2 94.17330535 3.423849312 3.635690 87.46256070 100.88405000
#> TVKA 1.25592770 0.098051252 7.807078 1.06374725 1.44810816
#> THETA_WT 0.65345745 0.236130854 36.135613 0.19064098 1.11627393
#> THETA_CRCL 0.56461168 0.221501166 39.230709 0.13046940 0.99875397
#> ETA_CL 0.10753126 0.028096431 26.128617 0.05246225 0.16260026
#> ETA_V1 0.10634762 0.028678266 26.966534 0.05013822 0.16255702
#> ETA_Q 0.05124926 0.014007549 27.332194 0.02379447 0.07870406
#> ETA_V2 0.03793979 0.010328798 27.224185 0.01769534 0.05818423
#> ETA_KA 0.17543723 0.048574875 27.687895 0.08023047 0.27064398
#> PROP_ERR 0.01961856 0.001123479 5.726613 0.01741654 0.02182058The full variance-covariance matrix of the estimates is fit$cov_matrix, and its correlation form is fit$cor_matrix (Diagnosing the model).
Choosing the covariance estimator
| Setting | Values | Default | Description |
|---|---|---|---|
covariance_method |
r, s, rsr
|
r |
The estimator that produced cov_matrix and the standard errors: “r”, “s” or “rsr”, the [fit_options] token (ferx-core #1382). |
covariance_fallback |
none, sir
|
none |
What to do when the covariance Hessian is non-PD, whether R is analytic or finite-difference. |
analytic_cov_hessian |
true, false
|
true |
Use the exact analytic R-matrix for the covariance step when the model is in scope, instead of the finite-difference-of-OFV Hessian. |
cov_inner_tol |
float |
inner_tol (LTBS: min(inner_tol, 1e-8)) |
Inner EBE-reconvergence tolerance used only by the covariance step, decoupled from inner_tol. |
fd_hessian_step |
float | 1e-2 |
Initial relative step size for the finite-difference Hessian used in the covariance step. |
fd_hessian_step is also a dedicated ferx_fit() argument. ferx_covariance() runs the covariance step on an existing fit: to add standard errors to a fit made without them, or to try another estimator without re-estimating. It takes the fitted model as fit, re-reads the model and data files, and stops if they changed since the fit:
fit_nocov <- ferx_fit(cov$model, cov$data, covariance = FALSE, verbose = FALSE)
#> Warning: Model file [fit_options] sets `covariance = true` but ferx_fit()
#> argument overrides it with `false`. The call-time value will be used.
se_by_method <- sapply(c(r = "r", s = "s", rsr = "rsr"), function(m) {
ferx_covariance(fit_nocov, covariance_method = m)$se_theta
})
round(se_by_method, 4)
#> r s rsr
#> TVCL 0.3595 0.3819 0.4216
#> TVV1 2.9522 4.5246 2.8639
#> TVQ 0.4078 0.6359 0.3967
#> TVV2 3.4238 4.7218 3.3730
#> TVKA 0.0981 0.1320 0.0954
#> THETA_WT 0.2361 0.2322 0.3255
#> THETA_CRCL 0.2215 0.3284 0.2159"r" (inverse Hessian, the default) reproduces the inline standard errors, "s" is the inverse cross-product and "rsr" the sandwich estimator. ferx_covariance() also takes mu_referencing, which by default is the value the fit was made with, and verbose. It repeats the inner-loop settings the fit recorded in fit$scoring_settings. threads = sets the worker threads of the step (by default the width the fit ran on, fit$n_threads_used). It changes the resources the step uses, not the result:
one_thread <- ferx_covariance(fit_nocov, threads = 1)
one_thread$n_threads_used
#> [1] 1
identical(one_thread$se_theta, se_by_method[, "r"])
#> [1] TRUESIR
SIR draws parameter vectors from the asymptotic distribution, weights them by how well they fit the data, and resamples according to the weights. The resampled vectors give intervals that need not be symmetric. Run it inline with ferx_fit(sir = TRUE), or afterwards with ferx_sir(), which takes the fitted model as fit:
fit_sir <- ferx_sir(fit, sir_samples = 1000, sir_resamples = 250)
fit_sir$sir_ess
#> [1] 190.8624
fit_sir$sir_ci_theta
#> lower upper
#> TVCL 4.1551002 5.7217957
#> TVV1 39.3153608 55.8373623
#> TVQ 8.8920108 10.7101110
#> TVV2 87.0715792 101.6714739
#> TVKA 1.0740876 1.4939643
#> THETA_WT 0.2488871 1.2221405
#> THETA_CRCL 0.1568595 0.9282956sir_ci_omega and sir_ci_sigma hold the intervals for the variances and residual error. sir_ess is the effective sample size of the sir_samples weighted draws: the larger it is, the better the proposal matches the data. The ferx-core SIR page reads one above sir_resamples as a well-matched proposal and one below 100 as too few draws for the intervals. sir_keep_samples = TRUE keeps the resampled vectors on the fit, which ferx_simulate_with_uncertainty() needs (Simulating scenarios). verbose = TRUE prints progress, and threads = sets the worker threads, as for ferx_covariance() above: identical(ferx_sir(fit, sir_samples = 1000, sir_resamples = 250, threads = 1)$sir_ci_theta, fit_sir$sir_ci_theta) is TRUE.
ImportantNeeds ferx-r newer than 0.4.0
The sir_df, sir_scale and threads arguments of ferx_sir(), the threads argument of ferx_covariance(), and the sir_scale setting, are not in ferx-r 0.4.0.
NoteChanged after ferx-r 0.4.0
In ferx-r 0.4.0, ferx_sir() always used its own defaults (1000 samples, 250 resamples, seed 12345), and ferx_covariance() its own inner-loop settings. Since then, both take whatever you leave out from the fit, as below.
Two arguments shape the proposal and the target. sir_df is the degrees of freedom of the Student-t proposal. sir_scale sets the scale on which the importance-sampling target is flat: "packed", the optimizer’s internal scale, or "natural", the scale the variances are reported on. They give different intervals for the variances:
fit_sir_natural <- ferx_sir(fit, sir_samples = 1000, sir_resamples = 250, sir_scale = "natural")
c(packed = fit_sir$sir_ess, natural = fit_sir_natural$sir_ess)
#> packed natural
#> 190.8624 231.7793
cbind(packed = fit_sir$sir_ci_omega, natural = fit_sir_natural$sir_ci_omega)
#> lower upper lower upper
#> ETA_CL 0.07671637 0.24934689 0.07657660 0.31573914
#> ETA_V1 0.07162684 0.22892543 0.07353742 0.24691423
#> ETA_Q 0.03222074 0.09464903 0.03579140 0.10680637
#> ETA_V2 0.02668870 0.07609299 0.02711921 0.08214648
#> ETA_KA 0.12017729 0.33033935 0.13288935 0.35603110Every SIR argument you leave out is taken from the SIR the fit already records in fit$sir_settings, and only otherwise from the defaults. So ferx_sir() on a fit that carries an SIR repeats it with the same draws, seed, sir_df and sir_scale:
| Setting | Values | Default | Description |
|---|---|---|---|
sir_samples |
1000 |
Number of proposal samples (M) | |
sir_resamples |
250 |
Number of resampled vectors (m) | |
sir_seed |
12345 |
RNG seed for reproducibility | |
sir_keep_samples |
false |
Retain resampled parameter vectors for simulate_with_uncertainty()
|
|
sir_df |
5.0 |
Degrees of freedom for the Student-t proposal (>= 1); higher values approach a normal proposal, and normal / mvn selects the multivariate-normal proposal itself (ν = ∞) |
|
sir_scale |
packed |
Scale the SIR target is flat on: packed (log-sd for variances) or natural. |
The effective sample size depends strongly on the random seed:
Check sir_ess on every run. When it is low, increase sir_samples and confirm that the intervals are stable across seeds before using them.
Bootstrap
ferx_bootstrap() resamples whole subjects with replacement, refits the model to each replicate and summarises the spread of the estimates. It needs the model and data rather than a fit, because every replicate is a new fit. With a directory it writes its tables to disk, so a run can be re-summarised or resumed later:
boot_dir <- file.path(tempdir(), "bootstrap")
dir.create(boot_dir, showWarnings = FALSE)
boot <- ferx_bootstrap(cov$model, cov$data, samples = 100, seed = 1, directory = boot_dir)
boot
#> ferx bootstrap
#> Model: /home/runner/work/_temp/Library/ferx/examples/models/two_cpt_oral_cov.ferx
#> Data: /home/runner/work/_temp/Library/ferx/examples/data/two_cpt_oral_cov.csv
#> Samples: 100 completed of 100 requested; 99 included
#> excluded 1: estimate near boundary
#> Wrote: /tmp/RtmpFOsTdr/bootstrap
#>
#> 95% confidence intervals (percentile / normal approximation):
#> parameter original mean bias standard_error median
#> TVCL 4.94800 4.94000 -0.0073140 0.2973000 4.97200
#> TVV1 48.59000 48.60000 0.0158100 2.8920000 48.69000
#> TVQ 9.57700 9.54800 -0.0292900 0.4054000 9.54100
#> TVV2 94.17000 94.60000 0.4270000 3.2670000 94.99000
#> TVKA 1.25600 1.26400 0.0078200 0.0905800 1.25300
#> THETA_WT 0.65350 0.69760 0.0441600 0.2357000 0.68190
#> THETA_CRCL 0.56460 0.55030 -0.0143300 0.2112000 0.53910
#> OMEGA(ETA_CL,ETA_CL) 0.10750 0.09941 -0.0081230 0.0246500 0.09537
#> OMEGA(ETA_V1,ETA_V1) 0.10630 0.09876 -0.0075900 0.0298800 0.09476
#> OMEGA(ETA_Q,ETA_Q) 0.05125 0.04858 -0.0026740 0.0146500 0.04815
#> OMEGA(ETA_V2,ETA_V2) 0.03794 0.03860 0.0006598 0.0101900 0.03828
#> OMEGA(ETA_KA,ETA_KA) 0.17540 0.16780 -0.0075950 0.0581400 0.16190
#> PROP_ERR 0.01962 0.01948 -0.0001433 0.0009454 0.01952
#> ci_lower ci_upper ci_lower_normal ci_upper_normal
#> 4.25800 5.54300 4.36500 5.53000
#> 42.40000 54.75000 42.92000 54.26000
#> 8.85300 10.56000 8.78200 10.37000
#> 88.47000 101.40000 87.77000 100.60000
#> 1.11600 1.51000 1.07800 1.43300
#> 0.24290 1.17100 0.19150 1.11500
#> 0.13230 0.99130 0.15060 0.97860
#> 0.06062 0.15770 0.05922 0.15580
#> 0.05682 0.16650 0.04779 0.16490
#> 0.02407 0.08183 0.02253 0.07996
#> 0.01755 0.05804 0.01797 0.05790
#> 0.07277 0.30700 0.06149 0.28940
#> 0.01737 0.02137 0.01777 0.02147print() on the ferx_bootstrap result shows the run summary and the boot$parameters table: original estimate, bootstrap mean, bias, standard error, median, and the percentile and normal-approximation intervals. boot$raw has one row per fitted replicate, and boot$diagnostics records the counts and exclusions:
boot$diagnostics
#> statistic value
#> 1 samples_requested 100.0000000
#> 2 samples_completed 100.0000000
#> 3 samples_included 99.0000000
#> 4 chi_square_df 13.0000000
#> 5 excluded: estimate near boundary 1.0000000
#> 6 mean: minimization_successful 1.0000000
#> 7 mean: estimate_near_boundary 0.0100000
#> 8 mean: covariance_step_successful 0.0000000
#> 9 mean: covariance_step_warnings 0.0000000
#> 10 mean: ofv -1214.7435231
#> 11 mean: subproblem_est_time 0.2408992By default, replicates whose minimization terminated or whose estimate lies on a boundary are fitted but excluded from the statistics. plot() on a ferx_bootstrap object draws one histogram per parameter, with the original estimate and the interval:
ferx_bootstrap_summarize() recomputes the statistics from the run directory under different exclusion criteria or a different ci level, without refitting:
relaxed <- ferx_bootstrap_summarize(boot_dir, skip_estimate_near_boundary = FALSE, ci = 90)
relaxed$n_included
#> [1] 100
head(relaxed$parameters[, c("parameter", "ci_lower", "ci_upper")], 7)
#> parameter ci_lower ci_upper
#> 1 TVCL 4.3820545 5.4518400
#> 2 TVV1 43.0779434 53.1257863
#> 3 TVQ 8.9843180 10.2807022
#> 4 TVV2 89.1504841 100.5879571
#> 5 TVKA 1.1364730 1.4393714
#> 6 THETA_WT 0.2968935 1.0969092
#> 7 THETA_CRCL 0.1816331 0.8904115ferx_bootstrap() arguments:
| Argument | Default | Meaning |
|---|---|---|
model, data
|
— | Model and dataset (data = NULL uses [data]) |
samples |
200 | Number of bootstrap datasets |
seed |
1 | Master seed; results do not depend on threads
|
threads |
NULL |
Replicates fitted concurrently |
stratify_on |
NULL |
Column defining resampling strata |
sample_size |
NULL |
Subjects per replicate (a named vector gives counts per stratum) |
update_inits |
TRUE |
Start each replicate from the original fit’s estimates |
run_base_model |
TRUE |
Fit the original dataset first (needed for bias and update_inits) |
keep_covariance |
FALSE |
Run the covariance step per replicate |
dofv |
FALSE |
Evaluate each replicate’s estimates on the original data (about doubles the run time) |
skip_minimization_terminated |
TRUE |
Exclude replicates whose minimization terminated |
skip_estimate_near_boundary |
TRUE |
Exclude replicates with an estimate on a boundary |
skip_covariance_step_terminated, skip_with_covstep_warnings
|
FALSE |
Exclude on the per-replicate covariance step (needs keep_covariance = TRUE) |
ci |
95 | Confidence level in percent |
directory |
NULL |
Where to write the run; NULL keeps it in memory |
resume, retry_failed
|
FALSE |
Continue an interrupted run in directory, optionally refitting failed replicates |
progress, verbose
|
— | Progress bar (interactive sessions) and a run header |
ferx_bootstrap_summarize() takes the run directory plus the four skip_* filters and ci.
Bayesian estimation
method = "bayes" returns posterior summaries instead of a covariance step (Estimation methods and controlling the fit). Posterior intervals are only meaningful when the chains have converged, which ferx reports as a largest R-hat at or below 1.01.
Three runs try: the default, four times longer chains, and four times longer chains with HMC proposals for the random effects (n_leapfrog = 10), which make the base model converge in Estimation methods and controlling the fit. The table adds the 2.5% posterior quantile of the two covariate exponents (WT_q025, CRCL_q025), whose lower bounds the first lines print:
grep("theta THETA_(WT|CRCL)", readLines(cov$model), value = TRUE)
#> [1] " theta THETA_WT(0.75, 0.01, 5.0)" " theta THETA_CRCL(0.50, 0.01, 5.0)"
bayes_settings <- list(default = list(), four_times_longer = list(bayes_warmup = 4000, bayes_iters = 4000),
four_times_longer_hmc = list(bayes_warmup = 4000, bayes_iters = 4000, n_leapfrog = 10))
# suppress the R warnings that report overriding the model file's method
bayes_runs <- lapply(bayes_settings, function(s) {
suppressWarnings(ferx_fit(cov$model, cov$data, method = "bayes", settings = s, verbose = FALSE))
})
exponent_q025 <- function(b, name) b$q025[b$param_names == name]
do.call(rbind, lapply(names(bayes_runs), function(run) {
b <- bayes_runs[[run]]$bayes
data.frame(run = run, draws_per_chain = b$n_draws_per_chain, converged = bayes_runs[[run]]$converged,
max_rhat = round(b$max_rhat, 3), WT_q025 = signif(exponent_q025(b, "THETA_WT"), 2),
CRCL_q025 = signif(exponent_q025(b, "THETA_CRCL"), 2))
}))
#> run draws_per_chain converged max_rhat WT_q025 CRCL_q025
#> 1 default 1000 TRUE 1.006 0.20 0.15
#> 2 four_times_longer 4000 TRUE 1.001 0.20 0.14
#> 3 four_times_longer_hmc 4000 TRUE 1.009 0.19 0.15The largest R-hat is 1.006 with the default chains, 1.001 with the longer ones and 1.009 with HMC proposals: all three runs converge. The covariate exponents stay inside their bounds, with 2.5% quantiles of at least 0.14 against a lower bound of 0.01. The intervals of the longer run go into the comparison below.
NoteChanged after ferx-r 0.4.0
With ferx-r 0.4.0, none of these three runs converges on this model, and the HMC run draws both covariate exponents far below their lower bound of 0.01: its sampler does not enforce theta bounds (ferx-core #1626, fixed after that release). Check max_rhat and the quantiles against the bounds before using Bayesian intervals there.
Comparing the intervals
thetas <- names(fit$theta)
intervals <- bind_rows(
data.frame(method = "covariance step", parameter = thetas,
lower = fit$estimates[thetas, "lower_95"], upper = fit$estimates[thetas, "upper_95"]),
data.frame(method = "SIR", parameter = thetas,
lower = fit_sir$sir_ci_theta[thetas, "lower"], upper = fit_sir$sir_ci_theta[thetas, "upper"]),
data.frame(method = "bootstrap", parameter = thetas,
lower = boot$parameters$ci_lower[match(thetas, boot$parameters$parameter)],
upper = boot$parameters$ci_upper[match(thetas, boot$parameters$parameter)]),
data.frame(method = "Bayes", parameter = thetas,
lower = bayes_runs$four_times_longer$bayes$q025[match(thetas, bayes_runs$four_times_longer$bayes$param_names)],
upper = bayes_runs$four_times_longer$bayes$q975[match(thetas, bayes_runs$four_times_longer$bayes$param_names)])
) |>
left_join(data.frame(parameter = thetas, estimate = fit$theta), by = "parameter") |>
mutate(lower_rel = lower / estimate, upper_rel = upper / estimate)
ggplot(intervals, aes(y = method, xmin = lower_rel, xmax = upper_rel, colour = method)) +
geom_vline(xintercept = 1, linetype = "dashed") +
geom_errorbar(orientation = "y", width = 0.3) +
facet_wrap(~parameter, ncol = 1, strip.position = "left") +
labs(x = "Interval relative to the estimate", y = NULL) +
labs(colour = NULL) +
theme(legend.position = "bottom", strip.text.y.left = element_text(angle = 0),
axis.text.y = element_blank())
The structural parameters are precisely estimated by every route. The covariate exponents THETA_WT and THETA_CRCL have wide intervals, and the four routes differ in how wide. For a small dataset like this one, report the method you used and, where it matters, check that the conclusions hold under a second method.
Pitfalls
An unreliable covariance step. A fit can report standard errors that should not be trusted. The bundled emax_pkpd model is an example: a joint PK/PD model fitted to three subjects (PK/PD and multiple endpoints), whose covariance matrix is ill-conditioned and whose PD parameters are almost perfectly correlated.
emax <- ferx_example("emax_pkpd")
fit_bad <- ferx_fit(emax$model, emax$data, verbose = FALSE)
w_bad <- ferx_get_warnings(fit_bad, as_df = TRUE)
w_bad[w_bad$category %in% c("condition_number", "high_correlation", "inflated_rse"), c("severity", "category")]
#> severity category
#> 1 warning inflated_rse
#> 2 warning high_correlation
#> 3 critical condition_number
fit_bad$estimates[c("TVE0", "TVEMAX", "TVEC50"), c("estimate", "rse_pct")]
#> estimate rse_pct
#> TVE0 3.208106 123.49032
#> TVEMAX 45.243463 30.55821
#> TVEC50 3.992944 21.51878In such a case, fix the model before quantifying uncertainty. covariance_fallback = "sir" runs SIR automatically when the covariance step fails.
Other points:
-
ferx_sir()andferx_covariance()check that the model and data files are unchanged since the fit, and stop otherwise. - A bootstrap is
samplescomplete fits. Setdirectoryfor long runs, so an interrupted run can be resumed and re-summarised. - Bayesian intervals require converged chains. Check
fit$bayes$rhatand the effective sample sizes.
Warnings you may see here
See convergence and covariance warnings and resampling warnings.
-
covariance_step(info): a note on the covariance step, for example its cost. -
covariance_failed(critical): the covariance step failed and there are no standard errors. Usecovariance_fallback = "sir"or refit with different starting values. -
covariance_regularized(warning): the step succeeded but was regularised, so the standard errors may be optimistic. Inspect the eigenvalues or use SIR or bootstrap intervals. -
sir(warning): SIR failed or was requested without a covariance matrix, or its proposal did not cover some directions (named in the message).
Summary
- The covariance step gives asymptotic standard errors in
fit$estimates.ferx_covariance()reruns it with another estimator. -
ferx_sir()(orsir = TRUE) gives resampling intervals. Checksir_ess. -
ferx_bootstrap()refits resampled datasets.ferx_bootstrap_summarize()andplot()work on its results. - Compare routes for poorly determined parameters such as covariate effects.
Next: Simulating scenarios simulates scenarios with the final model, including parameter uncertainty.
TipReference
- R help:
?ferx_covariance,?ferx_sir,?ferx_bootstrap,?ferx_bootstrap_summarize,?plot.ferx_bootstrap - ferx-core: covariance and standard errors, SIR, bootstrap