Non-parametric case bootstrap

Description

Resample whole subjects with replacement, refit the same model to each replicate, and summarise the spread of the estimates into bias, standard errors and confidence intervals. Feature target: PsN’s bootstrap.

Usage

ferx_bootstrap(
  model,
  data = NULL,
  samples = 200,
  seed = 1,
  threads = NULL,
  stratify_on = NULL,
  sample_size = NULL,
  update_inits = TRUE,
  run_base_model = TRUE,
  keep_covariance = FALSE,
  dofv = FALSE,
  skip_minimization_terminated = TRUE,
  skip_estimate_near_boundary = TRUE,
  skip_covariance_step_terminated = FALSE,
  skip_with_covstep_warnings = FALSE,
  ci = 95,
  directory = NULL,
  resume = FALSE,
  retry_failed = FALSE,
  progress = interactive(),
  verbose = FALSE
)

Arguments

  • model: Path to a .ferx model file, or a ferx_model object from [ferx_model](ferx_model.qmd).
  • data: Path to a NONMEM-format CSV. NULL (the default) uses the model’s [data] block.
  • samples: Number of bootstrap datasets. Default 200, PsN’s default and its stated rule of thumb for standard errors.
  • seed: Master seed. Each replicate’s draw is derived from this and its own index, so a run is reproducible and independent of threads.
  • threads: Replicates to fit concurrently. NULL (the default) uses the engine default.
  • stratify_on: Name of a data column defining the resampling strata, or NULL (the default) for an unstratified run. Stratifying on study or arm keeps each replicate’s composition close to the original dataset’s.
  • sample_size: How many subjects each replicate draws. NULL (the default) draws as many as the original dataset has - the only case where a stratified proportional split is guaranteed to sum back to the request. A single number sets one total for the replicate; a named numeric vector (c("1001" = 12, "1002" = 24)) sets an explicit count per stratum and is only meaningful together with stratify_on. Every count must be a whole number >= 1. It is PsN’s -sample_size=1001=>12,1002=>24, spelled the R way.
  • update_inits: Start each replicate from the base fit’s final estimates (PsN default, TRUE). Requires run_base_model = TRUE.
  • run_base_model: Fit the original dataset first. Required for update_inits, for the bias column, and for the normal-approximation intervals. Default TRUE.
  • keep_covariance: Run the covariance step for each replicate. Off by default: it is most of the cost, and the bootstrap standard error comes from the spread of the estimates, not from any per-replicate R^-1. Turning it on adds se_<parameter> columns to $raw and is what the two covariance-step exclusion filters read.
  • dofv: Compute the change in objective function value: evaluate each replicate’s parameter vector on the original dataset with no estimation (NONMEM MAXEVAL=0) and subtract the original fit’s OFV. Roughly doubles the run time. Default FALSE.
  • skip_minimization_terminated: Exclude replicates whose minimization terminated. Default TRUE (PsN’s default).
  • skip_estimate_near_boundary: Exclude replicates whose estimate sits on a boundary. Default TRUE (PsN’s default).
  • skip_covariance_step_terminated, skip_with_covstep_warnings: Exclude replicates on the covariance step’s outcome. Both default FALSE, and both require keep_covariance = TRUE - asking for one without it is an error rather than a silently dropped filter.
  • ci: Two-sided confidence level, in percent. Default 95.
  • directory: Where to write the CSV artefacts (raw_results.csv, bootstrap_results.csv, bootstrap_diagnostics.csv, the individual and key files, and delta_ofv.csv when dofv = TRUE). NULL, the default, computes everything in memory and writes nothing - an R user has the data frames in hand and rarely wants eight files appearing in the working directory. The CLI defaults the other way. Set it if you want to be able to call [ferx_bootstrap_summarize](ferx_bootstrap_summarize.qmd) later.
  • resume: Continue an interrupted run in directory instead of starting a fresh one, refitting only the replicates that directory does not already hold. Default FALSE. Needs directory, and needs the arguments that shape the replicates to match the ones the directory was written with - samples, seed, sample_size, stratify_on, run_base_model, update_inits, keep_covariance and dofv, plus the model and the data themselves. A mismatch is an error, not a fresh run: resuming with a larger samples does not extend the earlier run.
  • retry_failed: Refit the replicates a resumed run finds recorded as failed, instead of carrying the failure forward. Default FALSE (PsN’s default). Needs resume = TRUE.
  • progress: Show a progress bar while the replicates fit. Default interactive(), so a script or a knitted document prints nothing. Uses the cli package when it is installed and utils::txtProgressBar() otherwise. The bar is drawn by this R session while the fits run in the engine, and reports what the run will actually do: a resume = TRUE run whose directory already holds most of the replicates counts only the ones still to fit.
  • verbose: Print a one-line run header to stderr. Default FALSE.

Details

The bootstrap runs in the engine (ferx-core’s ferx-tools crate); this function is a thin wrapper. The percentile estimator, the exclusion filters, the parameter naming and the sample_size syntax all live there, so an R session and the ferx bootstrap CLI produce the same numbers from the same seed.

What the bootstrap buys over the covariance step

ferx’s standard errors are the asymptotic R^-1 matrix, symmetric by construction. The bootstrap distribution assumes neither normality nor symmetry: it survives a failed or non-positive-definite covariance step, and it shows asymmetry in poorly identified parameters instead of averaging it away. Both intervals are returned side by side (ci_lower / ci_upper for the percentile interval, ci_lower_normal / ci_upper_normal for the normal approximation built from the bootstrap standard error) precisely so the disagreement between them is visible.

Exclusions

A replicate whose minimization terminated, or whose estimate sits on a boundary, is not a draw from the sampling distribution of a converged fit, so by default it is excluded from the statistics (PsN’s defaults). The replicate is still fitted and still appears in $raw; only the summary drops it. $diagnostics reports how many were dropped and why. To change the criteria after a run has finished, use [ferx_bootstrap_summarize](ferx_bootstrap_summarize.qmd)instead of refitting - but that needs directory to have been set.

Cost

This is samples (+ 1) complete fits. At PsN’s rule-of-thumb 200 samples a model that takes a minute to fit takes over three hours, so threads is the argument that matters most. progress (on by default in an interactive session) draws a bar as the replicates come in, so a long run says where it is. A run is interruptible: Ctrl-C is checked several times a second, the fits in flight stop at their next iteration, and the call raises ferx_bootstrap: cancelled by user rather than returning. It raises rather than returning the partial run because the replicates the interrupt aborted come back as failed fits, so a returned summary would be a normal-looking table computed over however many replicates happened to finish. The ones that did finish are not lost if directory was set - they are written there as they land - so set it before starting a long run, then either summarise them with [ferx_bootstrap_summarize](ferx_bootstrap_summarize.qmd) or carry on with resume = TRUE.

Resuming an interrupted run

resume = TRUE continues the run in directory, refitting only the sample indices its raw_results.csv does not already carry. It is sound because a replicate’s draw is a pure function of (seed, index): a reused replicate is bit-for-bit the one an uninterrupted run would have produced, so a resumed run’s artefacts are identical to that run’s. The base fit is reused too and is deliberately not refitted - a second fit can land on a slightly different optimum, and with update_inits = TRUE that would start the new replicates from a different point than the ones already on disk. The engine refuses to resume from a directory that belongs to a different run: the model and data hashes, the parameter names and the settings that shape the replicates are recorded alongside them and are checked before any is reused. Those settings are samples, seed, sample_size, stratify_on, run_base_model, update_inits, keep_covariance and dofv - all of them pinned, so a resumed run cannot extend an earlier one. Calling again with a larger samples is refused rather than topping the run up; a bigger bootstrap means a fresh directory. A replicate whose fit errored is carried forward as a failure rather than refitted, matching PsN - a fit that failed usually fails again. retry_failed = TRUE (which needs resume = TRUE) refits those instead, and is for a failure that was transient - an out-of-memory kill, a full disk - rather than one of the model.

Seealso

[ferx_bootstrap_summarize](ferx_bootstrap_summarize.qmd) to change the exclusion criteria on a finished run, [ferx_sir](ferx_sir.qmd) and [ferx_covariance](ferx_covariance.qmd) for the parametric uncertainty routes. Other fitting: [ferx_bootstrap_summarize](ferx_bootstrap_summarize.qmd), [ferx_check_init](ferx_check_init.qmd), [ferx_coef](ferx_coef.qmd), [ferx_collect](ferx_collect.qmd), [ferx_covariance](ferx_covariance.qmd), [ferx_fit](ferx_fit.qmd), [ferx_inits_from_nca](ferx_inits_from_nca.qmd), [ferx_sir](ferx_sir.qmd), [ferx_stop](ferx_stop.qmd), [print.ferx_job](ferx_fit_async.qmd)

Concept

fitting

Value

An object of class ferx_bootstrap, a list with:

  • parameters: Data frame, one row per parameter: parameter, original, mean, bias, standard_error, median, ci_lower, ci_upper, ci_lower_normal, ci_upper_normal. This is bootstrap_results.csv.
  • raw: Data frame, one row per fit with the original dataset first (sample = 0), carrying the estimates and the termination diagnostics the exclusion filters read. This is raw_results.csv, and it is what makes a custom R-side filter or re-summarisation possible.
  • diagnostics: Data frame of statistic / value: the run counts, the exclusion tallies (excluded: ...) and the diagnostic means (mean: ...).
  • delta_ofv: Data frame of sample / delta_ofv when dofv = TRUE, otherwise NULL.
  • parameter_names, subject_ids, n_completed, n_included, chi_square_df, confidence_level, samples, seed, model, data, directory: Run metadata. chi_square_df is the number of estimated parameters, the reference degrees of freedom for the dofv panel of [plot.ferx_bootstrap](plot.ferx_bootstrap.qmd).

Examples

ex <- ferx_example("warfarin")

bs <- ferx_bootstrap(ex$model, ex$data, samples = 200, seed = 12345,
                     threads = 8)
bs
bs$parameters
plot(bs)

# Stratified, with artefacts on disk so the run can be re-summarised later
bs <- ferx_bootstrap(ex$model, ex$data, samples = 200,
                     stratify_on = "SEX", directory = "warfarin-bootstrap")
ferx_bootstrap_summarize("warfarin-bootstrap",
                         skip_estimate_near_boundary = FALSE)

# Continue that run after it was interrupted: only the replicates missing
# from the directory are fitted again.
bs <- ferx_bootstrap(ex$model, ex$data, samples = 200,
                     stratify_on = "SEX", directory = "warfarin-bootstrap",
                     resume = TRUE)