Skip to main content

Model development and selection

DATA → MODEL → ESTIMATE ⇄ EVALUATE → SIMULATE → REPORT

Where you are

The base model is fitted (Initial estimates and a first fit) and evaluated (Diagnosing the model, Simulation-based evaluation: VPC). The diagnostics hinted at covariate relationships (ETA_CL correlated with CRCL and WT) and flagged autocorrelated residuals. Model development means fitting alternatives and choosing between them. You can do that by hand: write a candidate, fit it and compare. Or let ferx’s search tools generate, fit, gate and rank candidates from a search space. This chapter shows both.

The data

base <- ferx_example("two_cpt_oral_base")
cov <- ferx_example("two_cpt_oral_cov")
# Search runs and the files this chapter writes go to a scratch directory, one
# subdirectory per run; the search tools create a run directory that does not exist yet.
search_dir <- file.path(tempdir(), "searches")
dir.create(search_dir, showWarnings = FALSE)

Comparing fitted models

two_cpt_oral_cov is a prespecified covariate model for the same data. Body weight scales CL and V1 through a shared exponent THETA_WT, and creatinine clearance scales CL:

invisible(ferx_model_get_section(cov$model, "individual_parameters"))
#> # [individual_parameters]
#>   CL = TVCL * (WT / 70)^THETA_WT * (CRCL / 100)^THETA_CRCL * exp(ETA_CL)
#>   V1 = TVV1 * (WT / 70)^THETA_WT * exp(ETA_V1)
#>   Q  = TVQ  * exp(ETA_Q)
#>   V2 = TVV2 * exp(ETA_V2)
#>   KA = TVKA * exp(ETA_KA)
fit_base <- ferx_fit(base$model, base$data, verbose = FALSE)
fit_cov <- ferx_fit(cov$model, cov$data, verbose = FALSE)

Likelihood ratio test and information criteria

Both are FOCEI fits to the same data, and the base model is nested in the covariate model (fix both exponents at 0). The difference in OFV is therefore a likelihood ratio statistic, with degrees of freedom equal to the number of added parameters:

d_ofv <- fit_base$ofv - fit_cov$ofv
df <- fit_cov$n_parameters - fit_base$n_parameters
data.frame(d_ofv = d_ofv, df = df, p_value = pchisq(d_ofv, df, lower.tail = FALSE))
#>      d_ofv df     p_value
#> 1 13.86445  2 0.000975825

For non-nested models, use information criteria. fit$aic and fit$bic are the classical AIC and BIC. ferx_bic() computes four BIC variants that differ in how parameters are penalised:

type Penalty added to the OFV
"mixed" (default) random-effect parameters × log(subjects) + fixed-effect parameters × log(observations)
"fixed" all parameters × log(observations); equals fit$bic
"iiv" variance parameters × log(subjects), for comparing variability structures
"random" all parameters × log(subjects)
Table 9.1: Base and covariate model compared on OFV and information criteria (lower is better).
criteria <- function(f) {
  c(ofv = f$ofv, parameters = f$n_parameters, aic = f$aic, bic = f$bic,
    bic_mixed = ferx_bic(f, "mixed"), bic_iiv = ferx_bic(f, type = "iiv"),
    bic_random = ferx_bic(f, "random"))
}
round(rbind(base = criteria(fit_base), covariate = criteria(fit_cov)), 2)
#>                ofv parameters      aic      bic bic_mixed  bic_iiv bic_random
#> base      -1185.46         11 -1163.46 -1122.72  -1145.75 -1168.46   -1148.05
#> covariate -1199.33         13 -1173.33 -1125.18  -1152.81 -1182.32   -1155.11

Here the covariate model is better on every criterion.

Is the fit eligible at all?

A converged fit can still be unfit for comparison: it may never have left its initial estimates, have a parameter at a bound, or have an ill-conditioned covariance matrix. check_strictness() applies such gates and returns the reasons for failure:

check_strictness(fit_cov, require_covariance = TRUE)
#> $passed
#> [1] TRUE
#> 
#> $failures
#> character(0)
#> 
#> $skipped
#> character(0)

The gates judge whether a fit is healthy, not whether it found the best optimum. Here they are applied to the fit from NCA-based starting values in Initial estimates and a first fit, and to the emax_pkpd example of PK/PD and multiple endpoints, three subjects fitted with a joint PK/PD model:

fit_nca <- ferx_fit(base$model, base$data, inits_from_nca = TRUE, verbose = FALSE)
nca_gate <- check_strictness(fit_nca)
nca_gate$passed
#> [1] TRUE
emax <- ferx_example("emax_pkpd")
emax_gate <- check_strictness(ferx_fit(emax$model, emax$data, verbose = FALSE))
emax_gate$failures
#> [1] "condition number 1.0555e+04 exceeds 1.0000e+03"   
#> [2] "parameter correlation |r| = 0.9933 exceeds 0.9500"

The NCA-start fit passes every gate, and here it also reaches the optimum of fit_base (the OFVs differ by 0.00003). The gates would pass it just the same had it stopped at a worse point, as this fit does with ferx-r 0.4.0 (Initial estimates and a first fit): only comparing OFVs shows that. The emax_pkpd fit is refused because its parameters cannot be told apart.

The gates and their defaults: require_converged = TRUE, require_covariance = FALSE, max_condition_number = 1000, max_correlation = 0.95, reject_on_boundary = TRUE and reject_init_stall = TRUE. Set a threshold to NULL, or a flag to FALSE, to disable that gate. reject_init_stall assumes a cold start from the model file’s initial values, so turn it off when candidates start from a previous model’s estimates.

Search configurations

WarningMaturity: alpha

The ferx-core pages for the search tools carry the maturity label alpha. See feature maturity.

The search tools read a .ferxsearch configuration file (TOML) that names the base model and data and states the search. Eight bundled examples carry one as $search:

Example Search file for
two_cpt_oral_base covariate search ([covsearch]), plus an [allometry] section (Covariate modeling)
two_cpt_oral_cov a configuration with a BIC ranking, used here to show loading
warfarin structural search ([modelsearch])
one_cpt_transit residual error search ([ruvsearch])
warfarin_block_omega between-subject variability search ([iivsearch])
warfarin_iov inter-occasion variability search ([iovsearch])
two_cpt_oral_global global search over structure and covariates together ([globalsearch])
warfarin_amd an automatic model development plan ([amd], with its stages)

ferx_search_config() loads a file with the engine’s own strict loader, so an invalid space or a misspelt section fails before anything is fitted:

cfg <- ferx_search_config(base$search)
cfg
#> <ferx search configuration>
#>   file:  two_cpt_oral_base.ferxsearch
#>   base:  /home/runner/work/_temp/Library/ferx/examples/search/../models/two_cpt_oral_base.ferx
#>   data:  /home/runner/work/_temp/Library/ferx/examples/search/../data/two_cpt_oral_cov.csv
#> 
#> Search space (1 feature):
#>   mfl = COVARIATE?(@IIV, @CONTINUOUS, [pow, lin])
#>   COVARIATE?(@IIV,@CONTINUOUS,[pow,lin])         (exploratory)
#> 
#> Rank: (tool default)   cutoff: (tool default)
#> Strictness (* = set by the file):
#>   * require_converged      TRUE
#>     require_covariance     FALSE
#>     max_condition_number   1000
#>     max_correlation        0.95
#>     reject_on_boundary     TRUE
#>   * reject_init_stall      TRUE
#> Run: threads = auto, retries = 2, resume = FALSE
#> Tool sections: allometry, covsearch

The file’s sections, as shown in the printout: base and data (paths relative to the file); [space] with the search space as mfl; [rank] with the ranking criterion; [strictness] with the gates from above; [run] with threads, retries and resume; and a section per tool. The object returned holds the same information as a list:

cfg_cov <- ferx_search_config(ferx_example("two_cpt_oral_cov")$search)
cfg_cov$rank$type
#> [1] "bic"
cfg$tools
#> [1] "allometry" "covsearch"

The search space

The space is written in the Model Feature Language (MFL). COVARIATE?(@IIV, @CONTINUOUS, [pow, lin]) means: every parameter with a random effect, crossed with every continuous covariate, as a power or linear effect, each optional (?). The ? is accepted on four keywords only – COVARIATE, IIV, IOV and COVARIANCE – and marks that statement as something the search may decline; written without it, every candidate carries the feature. The other keywords, ABSORPTION and PERIPHERALS among them, enumerate the alternatives to try instead. The @ symbols resolve against the model’s [covariates] block and its random effects, so one space travels between models: @CATEGORICAL selects the categorical covariates as @CONTINUOUS selects the continuous ones. ferx_search_space() parses MFL text and, given a model and data, expands the symbols into the concrete candidates:

ferx_search_space("COVARIATE?(@IIV, @CONTINUOUS, [pow, lin])")
#> <ferx search space> (as written)
#>   mfl = COVARIATE?(@IIV, @CONTINUOUS, [pow, lin])
#> 
#>                                  feature   keyword optional
#> 1 COVARIATE?(@IIV,@CONTINUOUS,[pow,lin]) COVARIATE     TRUE
space <- ferx_search_space(cfg$mfl, model = base$model, data = base$data)
nrow(attr(space, "covariate_effects"))
#> [1] 20
head(attr(space, "covariate_effects"))
#>   parameter covariate effect op optional
#> 1        CL        WT    pow  *     TRUE
#> 2        CL        WT    lin  *     TRUE
#> 3        CL      CRCL    pow  *     TRUE
#> 4        CL      CRCL    lin  *     TRUE
#> 5        V1        WT    pow  *     TRUE
#> 6        V1        WT    lin  *     TRUE

ferx_search_coverage() reports, without fitting, whether the engine can build candidates for each feature of a space:

ferx_search_coverage(space)
#>                                  feature covered reason
#> 1 COVARIATE?(@IIV,@CONTINUOUS,[pow,lin])    TRUE   <NA>
ferx_search_coverage("ABSORPTION(SEQ-ZO-FO)")
#>                 feature covered
#> 1 ABSORPTION(SEQ-ZO-FO)   FALSE
#>                                                                                                                                                                                                                                                                                                                       reason
#> 1 sequential zero-order-then-first-order absorption is not one input term on a standard disposition but a depot of its own, filled at a constant rate and emptied by `ka`; there is no template for it and the search does not generate one. Write it by hand — `examples/sequential_absorption.ferx` — and search around it

Sequential zero- then first-order absorption is not generated by the search. Absorption and bioavailability shows the bundled model for it, which you can use as a base model instead.

An uncovered feature is not quietly dropped. Asking for one in a configuration file is an error at load, before anything is fitted, because a search that silently narrowed its space would answer a different question than the one you posed:

uncovered <- file.path(search_dir, "uncovered.ferxsearch")
writeLines(sub('mfl = .*', 'mfl = "ABSORPTION([FO,SEQ-ZO-FO])"', readLines(base$search)), uncovered)
tryCatch(ferx_search_config(uncovered), error = function(e) sub(".*mfl: ", "", conditionMessage(e)))
#> [1] "the search space asks for 1 feature ferx cannot build a candidate for:\n  - ABSORPTION(SEQ-ZO-FO): sequential zero-order-then-first-order absorption is not one input term on a standard disposition but a depot of its own, filled at a constant rate and emptied by `ka`; there is no template for it and the search does not generate one. Write it by hand — `examples/sequential_absorption.ferx` — and search around it\nSee the coverage table at https://ferx-nlme.org/ferx-core/tools/search.html#coverage"

That is what ferx_search_coverage() is for: check a space before you file it.

All axes at once

Every tool so far walks one axis with the others held fixed: ferx_modelsearch() decides the structure with the covariate model fixed, ferx_covsearch() the covariates with the structure fixed. That is what keeps them cheap, and it is blind to one thing – an effect that only earns its place after another decision has gone the other way. A clearance covariate that pays off only once the distribution phase has a compartment of its own is invisible to a forward step taken on the one-compartment model.

ferx_globalsearch() lays both kinds of decision out as one grid and searches it whole. Each structural category in the space becomes an axis with its values as alleles; each optional COVARIATE? pair becomes an axis whose alleles are none and each of its forms. algorithm = "exhaustive" fits every point; algorithm = "ga", the default, runs a genetic algorithm and fits a fraction of them.

The bundled two_cpt_oral_global example is built to have something to find: a one-compartment, covariate-free model on data carrying WT and CRCL that were simulated from two compartments, so all four axes are live.

gs_ex <- ferx_example("two_cpt_oral_global")
gs <- ferx_globalsearch(config = gs_ex$search,
                        directory = file.path(search_dir, "globalsearch"), progress = FALSE)
gs
#> ferx global model search (exhaustive, ranked on penalized)
#>   Data:  /home/runner/work/_temp/Library/ferx/examples/search/../data/two_cpt_oral_cov.csv
#>   Wrote: /tmp/RtmpdcK7uF/searches/globalsearch
#>   Grid:  16 points over 4 axes
#>     LAGTIME: OFF | ON
#>     PERIPHERALS: 0 | 1
#>     CL-WT: none | power
#>     CL-CRCL: none | power
#>   Input: OFV -172.2, penalized -102.2
#>   Fitness: -102.2 (input) -> -808.2 (run6)
#>   17 models fitted of 17 evaluated
#> 
#> Ranked models (best first):
#>  rank    id                                              genome    ofv
#>     1  run6  LAGTIME=OFF;PERIPHERALS=1;CL-WT=none;CL-CRCL=power -908.2
#>     2  run5   LAGTIME=OFF;PERIPHERALS=1;CL-WT=none;CL-CRCL=none -889.5
#>     3  run7  LAGTIME=OFF;PERIPHERALS=1;CL-WT=power;CL-CRCL=none -895.8
#>     4  run8 LAGTIME=OFF;PERIPHERALS=1;CL-WT=power;CL-CRCL=power -901.4
#>     5 run14   LAGTIME=ON;PERIPHERALS=1;CL-WT=none;CL-CRCL=power -896.1
#>     6 run15   LAGTIME=ON;PERIPHERALS=1;CL-WT=power;CL-CRCL=none -895.8
#>     7  run1   LAGTIME=OFF;PERIPHERALS=0;CL-WT=none;CL-CRCL=none -172.2
#>     8 input                                                <NA> -172.2
#>     9  run2  LAGTIME=OFF;PERIPHERALS=0;CL-WT=none;CL-CRCL=power -179.4
#>    10  run3  LAGTIME=OFF;PERIPHERALS=0;CL-WT=power;CL-CRCL=none -179.3
#>    11  run4 LAGTIME=OFF;PERIPHERALS=0;CL-WT=power;CL-CRCL=power -185.5
#>  criterion fitness converged passed
#>    -808.20 -808.20      TRUE   TRUE
#>    -799.50 -799.50      TRUE   TRUE
#>    -795.80 -795.80      TRUE   TRUE
#>    -791.40 -791.40      TRUE   TRUE
#>    -776.10 -776.10      TRUE   TRUE
#>    -775.80 -775.80      TRUE   TRUE
#>    -102.20 -102.20      TRUE   TRUE
#>    -102.20 -102.20      TRUE   TRUE
#>     -99.36  -99.36      TRUE   TRUE
#>     -99.27  -99.27      TRUE   TRUE
#>     -95.50  -95.50      TRUE   TRUE
#> 
#> Models the search charged beyond the criterion:
#>     id                                             genome criterion
#>   run9   LAGTIME=ON;PERIPHERALS=0;CL-WT=none;CL-CRCL=none     16.53
#>  run10  LAGTIME=ON;PERIPHERALS=0;CL-WT=none;CL-CRCL=power     19.46
#>  run11  LAGTIME=ON;PERIPHERALS=0;CL-WT=power;CL-CRCL=none    119.60
#>  run12 LAGTIME=ON;PERIPHERALS=0;CL-WT=power;CL-CRCL=power    123.60
#>  run13   LAGTIME=ON;PERIPHERALS=1;CL-WT=none;CL-CRCL=none   -679.50
#>  run16 LAGTIME=ON;PERIPHERALS=1;CL-WT=power;CL-CRCL=power   -771.40
#>  charge_non_influential charge_gate charge_crash fitness
#>                       0         100            0   116.5
#>                       0         100            0   119.5
#>                       0         100            0   219.6
#>                       0         100            0   223.6
#>                       0         100            0  -579.5
#>                       0         100            0  -671.4

Read the top rows with both columns in view. The table is sorted on fitness, the criterion plus the charges described below, and the criterion itself adds to the OFV, so the best OFV need not win:

top <- head(gs$models[order(gs$models$fitness), c("id", "genome", "ofv", "criterion", "fitness")], 3)
top
#>     id                                             genome       ofv criterion
#> 7 run6 LAGTIME=OFF;PERIPHERALS=1;CL-WT=none;CL-CRCL=power -908.2372 -808.2372
#> 6 run5  LAGTIME=OFF;PERIPHERALS=1;CL-WT=none;CL-CRCL=none -889.5500 -799.5500
#> 8 run7 LAGTIME=OFF;PERIPHERALS=1;CL-WT=power;CL-CRCL=none -895.7898 -795.7898
#>     fitness
#> 7 -808.2372
#> 6 -799.5500
#> 8 -795.7898
charge <- top$criterion - top$ofv

Here the winner also has the better OFV (-908.2 against -889.5 for the runner-up), but the margin on the criterion is 8.7 rather than 18.7, because the criterion adds 100.0 and 90.0 to their OFVs, which is why the criterion and the OFV are separate columns.

The criterion is not the whole of the ranking

[rank] type defaults to "penalized" here – the objective function plus a charge per estimated parameter and per unhealthy fit – where every other tool defaults to a BIC. Whatever criterion you name, the global search then charges three things the criterion cannot see, and ranks on the sum:

Charge Paid by
charge_non_influential a gene that changed nothing in the rendered model, such as a covariate on a parameter the structural choice removed. A tie-break towards the simpler genotype among models that are otherwise identical.
charge_gate a fit the [strictness] gate refused. It can never be selected, but it still steers the genetic algorithm away from the region that produced it.
charge_crash a candidate that produced no fit at all. Large, and finite, so the algorithm can still rank it.

They are columns beside the criterion and the fitness they add up to, which is the “never a silent drop” rule of the other tools taken one step further: a gated genome does not merely appear in the table, it says what the gate cost it.

charged <- gs$models[gs$models$charge_gate > 0, ]
charged[, c("id", "genome", "criterion", "charge_gate", "fitness", "passed")]
#>       id                                             genome  criterion
#> 10  run9   LAGTIME=ON;PERIPHERALS=0;CL-WT=none;CL-CRCL=none   16.53243
#> 11 run10  LAGTIME=ON;PERIPHERALS=0;CL-WT=none;CL-CRCL=power   19.45919
#> 12 run11  LAGTIME=ON;PERIPHERALS=0;CL-WT=power;CL-CRCL=none  119.64797
#> 13 run12 LAGTIME=ON;PERIPHERALS=0;CL-WT=power;CL-CRCL=power  123.58167
#> 14 run13   LAGTIME=ON;PERIPHERALS=1;CL-WT=none;CL-CRCL=none -679.51882
#> 17 run16 LAGTIME=ON;PERIPHERALS=1;CL-WT=power;CL-CRCL=power -771.35621
#>    charge_gate   fitness passed
#> 10         100  116.5324  FALSE
#> 11         100  119.4592  FALSE
#> 12         100  219.6480  FALSE
#> 13         100  223.5817  FALSE
#> 14         100 -579.5188  FALSE
#> 17         100 -671.3562  FALSE
gs$penalties[c("non_influential", "gate", "crash")]
#> non_influential            gate           crash 
#>           1e-05           1e+02           1e+08

Without that column the first of those rows reads as a model that simply fitted worse than the winner, rather than one the gate refused and that sits 100 units away because of it.

The genetic algorithm

Sixteen points is small enough to enumerate, which is what makes the bundled file a reference: on a grid this size the algorithm has to find the same winner. On a grid of thousands, enumerating is not an option and finding it at all is the point.

ga_config <- file.path(search_dir, "global_ga.ferxsearch")
ga_lines <- readLines(gs_ex$search)
ga_lines <- sub("^base = .*", sprintf('base = "%s"', normalizePath(gs_ex$model, winslash = "/")), ga_lines)
ga_lines <- sub("^data = .*", sprintf('data = "%s"', normalizePath(gs_ex$data, winslash = "/")), ga_lines)
ga_lines <- sub('algorithm = "exhaustive"', 'algorithm = "ga"', ga_lines, fixed = TRUE)
writeLines(ga_lines, ga_config)
gs_ga <- ferx_globalsearch(config = ga_config,
                           directory = file.path(search_dir, "globalsearch-ga"), progress = FALSE)
gs_ga$generations
#>   index best best_fitness mean_fitness polished
#> 1     0 run5    -799.5500    -322.5866        0
#> 2     1 run9    -808.2372    -451.5442        0
#> 3     2 run9    -808.2372    -490.9802        0
#> 4     3 run9    -808.2372    -535.5488        1
data.frame(
  algorithm = c("exhaustive", "ga"),
  genome = c(gs$models$genome[gs$models$selected],
             gs_ga$models$genome[gs_ga$models$selected]),
  fitness = c(gs$final_fitness, gs_ga$final_fitness)
)
#>    algorithm                                             genome   fitness
#> 1 exhaustive LAGTIME=OFF;PERIPHERALS=1;CL-WT=none;CL-CRCL=power -808.2372
#> 2         ga LAGTIME=OFF;PERIPHERALS=1;CL-WT=none;CL-CRCL=power -808.2372

ga takes a named list checked against the engine’s own key list, so a mistyped knob is refused by name before anything is fitted. The seed is what makes resume = TRUE reuse a journal: the same seed on the same grid proposes the same genomes. algorithm is matched exactly rather than by prefix – algorithm = "e" is an error naming the value, not a silent decision to enumerate a grid nobody has sized – and an exhaustive grid above [globalsearch] max_models is an error naming the size, never a truncated search.

Warningferx_globalsearch() is not ferx_fit(settings = list(global_search = TRUE))

That setting is a global optimizer phase inside the estimation of one model: a wider search for the parameter values of a fixed model. ferx_globalsearch() is a global search over models. The name is shared by the engine, the .ferxsearch file ([globalsearch]) and the ferx globalsearch command, so a search stays portable between R and the command line.

The whole pipeline at once

ferx_amd() runs the tools above in order as one automatic model-development pipeline: structural, IIV, residual, IOV, allometry, covariates. One search space describes the whole pipeline, and the engine hands each step only the statements its tool can read. A step the space says nothing about is skipped, and says so.

ferx_amd_plan() shows that plan without fitting anything, which is the cheap way to check a configuration before spending the run:

amd_ex <- ferx_example("warfarin_amd")
plan <- ferx_amd_plan(config = amd_ex$search)
plan
#>   index       step        tool rerun      directory
#> 1     1 structural modelsearch FALSE 01-modelsearch
#> 2     2  iivsearch   iivsearch FALSE   02-iivsearch
#> 3     3   residual   ruvsearch FALSE   03-ruvsearch
#> 4     4  iovsearch   iovsearch FALSE   04-iovsearch
#> 5     5  allometry   allometry FALSE   05-allometry
#> 6     6 covariates   covsearch FALSE   06-covsearch
#>                                                                                                  skipped
#> 1                                                                                                   <NA>
#> 2                                                                                                   <NA>
#> 3                                                                                                   <NA>
#> 4 the starting model declares no `iov_column` in [fit_options], so the dataset's occasions were not read
#> 5                                                            the search space has no ALLOMETRY statement
#> 6                                                            the search space has no COVARIATE statement

3 of the 6 steps are already ruled out by the model and the space, and the skipped column says why: the model declares no iov_column, and the space carries no ALLOMETRY or COVARIATE statement because warfarin.csv has no covariate column. Running it fits the rest:

amd <- ferx_amd(config = amd_ex$search, directory = file.path(search_dir, "amd"), progress = FALSE)
amd
#> ferx AMD pipeline (default strategy, retries all_final)
#>   Model: /home/runner/work/_temp/Library/ferx/examples/search/../models/warfarin_amd.ferx
#>   Data:  /home/runner/work/_temp/Library/ferx/examples/search/../data/warfarin.csv
#>   Wrote: /tmp/RtmpdcK7uF/searches/amd
#>   OFV:   -286.004 (start) -> -286.004 (final, dOFV -0.000)
#>   3 of 6 steps ran, 28 models fitted, 1.0 s across the steps
#> 
#> Pipeline:
#>  index       step  status criterion value_before value_after    d_value
#>      1 structural     ran bic_mixed       -267.5      -267.5 -4.036e-12
#>      2  iivsearch     ran   bic_iiv       -279.1      -279.1 -1.137e-13
#>      3   residual     ran       ofv       -286.0      -286.0  0.000e+00
#>      4  iovsearch skipped      <NA>           NA          NA         NA
#>      5  allometry skipped      <NA>           NA          NA         NA
#>      6 covariates skipped      <NA>           NA          NA         NA
#>       d_ofv candidates converged passed seconds
#>  -4.036e-12          5      TRUE   TRUE  0.4257
#>  -1.137e-13         12      TRUE   TRUE  0.3062
#>   0.000e+00          7        NA     NA  0.2315
#>          NA          0        NA     NA  0.0000
#>          NA          0        NA     NA  0.0000
#>          NA          0        NA     NA  0.0000
#> 
#> Steps that did not run:
#>   4 iovsearch    skipped: the starting model declares no `iov_column` in [fit_options], so the dataset's occasions were not read
#>   5 allometry    skipped: the search space has no ALLOMETRY statement
#>   6 covariates   skipped: the search space has no COVARIATE statement
#> 
#> Selected:
#>   structural   FO, 0 peripherals
#>   iivsearch    [CL]+[KA]+[V]
#>   residual     no residual-error feature added
#> 
#> Notes:
#>   - [strictness] reject_init_stall is off inside every step: the model a step is handed starts at its own optimum, where "did not leave the initial estimates" (#751) is the expected outcome rather than a failed fit. It still applies to the pipeline's own start fit, and every other gate applies throughout
#>   - the retries pass on `02-iivsearch-retries` did not improve the fit (OFV -286.004 against -286.004); the selected model was kept
#>   - the retries pass on `03-ruvsearch-retries` did not improve the fit (OFV -286.004 against -286.004); the selected model was kept

The pipeline improved nothing: warfarin_amd.ferx is already the model these three steps would choose, so each step re-selected its own input and the final OFV equals the starting one. That is a useful result to see printed rather than hidden – the step table carries each step’s criterion before and after, its candidate count, the strictness verdict and its wall clock, so a step that “won” by having its siblings rejected looks different from one that found something. summary() adds every candidate of every step, and the same two tables are written to the run directory as steps.csv and candidates.csv.

Getting the result out of an AMD run takes a little care, because two fields both look final and are not the same thing. $final_model is the model text the pipeline ended on, written to $final_model_path; the fitted object is $fit. $final_ofv and $d_ofv give the end point and the change from the start, negative being an improvement:

c(final_model = class(amd$final_model), fit = class(amd$fit))
#> final_model         fit 
#> "character"  "ferx_fit"
data.frame(input_ofv = amd$input_ofv, final_ofv = amd$final_ofv, d_ofv = amd$d_ofv)
#>   input_ofv final_ofv        d_ofv
#> 1 -286.0042 -286.0042 -4.14957e-12
basename(amd$final_model_path)
#> [1] "final.ferx"

Each step that ran also has an entry under $tools, named by its position and pipeline step. Watch the naming: the entry is keyed on the pipeline step, while its $directory is the tool that implemented it, and the two need not agree – 01-structural was run by modelsearch:

names(amd$tools)
#> [1] "01-structural" "02-iivsearch"  "03-residual"
vapply(amd$tools, function(step) step$directory, "")
#>    01-structural     02-iivsearch      03-residual 
#> "01-modelsearch"   "02-iivsearch"   "03-ruvsearch"
names(amd$tools[["01-structural"]])
#>  [1] "index"            "step"             "tool"             "directory"       
#>  [5] "status"           "reason"           "criterion"        "selected"        
#>  [9] "seconds"          "candidates"       "path"             "models"          
#> [13] "input_model_path" "final_model_path" "model_paths"

An entry is a plain list, not the object the tool would return on its own: it carries what the step decided (status, reason, criterion, selected, seconds), its slice of the candidates, and the engine’s own table read back from the run directory. Run a tool directly when you want its fit and its own options.

The pipeline can also be stated inline, with a shorter order and steps left out. strategy chooses which steps run and in what order, and ferx_amd_plan() will show you each without fitting anything:

strategies <- c("default", "reevaluation", "SIR", "SRI", "RSI")
data.frame(strategy = strategies, order = vapply(strategies, function(s) {
  paste(ferx_amd_plan(model = amd_ex$model, data = amd_ex$data,
                      search_space = "PERIPHERALS(0..1); IIV?(@PK, exp)",
                      strategy = s)$step, collapse = " -> ")
}, ""), row.names = NULL)
#>       strategy
#> 1      default
#> 2 reevaluation
#> 3          SIR
#> 4          SRI
#> 5          RSI
#>                                                                                                  order
#> 1                          structural -> iivsearch -> residual -> iovsearch -> allometry -> covariates
#> 2 structural -> iivsearch -> residual -> iovsearch -> allometry -> covariates -> iivsearch -> residual
#> 3                                                                  structural -> iivsearch -> residual
#> 4                                                                  structural -> residual -> iivsearch
#> 5                                                                  residual -> structural -> iivsearch

reevaluation is the default order with a second iivsearch and residual pass at the end. The three-letter names are shorter pipelines, not reorderings of the same six: they run the three steps their initials spell and leave out iovsearch, allometry and covariates entirely. skip drops a step from whichever order was chosen; retries_on says when the extra perturbed-restart pass runs:

amd_sir <- ferx_amd(model = amd_ex$model, data = amd_ex$data,
                    search_space = "PERIPHERALS(0..1); IIV?(@PK, exp)",
                    strategy = "SIR", skip = "residual", retries_on = "final",
                    retries = 0, directory = file.path(search_dir, "amd-sir"), progress = FALSE)
amd_sir$steps[, c("step", "status", "criterion", "candidates", "seconds")]
#>         step  status criterion candidates    seconds
#> 1 structural     ran bic_mixed          2 0.03448793
#> 2  iivsearch     ran   bic_iiv          8 0.07338490
#> 3   residual skipped      <NA>          0 0.00000000

Configuration sections with no R tool

A .ferxsearch file may carry a section addressed to a tool this package does not bind. The loader validates section names against a fixed vocabulary, which runs ahead of what ferx-r binds and, in the case of [structsearch], ahead of the engine itself; a name outside that list is an error. The file loads, and ferx warns rather than running a search that quietly ignores part of its own configuration:

unconsumed <- function(section, body) {
  path <- file.path(search_dir, paste0(gsub("\\[|\\]", "", section), ".ferxsearch"))
  writeLines(c(readLines(base$search), "", section, body), path)
  tryCatch(ferx_search_config(path), warning = conditionMessage)
}
unconsumed("[structsearch]", "  x = 1")
#> [1] "ferx_search_config: [structsearch] has no R tool in this package, so it is ignored - a search run from this file does what its other sections say."

[structsearch] is accepted vocabulary with no engine module yet, so it is reported without a remediation that would name a command nobody can run.

What the warning reports is the complement of the sections ferx-r has a tool for, not a list of known-bad names. So a section a later ferx-core adds is named from the day a file can carry it, and stops being named on the day a binding lands – which is what happened to [globalsearch], warned about until ferx_globalsearch() shipped and silent since. See the ferx-core tools pages for what the engine knows.

Options that matter

Function Arguments
ferx_bic() fit, type ("mixed", "fixed", "iiv", "random")
check_strictness() fit, require_converged, require_covariance, max_condition_number, max_correlation, reject_on_boundary, reject_init_stall
ferx_search_config() path
ferx_search_space() mfl (text, config or space), model, data
ferx_search_coverage() space (space, config or MFL text)
ferx_search_results() directory, partial (NULL prefers the complete table), type ("candidates", "models", "steps")

The search functions share the entry arguments model, data and config; the run arguments threads, retries, directory, resume and progress; and, wherever candidates are ranked rather than tested, search_space, rank and cutoff. A search is stated either by config or by the inline arguments, not both:

Function Search arguments
ferx_covsearch() algorithm ("scm-forward-then-backward" or "scm-forward"), p_forward, p_backward, max_steps, adaptive_scope_reduction
ferx_modelsearch() algorithm ("reduced_stepwise", "exhaustive_stepwise", "exhaustive"), iiv_strategy ("absorption_delay", "add_diagonal", "no_add")
ferx_ruvsearch() groups, p_value, skip (families not to test), max_iter (1 to 3), cwres_prescreen (screen candidates on the parent’s CWRES before refitting)
ferx_iivsearch() algorithm ("top_down_exhaustive", "bottom_up_stepwise", "simultaneous_stepwise", "skip"), correlation_algorithm (the block stage: "top_down_exhaustive" or "skip"), as_fullblock (a bottom-up candidate blocks every eta it carries), block_retries (extra starts per eta in the largest block)
ferx_iovsearch() column (the occasion column; defaults to the model’s iov_column, and disagreeing with it is an error), distribution ("same-as-iiv", "disjoint", "joint", "explicit"), groups (the blocks "explicit" names), block_retries
ferx_globalsearch() algorithm ("ga" or "exhaustive", matched exactly), iiv_strategy, max_models (the cap on an exhaustive grid), ga (a named list of [globalsearch.ga] knobs), penalties (a named list of [rank.penalties] charges)
ferx_amd(), ferx_amd_plan() strategy ("default", "reevaluation", "SIR", "SRI", "RSI"), skip (steps to leave out), retries_on ("all_final", "final", "skip")

Only a .ferxsearch file can carry a per-tool section ([modelsearch], [iivsearch], [covsearch], …); the inline form runs each step at its own defaults.

print() and summary() exist for the results of all of them: ferx_covsearch, ferx_modelsearch, ferx_ruvsearch, ferx_iivsearch, ferx_iovsearch, ferx_globalsearch and ferx_amd objects. ferx_search_config and ferx_search_space objects have print() methods.

Pitfalls

  • Keep search directories. The book writes them to a temporary directory. In a project, write them next to the analysis: they hold every candidate model, the tables and final.ferx, and they make resume possible.
  • ferx_search_results() needs the right type. The covariate and residual error searches write steps.csv, the structural search models.csv. Neither wrote a candidates.csv here, so the default type = "candidates" finds no table for these runs.
  • Compare like with like. OFV differences and likelihood ratio tests need the same data, the same estimation method and nested models.
  • Thresholds change the answer. Record the space, the significance levels and the criterion together with the selected model. The .ferxsearch file is that record.

Summary

Next: Parameter uncertainty quantifies the uncertainty of the selected covariate model.