Skip to main content

Diagnosing the model

DATA → MODEL → ESTIMATE ⇄ EVALUATE → SIMULATE → REPORT

Where you are

A converged fit with a sensible OFV is only a candidate. Before trusting it, check whether the model describes the data: are predictions unbiased, are residuals structureless, are the random effects well behaved and informed by the data, and are the parameters identifiable? This chapter works through these checks for the base model. It uses the fit object’s tables and plain ggplot2, plus the simulation-based NPDE.

The data

base <- ferx_example("two_cpt_oral_base")
fit <- ferx_fit(base$model, base$data, verbose = FALSE)

Minimal runnable call

Most diagnostics come straight from fit$sdtab, which has one row per observation:

head(fit$sdtab)
#>   ID TIME     DV     PRED    IPRED      CWRES      IWRES   EBE_OFV N_OBS TAFD
#> 1  1  0.5 3.5262 2.232568 3.539015  0.8779360 -0.1847887 -40.04274    10  0.5
#> 2  1  1.0 4.1950 3.133090 4.134308  0.9917834  0.7491377 -40.04274    10  1.0
#> 3  1  2.0 3.3759 3.294108 3.438460 -1.5463846 -0.9284538 -40.04274    10  2.0
#> 4  1  4.0 2.1090 2.333916 2.086008  0.1242995  0.5624627 -40.04274    10  4.0
#> 5  1  6.0 1.4915 1.641966 1.485324  0.1433674  0.2121698 -40.04274    10  6.0
#> 6  1  8.0 1.1890 1.267651 1.212711 -0.8376236 -0.9977522 -40.04274    10  8.0
#>   TAD
#> 1 0.5
#> 2 1.0
#> 3 2.0
#> 4 4.0
#> 5 6.0
#> 6 8.0

PRED is the population prediction (all etas zero). IPRED is the individual prediction at the subject’s estimated random effects. CWRES and IWRES are the conditional weighted and individual weighted residuals. TAFD and TAD are time after first dose and time after dose.

Reading the result

The fit’s verdict

Before any plot, ask whether the fit can be used as it stands. fit$converged says that the optimizer stopped normally and, for FOCE, FOCEI, Laplace / AGQ and Gauss-Newton fits, at a stationary point: when the covariance step has an unfloored R matrix, ferx computes the Newton decrement at the estimate, the drop in objective a Newton step would still make, and reports converged = FALSE with a W_NON_STATIONARY_OPTIMUM warning when it is above one OFV unit (ferx-core #2785). fit$status adds what else qualifies the result: the evaluation budget, the critical warnings, the covariance step and a variance collapsed onto its guard that the step could not hold (ferx-core #2799). It is "converged", "converged_with_issues" or "not_converged", fit$status_label spells it as the STATUS line of print() does, and fit$status_reasons lists why, in a fixed order:

fit[c("converged", "status", "status_label", "status_reasons")]
#> $converged
#> [1] TRUE
#> 
#> $status
#> [1] "converged"
#> 
#> $status_label
#> [1] "CONVERGED"
#> 
#> $status_reasons
#> character(0)
fit$held_at_bound
#> [1] parameter kind      index     reason    class    
#> <0 rows> (or 0-length row.names)

The base fit is CONVERGED, with no reasons and no parameter held. A fit whose evaluation budget runs out is not. The warfarin example with method = "focei" and maxiter = 2 stops early (Initial estimates and a first fit):

warf <- ferx_example("warfarin")
# suppress the R warnings that report overriding the model file's method and maxiter
fit_budget <- suppressWarnings(ferx_fit(warf$model, warf$data, method = "focei",
                                        settings = list(maxiter = 2), verbose = FALSE))
fit_budget[c("converged", "status_label", "status_reasons")]
#> $converged
#> [1] FALSE
#> 
#> $status_label
#> [1] "NOT CONVERGED"
#> 
#> $status_reasons
#> [1] "budget_spent" "convergence"

The reasons are listed for a fit that did not converge too, so they say what to fix: budget_spent, a larger maxiter, and then the critical convergence warning that reports it. A fit that converged but carries a reason is "converged_with_issues". A run with maxiter = 0, which only scores the given values, is "not_converged" with the reason evaluation_only. In all, the reasons are evaluation_only, budget_spent, the category of each critical warning (including the condition_number warning that ferx-r raises above 1000), covariance_failed, and collapsed_at_guard: <parameter> for a variance the covariance step could not hold. ferx_covariance() and ferx_load_fit() recompute the verdict.

fit$held_at_bound lists every parameter the covariance step gives no standard error for, one row each: its parameter name, its kind ("theta", "omega", "sigma" or "kappa"), its index within that kind, and the reason (ferx-core #2797). The reason is "profiled_bound" for a profiled level held on its declared bound (Model-based meta-analysis), or "collapsed_at_guard" for a variance or sigma that collapsed onto the floor ferx keeps it above, or a theta without a lower bound that sank to the internal 1e-10 floor. Such a parameter shows SE N/A in print() and NA in fit$estimates. When the covariance step is otherwise clean it holds the parameter as it holds a FIX one, and the other standard errors are conditional on it. The fifth column, class, is the class of a [mixture] per-class override (omega(k) or sigma(k)), whose row is labelled <base> (class <k>), and NA for every other parameter (ferx-core #2809). The base fit has no such row; Variability: random effects, residual error and IOV shows variances that collapse. A held parameter is not a reason by itself: a fit whose covariance step holds a collapsed variance stays "converged", with the parameter listed here, so read fit$held_at_bound beside the status (Model-based meta-analysis shows such a fit).

ImportantNeeds ferx-r newer than 0.4.0

fit$status, fit$status_label, fit$status_reasons and fit$held_at_bound do not exist in ferx-r 0.4.0. Its STATUS line shows converged and the categories of the critical warnings, and a parameter collapsed onto the guard gets a standard error. Its converged does not require a stationary point, so a fit reported converged there can be FALSE here, with W_NON_STATIONARY_OPTIMUM.

Observations against predictions

fit$sdtab |>
  tidyr::pivot_longer(c(PRED, IPRED), names_to = "prediction", values_to = "value") |>
  ggplot(aes(value, DV)) +
  geom_abline(linetype = "dashed") +
  geom_point(alpha = 0.4) +
  geom_smooth(se = FALSE, method = "loess", formula = y ~ x) +
  scale_x_log10() + scale_y_log10() +
  facet_wrap(~prediction) +
  labs(x = "Prediction", y = "Observation")
Figure 7.1: Observations against population and individual predictions (log scales), with a smooth.

Points should scatter around the identity line, and the smooth should follow it. Scatter around PRED also includes between-subject variability, so it is wider than around IPRED.

Residuals

library(patchwork)
res_plot <- function(x, y, xlab) {
  ggplot(fit$sdtab, aes({{ x }}, {{ y }})) +
    geom_hline(yintercept = 0, linetype = "dashed") +
    geom_point(alpha = 0.4) +
    geom_smooth(se = FALSE, method = "loess", formula = y ~ x) +
    labs(x = xlab)
}
res_plot(TIME, CWRES, "Time (h)") + res_plot(PRED, CWRES, "PRED") + res_plot(IPRED, IWRES, "IPRED")
Figure 7.2: CWRES against time and against population prediction; IWRES against individual prediction.

Residuals should be centred on zero with no trend over time or predictions. Most should fall within ±2, and the spread should not change with the prediction. Under a correct model CWRES are roughly standard normal:

ggplot(fit$sdtab, aes(sample = CWRES)) +
  stat_qq(alpha = 0.5) +
  stat_qq_line() +
  labs(x = "Theoretical quantile", y = "CWRES")
Figure 7.3: Normal quantile plot of CWRES.

The fit also tests the residuals for serial correlation within subjects. fit$dw_statistic is the Durbin-Watson statistic of the individual residuals (3.06 here) and fit$iwres_lag1_r their lag-1 correlation (-0.53). Both pair consecutive observations of one subject. An occasion is a value of an IOV column, an occasion an iov_occasion rule derives, or the innermost level cell of a nested model. A subject’s series is cut where its occasions run in parallel and the clock restarts, such as study arms or level cells, and, under a correlation ... by line, at each residual block. Occasions that follow one another in time, such as dosing intervals or study periods, stay one series (ferx-core #2829). The Durbin-Watson denominator counts those pairs, so independent residuals give 2 however short each subject’s series is (ferx-core #2798). Values far from 2 and 0 respectively trigger the dw_autocorrelation warning this fit carries. The statistic reads in two directions, and they mean opposite things:

dw_statistic Reading Usually points at
below 1.5 Positive autocorrelation Missing dynamics: a delay the model does not have, an absorption phase too simple, variability between occasions
1.5 to 2.5 No serial correlation worth acting on –
above 2.5 Negative autocorrelation Over-parameterisation, or a residual error model that does not fit the data
NoteChanged after ferx-r 0.4.0

fit$dw_statistic and fit$iwres_lag1_r cut a subject’s residual series at parallel occasions (and, under a residual correlation line, at each residual block), and the Durbin-Watson denominator is normalised by the number of pairs, so independent residuals give 2 however short the series. ferx-r 0.4.0 computes the classic form, so its values differ, and so can which fits get the dw_autocorrelation warning. The 1.5 and 2.5 thresholds are the same.

ferx_get_warnings(fit)
#> ferx fit warnings  (two_cpt_oral_base)
#> -------------------------------------------------
#> [WARNING]  general
#>             gradient = fd also changed the outer optimizer: with optimizer = auto
#>             left in place it resolved to bobyqa rather than the nlopt_lbfgs this
#>             model would have used under gradient = auto, because auto picks the
#>             derivative-free optimizer when there is no exact analytic gradient to
#>             feed a gradient-based one. Two factors moved, not one — if you are
#>             comparing an fd arm against an auto arm, pin optimizer = bobyqa in
#>             both so the difference you measure is the gradient.
#> 
#> [WARNING]  dw_autocorrelation
#>             Negative IWRES autocorrelation detected (Durbin-Watson = 3.06).
#>             Possible over-parameterization or misspecified error model.
#>             Negative IWRES autocorrelation suggests over-parameterisation or a
#>             misspecified error model. Consider removing a parameter or
#>             simplifying the residual model.
#>             details: durbin_watson, iwres_lag1_autocorr (ferx_get_warnings(fit,
#>             as_df = TRUE)$details)
#> 
#> -------------------------------------------------
#> 0 CRITICAL   2 WARNING   0 INFO

The first entry is the note about the model file’s gradient = fd that Initial estimates and a first fit explains. The second is the residual warning: ferx suggests over-parameterisation or a misspecified residual error model as possible causes. Model development and selection tests alternative residual error models systematically with ferx_ruvsearch().

Individual fits

fit$sdtab |>
  filter(ID %in% unique(ID)[1:9]) |>
  ggplot(aes(TIME)) +
  geom_point(aes(y = DV)) +
  geom_line(aes(y = IPRED)) +
  geom_line(aes(y = PRED), linetype = "dashed") +
  scale_y_log10() +
  facet_wrap(~ID) +
  labs(x = "Time (h)", y = "Concentration")
Figure 7.4: Observations (points) with individual (solid) and population (dashed) predictions for the first nine subjects.

These lines connect predictions at the observation times only. For smooth curves between samples, predict on a dense time grid with ferx_predict() (Simulating scenarios).

Random effects and shrinkage

fit$ebe_etas holds each subject’s estimated random effects:

fit$ebe_etas |>
  tidyr::pivot_longer(-ID, names_to = "eta") |>
  ggplot(aes(value)) +
  geom_histogram(bins = 12) +
  geom_vline(xintercept = 0, linetype = "dashed") +
  facet_wrap(~eta, nrow = 1)
Figure 7.5: Distribution of the estimated random effects.

Shrinkage measures how strongly the individual estimates are pulled towards zero. High eta shrinkage means the data carry little information about that random effect for individual subjects, which makes eta-based plots (including the covariate plots below) unreliable for that parameter:

data.frame(eta = fit$eta_names, shrinkage_pct = round(100 * fit$shrinkage_eta, 1))
#>           eta shrinkage_pct
#> ETA_CL ETA_CL           0.2
#> ETA_V1 ETA_V1           2.7
#> ETA_Q   ETA_Q           2.3
#> ETA_V2 ETA_V2           2.2
#> ETA_KA ETA_KA           3.5
c(eps_shrinkage_pct = round(100 * fit$shrinkage_eps, 1))
#> eps_shrinkage_pct 
#>              27.8

fit$eta_normality holds a Shapiro-Wilk test per eta. A flag marks p < 0.05:

fit$eta_normality
#>      eta     W  p_val flag
#> 1 ETA_CL 0.964 0.3875     
#> 2 ETA_V1 0.975 0.6781     
#> 3  ETA_Q 0.986 0.9535     
#> 4 ETA_V2 0.977 0.7472     
#> 5 ETA_KA 0.978 0.7737

Random effects and covariates

fit$eta_cov correlates each eta with each subject-level numeric covariate in the dataset, sorted by descending |r|, and marks the pairs worth following up in its flag column at |r| >= 0.3. It is a quick first look for covariate relationships, and it is NULL when the model declares no etas, the dataset carries no usable numeric covariate, or the data file can no longer be read from fit$data_path:

fit$eta_cov
#>       eta covariate      r  p_val flag
#> 1  ETA_CL      CRCL  0.468 0.0090  [!]
#> 2  ETA_CL        WT  0.382 0.0372  [!]
#> 3  ETA_V1        WT  0.305 0.1010  [!]
#> 4  ETA_KA        WT  0.133 0.4828     
#> 5   ETA_Q        WT  0.117 0.5376     
#> 6  ETA_V2      CRCL  0.086 0.6509     
#> 7  ETA_V2        WT  0.082 0.6675     
#> 8  ETA_V1      CRCL  0.062 0.7465     
#> 9   ETA_Q      CRCL  0.045 0.8130     
#> 10 ETA_KA      CRCL -0.003 0.9894
with(fit$eta_cov, range(abs(r)[nzchar(trimws(flag))]))
#> [1] 0.305 0.468
subjects <- read.csv(base$data, na.strings = ".") |> distinct(ID, WT, CRCL)
fit$ebe_etas |>
  mutate(ID = as.integer(ID)) |>
  left_join(subjects, by = "ID") |>
  tidyr::pivot_longer(c(WT, CRCL), names_to = "covariate", values_to = "value") |>
  ggplot(aes(value, ETA_CL)) +
  geom_hline(yintercept = 0, linetype = "dashed") +
  geom_point() +
  geom_smooth(method = "lm", formula = y ~ x, se = FALSE) +
  facet_wrap(~covariate, scales = "free_x") +
  labs(x = NULL)
Figure 7.6: Clearance random effect against body weight and creatinine clearance.

ETA_CL has the strongest correlations here, with CRCL (r = 0.468) and WT (r = 0.382). That motivates the covariate search in Model development and selection. Covariate modeling covers covariate screening tools in more detail.

Parameter identifiability

The covariance step yields the parameter correlation matrix. Correlations near ±1 point to parameters the data cannot separate:

as.data.frame(as.table(fit$cor_matrix)) |>
  ggplot(aes(Var1, Var2, fill = Freq)) +
  geom_tile() +
  scale_fill_gradient2(limits = c(-1, 1)) +
  labs(x = NULL, y = NULL, fill = "r") +
  theme(axis.text.x = element_text(angle = 45, hjust = 1))
Figure 7.7: Correlation matrix of the parameter estimates.
c(max_abs_correlation = fit$max_abs_correlation, condition_number = fit$condition_number)
#> max_abs_correlation    condition_number 
#>          0.04693733          1.20656195

The condition number is the ratio of the largest to the smallest eigenvalue of this correlation matrix (fit$eigenvalues). ferx flags values above 1000.

NPDE

Normalized prediction distribution errors (NPDE) compare each observation with its distribution simulated under the fitted model. Unlike CWRES, they do not rely on a linearisation, and under a correct model they follow a standard normal distribution. ferx_calc_npde() computes them after the fit and returns the fit with NPDE and NPD columns added to sdtab:

fit <- ferx_calc_npde(fit, nsim = 1000, seed = 1)
summary(fit$sdtab[, c("NPDE", "NPD")])
#>       NPDE               NPD           
#>  Min.   :-3.29053   Min.   :-2.652070  
#>  1st Qu.:-0.67135   1st Qu.:-0.687926  
#>  Median :-0.01003   Median :-0.006267  
#>  Mean   : 0.03570   Mean   : 0.048408  
#>  3rd Qu.: 0.70953   3rd Qu.: 0.640349  
#>  Max.   : 2.57583   Max.   : 3.290527
(ggplot(fit$sdtab, aes(TIME, NPDE)) +
   geom_hline(yintercept = c(-1.96, 0, 1.96), linetype = c("dotted", "dashed", "dotted")) +
   geom_point(alpha = 0.4) +
   geom_smooth(se = FALSE, method = "loess", formula = y ~ x) +
   labs(x = "Time (h)")) +
  (ggplot(fit$sdtab, aes(sample = NPDE)) + stat_qq(alpha = 0.5) + stat_qq_line() +
     labs(x = "Theoretical quantile", y = "NPDE"))
Figure 7.8: NPDE against time and their normal quantile plot.

nsim must exceed each subject’s number of observations; subjects that don’t get NA NPDE. seed makes the result reproducible. model and data default to the paths stored in the fit. NPDE can also be computed during the fit with the npde_nsim and npde_seed settings:

Table 7.1: NPDE settings.
Setting Values Default Description
npde_nsim integer ≥ 0 0 Number of Monte-Carlo replicates per subject used to compute the simulation-based NPDE/NPD diagnostics after the fit.
npde_seed integer — RNG seed for the NPDE/NPD simulation, for reproducible diagnostics.

Options that matter

Function Argument Meaning
ferx_calc_npde() fit The fit to evaluate
nsim Monte Carlo replicates per subject (default 1000)
seed Random seed (default: the engine’s)
model, data Override the model and data paths stored in the fit
ferx_xpose() fit Fit to convert
backend "xpose" (default) or "xpose4"
continuous, categorical Override which covariates are treated as continuous or categorical
runno Run number stored on the object
iterations Include the per-parameter optimizer trace (needs optimizer_trace = TRUE)

Using xpose

ferx_xpose() converts a fit into a data object for the xpose (or xpose4) diagnostics packages, which provide ready-made goodness-of-fit, parameter and covariate plots. It maps sdtab, the individual parameters, the random effects and the covariates to xpose’s tables. xpose is not a dependency of this book, so the code is not run here:

fit_trace <- ferx_fit(base$model, base$data, optimizer_trace = TRUE, verbose = FALSE)
xpdb <- ferx_xpose(fit_trace, backend = "xpose", runno = 1, iterations = TRUE)
xpose::dv_vs_ipred(xpdb)
xpose::prm_vs_iteration(xpdb)   # parameter values over the optimizer trace

Pitfalls

  • Shrinkage limits eta-based diagnostics. When eta shrinkage is high, eta histograms and eta-covariate plots mostly show the prior, not the data.
  • Small negative shrinkage is possible. ETA_CL shows 0.2% here; this is not an error.
  • The plotted lines are not the model curve. sdtab has predictions only at observation times. Use ferx_predict() on a dense grid for smooth individual curves.
  • fit$eta_cov is a screen, not a test of a covariate model. Confirm relationships by fitting covariate models (Model development and selection).

Warnings you may see here

See the ferx-core warnings page for details.

  • dw_autocorrelation (warning): the individual residuals are autocorrelated (Durbin-Watson out of range). Revisit the residual error or structural model.
  • eta_normality (warning): an eta departs from normality.
  • eps_shrinkage (warning): residual shrinkage is high or negative. Sparse data for the error model; consider simplifying it.
  • eta_shrinkage (warning): one or more eta shrinkages exceed about 30%. The data poorly inform that variability.
  • boundary_estimate (warning): a theta estimate sits at a bound. Relax the bound or simplify the model.
  • inflated_rse (warning): a theta has a relative standard error above about 50%.
  • high_correlation (warning): a pair of thetas has |correlation| ≥ 0.95.
  • condition_number (critical): the covariance matrix is ill-conditioned. The model is likely over-parameterised.

Initial estimates and a first fit shows a fit that carries several of these at once.

Summary

  • Plot fit$sdtab: observations against predictions, residuals against time and predictions, and individual fits.
  • Check random effects with fit$ebe_etas, shrinkage and fit$eta_normality. Use fit$eta_cov for a first covariate screen.
  • Check identifiability with fit$cor_matrix and the condition number.
  • Add simulation-based NPDE with ferx_calc_npde(). Use ferx_xpose() if you work with xpose.

Next: Simulation-based evaluation: VPC evaluates the model by simulation.

TipReference