Studentized bootstrap test of strength of scoreFn(yValues,model1Values) > scoreFn(yValues,model1Values).

Source: R/EmpiricalFns.R

Studentized bootstrap test of strength of scoreFn(yValues,model1Values) > scoreFn(yValues,model1Values) sampled with replacement.

resampleScoreModelPair(
  model1Values,
  model2Values,
  yValues,
  scoreFn,
  ...,
  na.rm = FALSE,
  returnScores = FALSE,
  nRep = 100,
  parallelCluster = NULL,
  sameSample = FALSE
)

Arguments

model1Values

numeric array of predictions (model to test).
model2Values

numeric array of predictions (reference model).
yValues

numeric/logical array of outcomes, dependent, or truth values
scoreFn

function with signature scoreFn(modelValues,yValues) returning scalar numeric score.
...

not used, forces later arguments to be bound by name.
na.rm

logical, if TRUE remove NA values
returnScores

logical if TRUE return detailed resampledScores.
nRep

integer number of repititions to perform.
parallelCluster

optional snow-style parallel cluster.
sameSample

logical if TRUE use the same sample in computing both scores during bootstrap replication (else use independent samples).

Value

summaries

Details

True confidence intervals are harder to get right (see "An Introduction to the Bootstrap", Bradely Efron, and Robert J. Tibshirani, Chapman & Hall/CRC, 1993.), but we will settle for simple p-value estimates.

Examples


set.seed(25325)
y <- 1:5
m1 <- c(1,1,2,2,2)
m2 <- c(1,1,1,1,2)
cor(m1,y)

#> [1] 0.8660254
cor(m2,y)

#> [1] 0.7071068
f <- function(modelValues,yValues) {
  if((sd(modelValues)<=0)||(sd(yValues)<=0)) {
    return(0)
  }
  cor(modelValues,yValues)
}
resampleScoreModelPair(m1,m2,y,f)

#> [1] "Studentized empirical test: is difference greater than zero on re-samples,  summary: e=n.s."