Data leakage when partitioning the data

[arani11]

In the model that I developed, for the development and internal validation, a 90:10 train/ test split by person was used.

TRIPOD+AI states that if the data has been partitioned into training (including any hyperparameter tuning data) and test data, confirm that there has been no data leakage.

Is there a way to check or confirm that there was no data leakage?

Thanks

[arani11]

I would also appreciate your advice re if I am able to do/ confirm this (below, from TRIPOD+AI guidance) using the PLP package:

If the data contain multiple records or samples from the same individual, and if the data has been partitioned into training (including
any hyperparameter tuning data) and test data, confirm there has been no leakage of individuals across any of the partitioned data or if not, how describe how this was handled in the analysis (see item 12c)

[jreps]

You can load the result and check the prediction data.frame as it has a column telling you whether the row was used in the Train or Test set. Here is example code:

connectionDetails <- Eunomia::getEunomiaConnectionDetails()
Eunomia::createCohorts(connectionDetails)

modelDesign <- PatientLevelPrediction::createModelDesign(
targetId = 4,
outcomeId = 3,
restrictPlpDataSettings = PatientLevelPrediction::createRestrictPlpDataSettings(),
populationSettings = PatientLevelPrediction::createStudyPopulationSettings(
requireTimeAtRisk = FALSE,
riskWindowStart = 1,
riskWindowEnd = 365
),
covariateSettings = FeatureExtraction::createCovariateSettings(
useDemographicsGender = TRUE,
useDemographicsAgeGroup = TRUE,
useConditionGroupEraLongTerm = TRUE,
useDrugEraLongTerm = TRUE,
useMeasurementLongTerm = TRUE,
useProcedureOccurrenceLongTerm = TRUE
),
preprocessSettings = PatientLevelPrediction::createPreprocessSettings(),
modelSettings = PatientLevelPrediction::setLassoLogisticRegression(),
splitSettings = PatientLevelPrediction::createDefaultSplitSetting(
testFraction = 0.1, trainFraction = 0.9,
type = 'subject'
)
)

PatientLevelPrediction::runMultiplePlp(
databaseDetails = PatientLevelPrediction::createDatabaseDetails(
connectionDetails = connectionDetails,
cdmDatabaseSchema = 'main',
cdmDatabaseName = 'eunomia',
tempEmulationSchema = 'main',
cohortDatabaseSchema = 'main',
cohortTable = 'cohort',
outcomeDatabaseSchema = 'main',
outcomeTable = 'cohort',
cdmVersion = 5
),
modelDesignList = list(modelDesign),
saveDirectory = file.path(tempdir(), 'example_plp')
)

see folders

dir(file.path(tempdir(), 'example_plp'))

plpResult <- PatientLevelPrediction::loadPlpResult(file.path(tempdir(), 'example_plp', "Analysis_1", 'plpResult'))

code to check test and train had different subjectIds (this is the personId in OMOP CDM)

with 'subject' splitting this should be 0

sum(plpResult$prediction$subjectId[plpResult$prediction$evaluationType == 'Train'] %in%
plpResult$prediction$subjectId[plpResult$prediction$evaluationType == 'Test'])

can also check rowId has no overlap

sum(plpResult$prediction$rowId[plpResult$prediction$evaluationType == 'Train'] %in%
plpResult$prediction$rowId[plpResult$prediction$evaluationType == 'Test'])