ATQ: Alarm time quality, an evaluation metric for assessing timely epidemic detection models within a school absenteeism-based surveillance system

Abstract

Model-based school absenteeism surveillance systems have been proposed to raise seasonal influenza epidemic alarms. Previous studies used metrics such as false alarm rate (FAR) and accumulated days delayed, for model evaluation and selection, however they were unable to optimize both alarm accuracy and timeliness. In this study, we developed a metric, alarm time quality (ATQ), that simultaneously evaluated both aspects by assessing alarms on a gradient, where alarms raised incrementally before or after an optimal time were informative, but penalized. Summary statistics of ATQ, average alarm time quality (AATQ) and first alarm time quality (FATQ), were used as model selection criterion. Alarms raised by ATQ and FAR-selected logistic regression models were compared. Daily school absenteeism and laboratory-confirmed influenza data collected by Wellington-Dufferin-Guelph Public Health was used for demonstration. A simulation study representative of Wellington-Dufferin-Guelph was conducted for further evaluation. ATQ-selected models were found to raise alarms that were timelier than the FAR-selected model.