A Reproducible, Provenance-Preserving Architecture for Sequential Intervention-Threshold Evaluation in Large Language Models
Abstract
Sequential evaluation of large language models (LLMs) needs an auditable way to analyse when intervention first occurs. The architecture presented here uses cumulative scenarios and reconstructs first-ACT/censoring and at-risk datasets from a master record for discrete-time event-history analysis. Its methodological contribution is the integration of established first-event reconstruction with a frozen instrument, independent current-state calls and serving provenance, while retaining later responses for audit. In a thirteen-model demonstration, 26,000 stage-level records yielded 25,999 decision-valid responses, 3,120 scenario-runs and 10,088 at-risk rows. The derived endpoint files and all 2,000 prompt coordinates were regenerated exactly from deposited records. Complete conditional hazards were stage-dependent but non-monotone; within-group linear trend estimates were OR 12.591 (95% CI 8.481–18.694) in Group A, 10.346 (9.292–11.518) in Group B and 4.377 (3.815– 5.022) in Group C, and a quadratic sensitivity confirmed nonlinearity. Later WAIT responses occurred in 21.05% of ACT-containing runs. The reconstruction checks establish analytical reproducibility of one realised first-intervention path under recorded serving conditions; they do not establish a stable latent threshold, normative safety validity or exact recreation of historical provider responses.References
[1] Allison, P.D. (1982) ‘Discrete-time methods for the analysis of event histories’, Sociological Methodology, 13, pp. 61–98. Available at: https://doi.org/10.2307/270718.
[2] Aloqalaa, M., Soiland-Reyes, S. and Goble, C. (2026) ‘A structured examination of reproducibility: a case study for HTS using the PRIMAD model and BioCompute Object’, Data Science Journal, 25, article 24. Available at: https://doi.org/10.5334/dsj-2026-024.
[3] Bai, G., Liu, J., Bu, X. et al. (2024) ‘MT-Bench-101: a fine-grained benchmark for evaluating large language models in multi-turn dialogues’, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 7421–7454. Available at: https://doi.org/10.18653/v1/2024.acl-long.401.
[4] Edmunds, S.C., Nogoy, N., Lan, Q. et al. (2026) ‘Integrating machine learning standards in disseminating machine learning research’, Data Science Journal, 25, article 1. Available at: https://doi.org/10.5334/dsj-2026-001.
[5] Gallifant, J., Afshar, M., Ameen, S. et al. (2025) ‘The TRIPOD-LLM reporting guideline for studies using large language models’, Nature Medicine, 31(1), pp. 60–69. Available at: https://doi.org/10.1038/s41591-024-03425-5.
[6] Green, D.M. and Swets, J.A. (1966) Signal detection theory and psychophysics. New York: Wiley.
[7] Kolesnikov, A. (2026a) ‘A color-coded urgency bar and intervention thresholds in sequential health decisions’, Journal of High School Science, 10(3), pp. 394–413. Available at: https://doi.org/10.64336/001c.166227.
[8] Kolesnikov, A. (2026b) ‘A reproducible computational pipeline for modelling sequential decision thresholds from stopping-rule data’, International Journal of Secondary Computing and Applications Research, 3(3), pp. 14–20. Available at: https://doi.org/10.67149/yhjs2024.5/3s7gdh1q.
[9] Kolesnikov, A. (2026c) A Reproducible, Provenance-Preserving Architecture for Sequential Intervention-Threshold Evaluation in Large Language Models: Reproducibility Materials. Open Science Framework. Available at: https://doi.org/10.17605/OSF.IO/T58R9.
[10] Kwan, W.-C., Zeng, X., Jiang, Y. et al. (2024) ‘MT-Eval: a multi-turn capabilities evaluation benchmark for large language models’, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 20153–20177. Available at: https://doi.org/10.18653/v1/2024.emnlp-main.1124.
[11] Liang, P., Bommasani, R., Lee, T. et al. (2023) ‘Holistic evaluation of language models’, Transactions on Machine Learning Research. Available at: https://openreview.net/forum?id=iO4LZibEqW (Accessed: 16 September 2026).
[12] Liu, X., Yu, H., Zhang, H. et al. (2024) ‘AgentBench: evaluating LLMs as agents’, The Twelfth International Conference on Learning Representations (ICLR 2024). Available at: https://openreview.net/forum?id=zAdUB0aCTQ (Accessed: 16 September 2026).
[13] Lu, J., Holleis, T., Zhang, Y. et al. (2025) ‘ToolSandbox: a stateful, conversational, interactive evaluation benchmark for LLM tool use capabilities’, Findings of the Association for Computational Linguistics: NAACL 2025, pp. 1160–1183. Available at: https://doi.org/10.18653/v1/2025.findings-naacl.65.
[14] Magar, I. and Schwartz, R. (2022) ‘Data contamination: from memorization to exploitation’, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 157–165. Available at: https://doi.org/10.18653/v1/2022.acl-short.18.
[15] Royal College of Physicians (2017) National Early Warning Score (NEWS) 2: standardising the assessment of acute-illness severity in the NHS. Updated report of a working party. London: Royal College of Physicians. Available at: https://www.rcp.ac.uk/resources/national-early-warning-score-news-2/ (Accessed: 16 September 2026).
[16] Sainz, O., Campos, J.A., García-Ferrero, I. et al. (2023) ‘NLP evaluation in trouble: on the need to measure LLM data contamination for each benchmark’, Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 10776–10787. Available at: https://doi.org/10.18653/v1/2023.findings-emnlp.722.
[17] Sclar, M., Choi, Y., Tsvetkov, Y. et al. (2024) ‘Quantifying language models’ sensitivity to spurious features in prompt design or: how I learned to start worrying about prompt formatting’, The Twelfth International Conference on Learning Representations (ICLR 2024). Available at: https://openreview.net/forum?id=RIu5lyNXjT (Accessed: 16 September 2026).
[18] Singer, J.D. and Willett, J.B. (2003) Applied longitudinal data analysis: modeling change and event occurrence. New York: Oxford University Press. Available at: https://doi.org/10.1093/acprof:oso/9780195152968.001.0001.
[19] Srivastava, A., Rastogi, A., Rao, A. et al. (2023) ‘Beyond the imitation game: quantifying and extrapolating the capabilities of language models’, Transactions on Machine Learning Research. Available at: https://openreview.net/forum?id=uyTL5Bvosj (Accessed: 16 September 2026).
[20] Wald, A. (1945) ‘Sequential tests of statistical hypotheses’, Annals of Mathematical Statistics, 16, pp. 117–186. Available at: https://doi.org/10.1214/aoms/1177731118.
[21] Yao, S., Shinn, N., Razavi, P. et al. (2025) ‘tau-bench: a benchmark for tool-agent-user interaction in real-world domains’, The Thirteenth International Conference on Learning Representations (ICLR 2025). Available at: https://openreview.net/forum?id=roNSXZpUDN (Accessed: 16 September 2026).
DOI:
https://doi.org/10.31449/inf.v50i15.15648Keywords:
large language models, sequential evaluation, intervention threshold, event-history data, reproducibility, provenanceDownloads
Published
Issue
Section
License
Authors retain copyright in their work. By submitting to and publishing with Informatica, authors grant the publisher (Slovene Society Informatika) the non-exclusive right to publish, reproduce, and distribute the article and to identify itself as the original publisher.
All articles are published under the Creative Commons Attribution license CC BY 3.0. Under this license, others may share and adapt the work for any purpose, provided appropriate credit is given and changes (if any) are indicated.
Authors may deposit and share the submitted version, accepted manuscript, and published version, provided the original publication in Informatica is properly cited.
