An Experimental Evaluation of Large Language Models in Supporting the DEX Multi-Criteria Decision-Making Process

Abstract

We experimentally assessed the capabilities of two mainstream artificial intelligence chatbots, ChatGPT and DeepSeek, to support the multi-criteria decision-making process. Specifically, we focused on using the method DEX (Decision EXpert) and investigated their performance in all stages of DEX model development and utilization. The results indicate that these tools may substantially contribute in the difficult stages of collecting and structuring decision criteria, and collecting data about decision alternatives. However, at the current stage of development, the support for the whole multi-criteria decision-making process is still lacking, mainly due to occasionally inconsistent and erroneous execution of methodological steps. To leverage the strengths of both approaches, we also propose a hybrid workflow for DEX model development that begins in the LLM and continues in the specialized DEXiWin software.

References

Bana e Costa, C., De Corte, J.-M., Vansnick, J.-C. (2003): MACBETH (Overview of MACBETH multicriteria decision analysis approach). International Journal of Information Technology & Decision Making 11:02, 359–387. https://doi.org/10.1142/S0219622012400068.

Banerjee, S., Agarwal, A., Singla, S. (2024). LLMs will always hallucinate, and we need to live with this. https://doi.org/10.48550/arXiv.2409.05746.

Bansal, A. (2024): Comprehensive Study on LLM Agents. International Journal of All Research Education & Scientific Methods 12(8):2455-6211.

Bohanec, M., Rajkovič, V., Bratko, I., Zupan, B., Žnidaršič, M. (2013): DEX methodology: Three decades of qualitative multi-attribute modelling. Informatica 37, 49–54.

Bohanec, M. (2022): DEX (Decision EXpert): A qualitative hierarchical multi-criteria method. In: Multiple Criteria Decision Making (ed. Kulkarni, A.J.), Studies in Systems, Decision and Control 407, Singapore: Springer, https://doi.org/10.1007/978-981-16-7414-3_3.

Bohanec, M. (2024): DEXiWin: DEX Decision Modeling Software, User’s Manual, Version 1.2. Ljubljana: Institut Jožef Stefan, Delovno poročilo IJS DP-14747. https://dex.ijs.si/dexisuite/dexiwin.html.

Bohanec, M. (2025): Using large language models for creating, modifying and interpreting decision tables in multi-criteria method DEX. CECIIS 2025, 36th Central European Conference on Information and Intelligent Systems, Varaždin, Croatia, 651–658.

Bohanec, M., Rajkovič, U., Rajkovič, V (2025): Utilizing large language models for supporting multi-criteria decision modelling method DEX. IS 2025, Proc. of the 28th International Multiconference Information Society, Volume A, Ljubljana, Slovenia. 19–22.

Bray, T., Paoli, J., Sperberg, C.M., Maler, E., Yergeau, F. (2008): Extensible Markup Language (XML) 1.0 (Fifth Edition). https://www.w3.org/TR/REC-xml/.

Hämäläinen, R.P., Luoma, J., Saarinen, E. (Eds.). (2016): Behavioral Operational Research: A Capabilities Approach. Cham: Palgrave Macmillan. https://doi.org/10.1007/978-3-030-25405-6.

Ishizaka, A., Nemery, P. (2013): Multi-criteria decision analysis: Methods and software. Chichester: Wiley. ISBN: 978-1-119-97407-9.

Kamath, U., Keenan, K., Somers, G., Sorenson, S. (2024): Large Language Models: A Deep Dive: Bridging Theory and Practice. Springer, 506p, ISBN-13 978-3031656460.

Kulkarni, A.J. (Ed.) (2022): Multiple Criteria Decision Making. Studies in Systems, Decision and Control 407, Singapore: Springer, https://doi.org/10.1007/978-981-16-7414-3.

Radovanović, S., Delibašić, B., Vukanović, S. (2024): Combining LLM and DIDEX method to predict Internal Migrations in Serbia. Proc. 24th International Conference on Group Decision and Negotiation & 10th International Conference on Decision Support System Technology (GDN ICDSST 2024), Vol. 1: Technology as a support tool (Eds. S.P. Duarte, P. Zaraté, A. Lobo, B. Delibašić, T. Wachowicz, M.C. Ferreira), University of Porto.

Saaty, T.L., Vargas, L.G. (2012): Models, Methods, Concepts & Applications of the Analytic Hierarchy Process. Springer, US, New York, https://doi.org/10.1007/978-1-4614-3597-6.

Svoboda, I., Lande, D. (2024): Enhancing multi-criteria decision analysis with AI: Integrating analytic hierarchy process and GPT-4 for automated decision support. https://arxiv.org/abs/2402.07404.

Wang, H., Zhang, F., Mu, C. (2025). One for all: A general framework of LLMs-based multi-criteria decision making on human expert level. arXiv e-prints, 2502.15778. https://arxiv.org/abs/2502.15778.

Wang, L., Ma, C., Feng, X. et al. A survey on large language model based autonomous agents. Frontiers of Computer Science 18, 186345. https://doi.org/10.1007/s11704-024-40231-1.

Yuan, W., Liu, P., Gallé, M (2024): LLMCRIT: Teaching large language models to use criteria. Findings of the Association for Computational Linguistics: ACL 2024, 7929–7960, https://doi.org/10.18653/v1/2024.findings-acl.472.

Authors

  • Marko Bohanec
  • Uroš Rajkovič
  • Vladislav Rajkovič

DOI:

https://doi.org/10.31449/inf.v49i4.12884

Downloads

Published

12/01/2025

How to Cite

Bohanec, M., Rajkovič, U., & Rajkovič, V. (2025). An Experimental Evaluation of Large Language Models in Supporting the DEX Multi-Criteria Decision-Making Process. Informatica, 49(4). https://doi.org/10.31449/inf.v49i4.12884

Issue

Vol. 49 No. 4 (2025)

Section

Regular papers

License

Authors retain copyright in their work. By submitting to and publishing with Informatica, authors grant the publisher (Slovene Society Informatika) the non-exclusive right to publish, reproduce, and distribute the article and to identify itself as the original publisher.

All articles are published under the Creative Commons Attribution license CC BY 3.0. Under this license, others may share and adapt the work for any purpose, provided appropriate credit is given and changes (if any) are indicated.

Authors may deposit and share the submitted version, accepted manuscript, and published version, provided the original publication in Informatica is properly cited.