Abstract. This paper is a continuation of the multi-part study devoted to the application of large language models (LLMs) in decision support systems (DSSs). Part II considers the issues of measuring the quality of a hybrid DSS that includes a human decision-maker (DM) and an LLM as an assistant. Many criteria for assessing the quality of a hybrid DSS and an expert assessment procedure are proposed. A key feature of the expert assessment of the system’s quality is a structural model of a reasonable expert; this model is used to carry out a quasi-experiment for assessing the hybrid system. The assessment is performed within Five Whys, a well-known problem-solving methodology. An example of the operation and assessment of a hybrid DSS is provided, and the experiment results are analyzed. The new method for measuring quality and assessing the DM’s satisfaction, based on the structural model of a reasonable expert and a quasi-experiment, can be useful at the development stage of a DSS that incorporates an LLM. This method yields high-quality assessments of hybrid DSSs while reducing the time and cost of assessment without engaging expert groups.
Keywords: decision support, large language model (LLM), hybrid system, assessment criteria, structural model of a reasonable expert.
Application of Large Language Models in Decision Support Systems. Part II: Measurement and Assessment of Decision-Maker’s Satisfaction