Software Quality Metrics Model (SQMM) for Scale Agile Environments: Value Through a Balanced Scorecard Approach

Abstract

Software quality metrics play a role in driving continuous improvement, enabling organisations to make informed decisions, enhance performance, deliver business value, and improve customer satisfaction in complex agile environments. Software metrics are increasingly popular for quality assessment in large-scale agile software development. The quality assessment process is considered necessary throughout the life of a project to test software quality before the deployment phase and meet customer expectations. The use of scaled Agile methodologies is becoming increasingly popular across large organisations. Software quality metrics offer a systematic way to monitor and improve both product and process quality in Agile and other approaches. However, their effective use in large-scale agile environments remains a challenge due to varying team structures, development phases, and business expectations.

This thesis aimed to explore quality metrics in large-scale agile development, seeking to identify and classify the most relevant software quality metrics and how they contribute to quality assessment. The quality metrics identified were developed into a structured, research-informed model that integrates both product-based and value-based perspectives. The study also investigated the role of Artificial Intelligence (AI) in software quality assessment and its implications for ethical, practical, and organisational use.

A mixed-methods approach was used in this doctoral study, combining findings in the literature with qualitative interviews and quantitative survey data. Semi-structured interviews were conducted with 22 experts from 11 companies to identify the most commonly used quality metrics and their related procedures for selection and implementation. While a quantitative approach was employed through a survey of 53 Agile practitioners across various roles and industries. The survey gathered responses on the usage, frequency, and perceived importance of different software quality metrics aligned with ISO/IEC 25010:2023 characteristics and value-based perspectives. The study proposes a new research model that contains 27 of the most commonly used quality metrics in large-scale agile software development, mapped to the characteristics of ISO/IEC 25010:2023.

Many organisations are starting to integrate AI into their quality assessment process, using AI’s capabilities to improve accuracy and efficiency. AI's rapid advancements have transformed many industries, including software engineering, where AI is rapidly being used to improve software quality assessment. While AI's automation, predictive analytics, and continuous monitoring improve software quality assessment, it is yet to be known how to best apply AI use in software quality assessment practices. This thesis examines the use AI for software quality evaluation, focusing on AI-driven tools' ability to improve assessment effectiveness, efficiency, and adaptability. The mixed-methods approach was also used, using qualitative interviews and quantitative survey data, to examine how AI affects software quality evaluation in large-scale agile software development environments.

Based on the findings of this doctoral study, two novel models are proposed to help organisations select appropriate quality metrics for their IT projects. The first research-informed model is the Software Quality Metrics Model (SQMM), which consists of ISO/IEC 25010:2023 characteristics and Balanced Scorecard (BSC) perspectives. This model provided 72 software quality metrics that were identified by interview participants. The second software quality model contains 25 of the most commonly used quality metrics in large-scale agile software development, mapped to the characteristics of ISO/IEC 25010:2023 by using the Goal Question Metric (GQM) approach. The proposed models were evaluated by a panel of 12 IT experts with experience in Agile and software quality practices. The expert evaluation confirmed the models’ relevance, clarity, completeness, and practical applicability, while the feedback informed minor refinements to improve their usability and alignment with industry needs.

In addition to the novel software quality models, this doctoral study also identified five quality metrics implementation challenges, including a lack of expertise in quality metrics, integrating quality metrics into Agile workflows, limitations in available tools and technologies, and resistance. It is understood that organisations that strategically select and update their metrics are better positioned to meet both technical and customer expectations. Regarding the AI role in the quality assessment process, the results of this thesis show that AI has become an integral part of modern Quality Assurance (QA) approaches, with measurable improvements in QA efficiency have been uncovered, and that test automation and defect detection remain the most popular areas where AI is applied in software quality management practice. Further insights offer guidance for practitioners aiming to optimise quality measurement practices, while also contributing to the academic understanding of how metrics function in agile software development at scale, supporting future advancements in both industry and research.