AI systems are increasingly used in areas where their decisions can have a direct impact on people and organisations. This makes it important not only to assess their technical performance, but also to understand whether they meet the needs and expectations of the people who use them. Through THEMIS 5.0 and FAITH, ATC has been working on this challenge from different perspectives, focusing in particular on the role of the user in the assessment of AI trustworthiness.
THEMIS 5.0 Human-centered Trustworthiness Optimisation in Hybrid Decision Support, is a 3-year HE RIA project that started in 2023 and that draws researchers and practitioners from diverse disciplines to implement an AI-driven, human-centred Trustworthiness-Optimisation ecosystem following an AI-driven risk assessment approach. In this project, ATC developed the Persona Analyser, a conversational agent that drives a conversation with the end-users to identify their trustworthiness preferences in terms of prioritising the AI system’s Fairness, Accuracy and Robustness. The Persona Analyser also acts as an intermediary, translating model-level technical concepts into non-technical, real-world examples, enabling users to express granular trustworthiness requirements in ways that can be mapped to AI model characteristics.
Figure1. ATC’s results from the THEMIS 5.0 and THEMIS projects
Building upon THEMIS 5.0, FAITH Fostering Artificial Intelligence Trust for Humans towards the optimization of trustworthiness through large-scale pilots in critical domains, is a 4-year HE IA project that started in 2024 and that to develop and validate a human-centric, trustworthiness optimization ecosystem, which enables measuring, optimizing and counteracting the risks associated with AI adoption and trustworthiness in critical domains by adopting a dynamic risk management approach following EU legislative instruments and ENISA guidelines. In this project, ATC developed TrustGuard, a checklist-based tool for implementing a risk-based methodology, compliant with ISO27005 and ISO42001, that aims to assist a risk assessor to assess the threats and estimate the underpinning risks of all components of an AI system during its entire lifecycle (design, development, deployment).
Challenges
Experience from THEMIS 5.0 and FAITH highlighted the importance of addressing the human factors of trustworthy AI in socio-technical systems and strengthening Human-in-the-Loop (HITL) components so that AI behaviour and optimisation priorities reflect user needs across sectors and use cases. Key challenges include:
- Technical Parameters interpretation: Non-technical users often struggle to understand model-level technical details and trade-offs. As a result, they cannot confidently prioritise technical improvements that best align the AI system with their needs.
- Use Case context: Translating the use-case context (e.g. constraints, requirements, and business targets) into measurable KPIs connected to model parameters is difficult. Since this process relies heavily on user input, it can be affected by bias and limited visibility of the interdependencies involved.
- GenAI: Many trustworthiness assessment frameworks and tools still face difficulties in analysing Generative AI systems. This gap needs to be addressed given the rapid uptake of GenAI tools and the distinct risk patterns they introduce (e.g., content reliability, hallucinations, misuse, and provenance).
Towards a human-centric AI Trustworthiness Assessment Framework
Building on knowledge and results from THEMIS 5.0 and FAITH, ATC aims to advance its HITL capabilities towards a human-centric AI Trustworthiness Assessment Framework. The proposed direction is an agentic approach that combines multiple AI paradigms (e.g., GenAI, machine learning, and symbolic AI) to integrate and operationalise different aspects of AI trustworthiness.
In particular, the approach will:
- Extract and represent domain knowledge from use-case documentation and datasets, capturing context, constraints, and objectives.
- Translate technical trustworthiness parameters into non-technical, real-world examples linked to business objectives, risks, and measurable KPIs, so users can express priorities in actionable terms.
This enables non-technical users to define use-case-specific technical requirements for improving trustworthiness, and to identify missing trustworthiness controls and improvement actions. By combining these outputs, the framework can also support a preliminary risk assessment, highlighting non-implemented controls that have the highest expected impact on trustworthiness for a given use case.
Figure2. Agentic Human-in-the-Loop component for a human-centric AI Trustworthiness Assessment Framework
Overall, the proposed framework builds on the work already carried out in THEMIS 5.0 and FAITH and focuses on making AI trustworthiness assessment more useful for non-technical users. The main goal is to better connect user needs and use-case requirements with technical parameters, controls and risks, so that the assessment can lead to more concrete improvement actions.