Intelligent Verification/Validation for Extended Reality Based Systems

Publications (full details on Zenodo)

Search-based Automated Play Testing of Computer Games: a model-based approach, SSBSE'2021 (to appear)

“Search-based Automated Play Testing of Computer Games: a model-based approach”

Ferdous, Raihana; Kifetew, Fitsum; Prandi, Davide; Prasetya, I. S. W. B; Shirzadehhajimahmood, Samira; Susi, Angelo

13th International Symposium on Search-Based Software Engineering, Bari, Italy, October 11-12, 2021

Computergametechnologyisincreasinglymorecomplexand applied in a wide variety of domains, beyond entertainment, such as training and educational scenarios. Testing games is a difficult task re- quiring a lot of manual effort since the interaction space in the game is very fine grained and requires a certain level of intelligence that cannot be easily automated. This makes testing a costly activity in the overall development of games.

This paper presents a model-based formulation of game play testing in such a way that search-based testing can be applied for test generation. An abstraction of the desired game behaviour is captured in an extended finite state machine (EFSM) and search-based algorithms are used to derive abstract tests from the model, which are then concretised into action sequences that are executed on the game under test.

The approach is implemented in a prototype tool EvoMBT. We carried out experiments on a 3D game to assess the suitability of the approach in general, and search-based test generation in particular. We applied 5 search algorithms for test generation on three different models of the game. Results show that search algorithms are able to achieve reasonable coverage on models: between 75% and 100% for the small and medium sized models, and between 29% and 56% for the bigger model. Mutation analysis shows that on the actual game application tests kill up to 99% of mutants. Tests have also revealed previously unknown faults.

Paper on Zenodo

A taxonomy of social roles for agents in games, ICEC'2021 (to appear)

“A taxonomy of social roles for agents in games”

Diogo Rato, Rui Prada

20th IFIP TC 14 International Conference, ICEC 2021, Coimbra, Portugal, November 2-5, 2021

Social agents have been used in games often, for example, to create a social dimension (e.g. the inhabitants of a village) or to provide challenges to players (e.g. the opponents players face). These agents have an essential role in the players’ experience, and, as such, their creation needs to carefully considered. In this paper we propose a taxonomy of social roles that agents can play in games as a step towards the formalization of the problem of the creation of social agents in games. We believe that this taxonomy can help researchers to reach some common ground on the subject and, therefore, promote common views of the research problems involved in the design and development of social agents for games. We discuss several open challenges in the creation of social agents for games and discuss some future directions of research that can be grounded on the analysis of the taxonomy. For instance, many of the social roles proposed are played by agents that do not have much agency or autonomy. Also, there is a large number of under-explored social roles in games at the moment. The taxonomy serves as inspiration to guide game design involving social interactions with game actors, promoting new kinds of gameplay built on the interactive space afforded by the social agents.

Paper on Zenodo

Using an Agent-Based Approach for Robust Automated Testing of Computer Games, A-Test'2021

“Using an Agent-Based Approach for Robust Automated Testing of Computer Games”

Shirzadehhajimahmood, Samira; Prasetya, I. S. W. B.; Dignum, Frank; Dastani, Mehdi; Keller, Gabriele

12th International Workshop on Automating TEST Case Design, Selection, and Evaluation (A-TEST ’21), August 23–24, 2021

Modern computer games typically have a huge interaction spaces and non-deterministic environments. Automation in testing can provide a vital boost in development and it further improves the overall software’s reliability and efficiency. Moreover, layout and game logic may regularly change during development or consecutive releases which makes it difficult to test because the usage of the system continuously changes. To deal with the latter, tests also need to be robust. Unfortunately, existing game testing approaches are not capable of maintaining test robustness. To address these challenges, this paper presents an agent-based approach for robust automated testing based on the reasoning type of AI.

Paper on Zenodo

Validating the plot of interactive Narrative games

“Validating the plot of interactive Narrative games”

Carolina Veloso, Rui Prada

The authoring of interactive dialogues in video games is an overwhelming and complex task for game writers. Developing an Interactive Narrative that balances authorial intent and players’ agency requires frequent in-depth testing. The limited range of tools to assist authors in verifying their story can limit the creation of more complex narratives. In this paper, we discuss the challenges of Interactive Story design and provide a model consisting of a set of metrics for testing interactive dialogues. Using this model, we developed a prototype for the Story Validator tool. This debugging tool allows game writers to experiment with different hypotheses and narrative properties in order to identify inconsistencies in the authored narrative and predict the output of different playthroughs with visual representation support. We conducted a series of user tests, Using the Story Validator, to investigate whether the tool adequately helps users identify problems that appear in the game’s story. The results showed that the tool enables content creators to easily test their stories, setting our model as a good step towards automated verification for assistance of authoring interactive narratives. 

Paper on Zenodo

iv4XR - Intelligent Verification/Validation for Extended Reality Based System, RCIS'2020

 “iv4XR – Intelligent Verification/Validation for Extended Reality Based System”
Wishnu Prasetya, Rui Prada, Tanja E. J. Vos, Fitsum Kifetew, Frank Dignum, Jason Lander, Jean-Yves Donnart, Alexandre Kazmierowski, Joseph Davidson, Fernando Pastor Ricos
RCIS’2020 – The 14th International Conference on Research Challenges in Information Science


“Extended Reality” (XR) systems are advanced interactive systems such as Virtual Reality (VR) and Augmented Reality (AR) systems. They have emerged in various domains, ranging from entertainment, cultural heritage, to combat train ing and mission critical applications. As the complexity of these systems keeps increasing, testing is getting more complex too. Current toolsets do no propose XR testing technology beyond rudimentary record and replay tools that only work for simple test scenarios. The following challenges need to be addressed:
1. Fine-grained interaction space. XR systems more accurately reflect the real world, so they allow fine grained, almost continuous, interactions. Also, XR worlds are inhabited by independent and dynamic entities simulating the corresponding real world entities. They interact with the user as well as with each other, and often lead to emerging behavior. These result in an interaction space far larger than in traditional interactive digital products, and intractable by existing automated testing approaches.
2. Assessing user experience (UX). High quality UX is very important for XR systems. If it is not smooth enough, is too boring, or too overwhelming, the users become unhappy, annoyed or can make mistakes. The latter is a serious concern for mission-critical XR applications. Since manually assessing the UX quality is very labour intensive, automation is needed. Unfortunately, existing tools are too simplistic and lack deeper models of human emotion and cognitive capabilities to be able to judge the different emotional states that an interaction event might evoke on users. Moreover, they are not able to deal with the diversity of users nor are they able to judge the progression of the UX that is built up over time as users engage in long term interactions.

 Paper on Zenodo

Tactical Agents for Testing Computer Games, EMAS'2020

“Tactical Agents for Testing Computer Games”
I. S. W. B. Prasetya, Mehdi Dastani, Rui Prada, Tanja E. J. Vos, Frank Dignum, Fitsum Kifetew
EMAS’2020 – Engineering Multi-Agent Systems workshop @ AAMAS’2020


Modern interactive software, such as computer games, employ complex user interfaces. Although these user interfaces make the games attractive and powerful, unfortunately they also make them extremely difficult to test. Not only do we have to deal with their functional complexity, but also the fine grained interactivity of their user interface blows up their interaction space, so that traditional automated testing techniques have trouble handling it. An agent-based testing approach offers an alternative solution: agents’ goal driven planning, adaptivity, and reasoning ability can provide an extra edge towards effective navigation in complex interaction space. This paper presents aplib, a Java library for programming intelligent test agents, featuring novel tactical programming as an abstract way to exert control over agents’ underlying reasoning-based behavior. This type of control is suitable for programming testing tasks. Aplib is implemented in such a way to provide the fluency of a Domain Specific Language (DSL). Its embedded DSL approach also means that aplib programmers will get all the advantages that Java programmers get: rich language features and a whole array of development tools.

 Paper on Zenodo

Adoption Dynamics and Societal Impact of AI Systems in Complex Networks, AIES'2020

“Adoption Dynamics and Societal Impact of AI Systems in Complex Networks”
Pedro M. Fernandes, Francisco C. Santos, Manuel Lopes
AIES’2020 – AAAI/ACM Conference on AI, Ethics, and Society


We propose a game-theoretical model to simulate the dynamics of AI adoption in adaptive networks. This formalism allows us to understand the impact of the adoption of AI systems for society as a whole, addressing some of the concerns on the need for regulation. Using this model we study the adoption of AI systems, the distribution of the different types of AI (from selfish to utilitarian), the appearance of clusters of specific AI types, and the impact on the fitness of each individual. We suggest that the entangled evolution of individual strategy and network structure constitutes a key mechanism for the sustainability of utilitarian and human-conscious AI. Differently, in the absence of rewiring, a minority of the population can easily foster the adoption of selfish AI and gains a benefit at the expense of the remaining majority.

Paper on Zenodo

Deploying TESTAR to Enable Remote Testing in an Industrial CI Pipeline: A Case-Based Evaluation

“Deploying TESTAR to Enable Remote Testing in an Industrial CI Pipeline: A Case-Based Evaluation”

Pastor Ricos, Fernando; Aho, Pekka; Vos, Tanja; Torres Boigues, Ismael; Calas Blasco, Ernesto; Martinez Martinez, Hector

Leveraging Applications of Formal Methods, Verification and Validation: Verification Principles, 2020

Companies are facing constant pressure towards shorter release cycles while still maintaining a high level of quality. Agile development, continuous integration and testing are commonly used quality assurance techniques applied in industry. Increasing the level of test automation is a key ingredient to address the short release cycles. Testing at the graphical user interface (GUI) level is challenging to automate, and therefore many companies still do this manually. To help find solutions for better GUI test automation, academics are researching scriptless GUI testing to complement the script-based approach. In order to better match industrial problems with academic results, more academia-industry collaborations for case-based evaluations are needed. This paper describes such an initiative to improve, transfer and integrate an academic scriptless GUI testing tool TESTAR into the CI pipeline of a Spanish company Prodevelop. The paper describes the steps taken, the outcome, the challenges, and some lessons learned for successful industry-academia collaboration.

Paper on Zenodo

Agent-based Testing of Extended Reality Systems, ICST'2020

“Agent-based Testing of Extended Reality Systems”
Rui Prada, I. S. W. B. Prasetya, Fitsum Kifetew, Frank Dignum, Tanja E. J. Vos, Jason Lander, Jean-yves Donnart, Alexandre Kazmierowski, Joseph Davidson, Pedro M. Fernandes
ICST-2020 – IEEE Conference on Software Testing, Validation and Verification


Testing for quality assurance (QA) is a crucial step in the development of Extended Reality (XR) systems that typically follow iterative design and development cycles. Bringing automation to these testing procedures will increase the productivity of XR developers. However, given the complexity of the XR environments and the User Experience (UX) demands, achieving this is highly challenging. We propose to address this issue through the creation of autonomous cognitive test agents that will have the ability to cope with the complexity of the interaction space by intelligently explore the most prominent interactions given a test goal and support the assessment of affective properties of the UX by playing the role of users.

Paper on ZenodoPoster on ZenodoPoster on Zenodo

Navigation and Exploration in 3D-Game Automated Play Testing

“Navigation and Exploration in 3D-Game Automated Play Testing”

Prasetya, Wishnu; Voshol, Maurin; Tanis, Tom; Smits, Adam; Smit, Bram; van Mourik, Jacco; Klunder, Menno; Hoogmoed, Frank; Hinlopen, Stijn; van Casteren, August; van de Berg, Jesse; Prasetya, Naraenda; Shirzadehhajimahmood, Samira; Gholizadeh Ansari, Saba

International Workshop on Automating Test case Design, Selection and Evaluation, co-located with ESEC.FSE (ATEST), 9 November, 2020


To enable automated software testing, the ability to automatically navigate to a state of interest and to explore all, or at least sufficient number of, instances of such a state is fundamental. When test- ing a computer game the problem has an extra dimension, namely the virtual world where the game is played on. This world often plays a dominant role in constraining which logical states are reach- able, and how to reach them. So, any automated testing algorithm for computer games will inevitably need a layer that deals with navigation on a virtual world. Unlike e.g. navigating through the GUI of a typical web-based application, navigating over a virtual world is much more challenging. This paper discusses how concepts from geometry and graph-based path finding can be applied in the context of game testing to solve the problem of automated navigation and exploration. As a proof of concept, the paper also briefly discusses the implementation of the proposed approach.

Aplib: An Agent Programming Library for Testing Games

“Aplib: An Agent Programming Library for Testing Games”

Prasetya, Wishnu; Dastani, Mehdi

International Conference on Autonomous Agents and Multiagent Systems (AAMAS), Auckland, New Zealand, 9-13 May, 2020


Testing modern computer games is notoriously hard. Highly dynamic behavior, inherent non-determinism, and fine grained inter activity blow up their state space; too large for traditional auto- mated testing techniques. An agent-based testing approach offers an alternative as agents’ goal driven planning, adaptivity, and reasoning ability can provide an extra edge. This paper provides a summary of aplib, a Java library for programming intelligent test agents, featuring tactical programming as an abstract way to exert control on agents’ underlying reasoning based behavior. Aplib is implemented in such a way to provide the fluency of a Domain Specific Language (DSL) while still staying in Java, and hence aplib programmers will keep all the advantages that Java programmers get: rich language features and a whole array of development tools.

 Paper on Zenodo