Intelligent Verification/Validation for Extended Reality Based Systems
Publications (full details on Zenodo)
iv4XR - Intelligent Verification/Validation for Extended Reality Based System, RCIS'2020
“iv4XR – Intelligent Verification/Validation for Extended Reality Based System”
Wishnu Prasetya, Rui Prada, Tanja E. J. Vos, Fitsum Kifetew, Frank Dignum, Jason Lander, Jean-Yves Donnart, Alexandre Kazmierowski, Joseph Davidson, Fernando Pastor Ricos
RCIS’2020 – The 14th International Conference on Research Challenges in Information Science
“Extended Reality” (XR) systems are advanced interactive systems such as Vir tual Reality (VR) and Augmented Reality (AR) systems. They have emerged in various domains, ranging from entertainment, cultural heritage, to combat train ing and mission critical applications. As the complexity of these systems keeps increasing, testing is getting more complex too. Current toolsets do no propose XR testing technology beyond rudimentary record and replay tools that only work for simple test scenarios. The following challenges need to be addressed:
1. Fine-grained interaction space. XR systems more accurately reflect the real world, so they allow fine grained, almost continuous, interactions. Also, XR worlds are inhabited by independent and dynamic entities simulating the corresponding real world entities. They interact with the user as well as with each other, and often lead to emerging behavior. These result in an interaction space far larger than in traditional interactive digital products, and intractable by existing automated testing approaches.
2. Assessing user experience (UX). High quality UX is very important for XR systems. If it is not smooth enough, is too boring, or too overwhelming, the users become unhappy, annoyed or can make mistakes. The latter is a serious concern for mission-critical XR applications. Since manually assessing the UX quality is very labour intensive, automation is needed. Unfortunately, existing tools are too simplistic and lack deeper models of human emotion and cognitive capabilities to be able to judge the different emotional states that an interaction event might evoke on users. Moreover, they are not able to deal with the diversity of users nor are they able to judge the progression of the UX that is built up over time as users engage in long term interactions.
Tactical Agents for Testing Computer Games, EMAS'2020
“Tactical Agents for Testing Computer Games”
I. S. W. B. Prasetya, Mehdi Dastani, Rui Prada, Tanja E. J. Vos, Frank Dignum, Fitsum Kifetew
EMAS’2020 – Engineering Multi-Agent Systems workshop @ AAMAS’2020
Modern interactive software, such as computer games, employ complex user interfaces. Although these user interfaces make the games attractive and powerful, unfortunately they also make them extremely difficult to test. Not only do we have to deal with their functional complexity, but also the fine grained interactivity of their user interface blows up their interaction space, so that traditional automated testing techniques have trouble handling it. An agent-based testing approach offers an alternative solution: agents’ goal driven planning, adaptivity, and reasoning ability can provide an extra edge towards effective navigation in complex interaction space. This paper presents aplib, a Java library for programming intelligent test agents, featuring novel tactical programming as an abstract way to exert control over agents’ underlying reasoning-based behavior. This type of control is suitable for programming testing tasks. Aplib is implemented in such a way to provide the fluency of a Domain Specific Language (DSL). Its embedded DSL approach also means that aplib programmers will get all the advantages that Java programmers get: rich language features and a whole array of development tools.
Adoption Dynamics and Societal Impact of AI Systems in Complex Networks, AIES'2020
“Adoption Dynamics and Societal Impact of AI Systems in Complex Networks”
Pedro M. Fernandes, Francisco C. Santos, Manuel Lopes
AIES’2020 – AAAI/ACM Conference on AI, Ethics, and Society
We propose a game-theoretical model to simulate the dynamics of AI adoption in adaptive networks. This formalism allows us to understand the impact of the adoption of AI systems for society as a whole, addressing some of the concerns on the need for regulation. Using this model we study the adoption of AI systems, the distribution of the different types of AI (from selfish to utilitarian), the appearance of clusters of specific AI types, and the impact on the fitness of each individual. We suggest that the entangled evolution of individual strategy and network structure constitutes a key mechanism for the sustainability of utilitarian and human-conscious AI. Differently, in the absence of rewiring, a minority of the population can easily foster the adoption of selfish AI and gains a benefit at the expense of the remaining majority.
Agent-based Testing of Extended Reality Systems, ICST'2020
“Agent-based Testing of Extended Reality Systems”
Rui Prada, I. S. W. B. Prasetya, Fitsum Kifetew, Frank Dignum, Tanja E. J. Vos, Jason Lander, Jean-yves Donnart, Alexandre Kazmierowski, Joseph Davidson, Pedro M. Fernandes
ICST-2020 – IEEE Conference on Software Testing, Validation and Verification
Testing for quality assurance (QA) is a crucial step in the development of Extended Reality (XR) systems that typically follow iterative design and development cycles. Bringing automation to these testing procedures will increase the productivity of XR developers. However, given the complexity of the XR environments and the User Experience (UX) demands, achieving this is highly challenging. We propose to address this issue through the creation of autonomous cognitive test agents that will have the ability to cope with the complexity of the interaction space by intelligently explore the most prominent interactions given a test goal and support the assessment of affective properties of the UX by playing the role of users.
Navigation and Exploration in 3D-Game Automated Play Testing
“Navigation and Exploration in 3D-Game Automated Play Testing”
Prasetya, Wishnu; Voshol, Maurin; Tanis, Tom; Smits, Adam; Smit, Bram; van Mourik, Jacco; Klunder, Menno; Hoogmoed, Frank; Hinlopen, Stijn; van Casteren, August; van de Berg, Jesse; Prasetya, Naraenda; Shirzadehhajimahmood, Samira; Gholizadeh Ansari, Saba
International Workshop on Automating Test case Design, Selection and Evaluation, co-located with ESEC.FSE (ATEST), 9 November, 2020
To enable automated software testing, the ability to automatically navigate to a state of interest and to explore all, or at least sufficient number of, instances of such a state is fundamental. When test- ing a computer game the problem has an extra dimension, namely the virtual world where the game is played on. This world often plays a dominant role in constraining which logical states are reach- able, and how to reach them. So, any automated testing algorithm for computer games will inevitably need a layer that deals with navigation on a virtual world. Unlike e.g. navigating through the GUI of a typical web-based application, navigating over a virtual world is much more challenging. This paper discusses how concepts from geometry and graph-based path finding can be applied in the context of game testing to solve the problem of automated navigation and exploration. As a proof of concept, the paper also briefly discusses the implementation of the proposed approach.
Aplib: An Agent Programming Library for Testing Games
“Aplib: An Agent Programming Library for Testing Games”
Prasetya, Wishnu; Dastani, Mehdi
International Conference on Autonomous Agents and Multiagent Systems (AAMAS), Auckland, New Zealand, 9-13 May, 2020
Testing modern computer games is notoriously hard. Highly dynamic behavior, inherent non-determinism, and fine grained inter activity blow up their state space; too large for traditional auto- mated testing techniques. An agent-based testing approach offers an alternative as agents’ goal driven planning, adaptivity, and reasoning ability can provide an extra edge. This paper provides a summary of aplib, a Java library for programming intelligent test agents, featuring tactical programming as an abstract way to exert control on agents’ underlying reasoning based behavior. Aplib is implemented in such a way to provide the fluency of a Domain Specific Language (DSL) while still staying in Java, and hence aplib programmers will keep all the advantages that Java programmers get: rich language features and a whole array of development tools.