1. Introduction
The following evaluation aims to serve as the first assessments of the user experience of the prototype of a virtual environment concept developed in the course of 3D Digitalization, Virtual Reality and Animation. The virtual experience assessed consists of a series of tasks to accomplish by the user to finish the game. The evaluation was carried out to a limited size sample. Three different methods were used to later merge them and obtain more holistic findings that help improve the experience of future users.
2. Background
The virtual experience of the evaluation reported was developed under the second module of the course of 3D Digitalization, Virtual Reality and Animation. Due to COVID-19 constraints, the development was made in a desktop-based virtual environment instead of using virtual reality lenses as it is aimed in the course.
The scenery of the game is a medieval church and its surroundings in the village of Lourinhã, to 68.7 km from Lisbon, Portugal. This space was adopted due to the material available from the assignments made throughout the semester, and was decided to keep its use in the last module of the course.
2.1. Technical aspects
The software used for developing the experience was Unity, in its version 2020.1.17f1. The collaborative tool used to simultaneously allow to work the two developers of the game, was the free trial of Unity Teams Advanced, which allowed to work with 3 persons in a team, with a storage capacity of 25 gigabytes. The project was developed on and for Windows 10 operating systems.
The main 3D model used was the previous material developed throughout the course of the Lourinda Church, which was modelled in Sketchup. Other props and assets in the environment were imported from public libraries such as Sketchfab and Sketchup 3D Warehouse. Furthermore, some animations, like the one from the fire in the candles and the living birds was acquired for free from the Unity store.
The animations of the home-built elements (church and context) were made with the scripting tool ErgoUX, developed by the laboratory of ergonomic and user experience of the FAUL. Since the knowledge of the developers of the game about the programming language used in Unity was limited, the animations were conditioned to the possibilities of the ErgoUX tool. The graphic elements of the user interface were edited on Photoshop.
2.2. Storyline and game mechanics
The virtual experience was named ‘The Baptism’ due to the subject of the narrative, which consists of helping a fictitious priest of the Lourinhã church to arrange the interior of the church before the guests of the celebration of a baptism arrives.
Briefly explained on the storyboard in Figure 1, in scene 1, the user receives the help request from the father, and then in scene 2, it finds the first clue of the whereabouts of the key. In frame 3 and 4 receives two more clues until finally finding the key on scene 5. After that, the user enters the church (frame 6) and gets hints for complete the tasks of scene 7, 8 and 9. Finally, the task finishes in scene 9 with the bells ringing, and the game shuts down in scene 11 at the doors of the church.
The chosen method of communication with the user during the experience was a 2D smartphone screen with an SMS dialogue format, which would eventually appear once the user entering certain areas of the model and after activating objects while the tasks were finished.
There were ambience sounds, both the interior and exterior of the church, and particular sounds to objects that were part of the tasks.
There were two fixed components in the interface to help the user navigate the environment and monitor its progress in the game. The box of progress of objects collected was in the upper-left corner of the screen, which was updated while the user collected the objects related to the objective; and the instructions of the game controls, both the keyboard (ASDW or ← ↑ ↓ →) for displacement and mouse movements for the camera rotation and point of view of the user, were in the lower-left corner of the screen.
3. Methodology
There were three different methods of evaluation applied to the experience of a limited number of participants; two of them was applied by the experimenter, and one post-activity was made by the participant at the end of the experience. The sample size of the participants was conditioned to the time limitations to recruit an acceptable amount of people to gather reliable feedback.
3.1. Participants
A total of 6 participants (1 Female and 5 Male) were recruited. The sample of people chosen did not have a defined criterion, that due to time constraints was not prior defined; nevertheless, the age ranged from 28 to 33 years old. One participant had no experience in playing desktop games before.
The executable files were shared with the participants through a Google Drive folder and was clearly explained how to run it properly. In technical aspects, all of the experiments were made under Window 10 operation systems; therefore, recompiling for the Mac version was not required; however, the dialogues within the game were translated both into Spanish and English from Portuguese to avoid language barriers of the targeted participants.
The experiments were conducted in semi-controlled spaces in the houses of the participants, but not exempt from disturbing noises and/or activities in their surroundings.
3.2. User feedback methods
There were three formal methods to evaluate the experience: a verbal method, the interview, and the non-verbal methods: the observation, and a questionnaire. For the verbal method, a semi-structured interview was conducted through the video conferences application ZOOM, which was recorded, under the permission of the participants, during and after the experiences, to collect and later analyse through observation their emotional responses. The timeframe for collecting the data from the participants was of 4 continuous days.
3.3. Interview
The interview questions were mainly centred on the difficulties that the participant faced during the experience, and finally, what should be added, removed, or modified to improve the experience. This user feedback method would allow to let the user express their thoughts consciously, and this information will be later merged with the findings of the observation and the survey.
3.4. Observation
The observation of the participants was oriented to collect emotional and behavioural information during the development of the tasks. During this time, there were no clues given or any help that could advantage one participant to another.
3.5. Questionnaire
Lastly, the participants had to answer a Likert-type seven-point questionnaire, which helped measure the different user experience dimensions. In there, were expressed 14 items with a quantitative value of agreement or disagreement, being 7 the highest level of agreement of the following statements:
| Dimension | Item number | Item |
|---|---|---|
| Feedback | 6 | I received information about my progress. |
| 7 | I was notified of new tasks immediately. | |
| Challenge | 8 | The experience provided image/graphic auxiliaries that helped you overcome the challenges. |
| 9 | The experience provided audio auxiliaries that helped you overcome the challenges. | |
| 10 | There were hints provided in text that help you overcome the challenges. | |
| Autonomy | 11 | I felt a sense of control over interactions between roles or objects. |
| 12 | I was aware of the next step in the game. | |
| Immersion | 13 | I forgot about time passing while playing the game. |
| 14 | I became unaware of my surroundings while playing the game. |
The statements above, chosen from the list of items provided by the course tutors, had been previously tested on other experiences and proved to work properly; therefore, they were adopted and slightly adapted to the circumstances of the virtual experience reported. The questionnaire was hosted in the platform of Google Forms, and was available to fill from the devices of the participants once they finished the experience and ended the game.
4. Results and discussion
4.1. Interview results
During the interview, the main issue pointed out by the participants was the visibility time-lapse of the text messages shown on the screen; 4 of 6 participants reported that the messages disappeared too quick and did not have time to read the messages; hence, they had to restart the game a couple of times, since the messages do not show again once they fade.
Furthermore, 3 of 6 participants reported difficulties with the navigation in the staircase because of the perspective; there was no clear notion from the forward and backward movements.
One participant reported a lack of introduction to the overall challenge of the game. Few of them reported the existence of the cue maps, and others reported that even though he realised the existences of the clue maps in the surfaces of different elements of the environment, the need to add a real-time position map.
Concerning the aesthetics and animations, there were positive reactions from the participants; however, one of them reported that there were some messages which were not coherent in accordance to the graphic that was referencing, such as the illumination in the interior of the church was light enough as to ask to lit a candle, or that the colour of the flowers in task-grave of the cemetery did not match with the colour of the flowers in the woods.
4.2. Observation results
In accordance with the comments given by the participants, it was observed frustration with the disappearing of the messages at the beginning of the game. Since the time frame of the messages is conditioned to the first-person character being within a certain area, and once the participant slightly moved away from it, the messages faded without leaving enough time for the participant to read the text of the following instruction.
Address the speed of the disappearing of the messages could be solved with other ways of displaying the tasks to overcome, such as a fixed canvas to show the text, in an unobtrusive way for the user be able to easily explore the environment, either pausing the game completely and showing the message, which would require of coding not know at the moment of the first stage of design, or with a floating taskbar in one of the corners. Solving this issue is the utmost of the future chores to address; since it is the main source of frustration in the participants, leading to the restart of the game till they read all the messages successfully and finish the tasks or not finishing the game at all.
In addition, it was noticed unease during the spiral stair navigation, since this is a narrow space and the participants only counted with the keyboard and the mouse of their computers, and these ones are not suitable devices for side movements, which is the natural movement when upping up a spiral stair; while in the contrary, the user moves towards the camera points, that said, it is always to the front.
Moreover, the area of the stairs is too narrow and creates troubles in the visualisation. The solution to this problem could be not to strictly attach to the reality of the built element of the church, like to the current size of the tower, and adapt its dimensions to a non-existent virtual environment since it is allowed this kind of distortions to the virtual environments.
It was also observed that the participants spent their first minutes exploring the exterior spaces and the details of the objects of the surroundings, which lead to finding the first message task some minutes after starting the application, but once found the first clue, the accomplishment of the task was solved relatively quickly.
Once asked for water for the interior fountain, the instinctive reaction was to look for water in the exterior fountain, placed to the right of the user when the game begins, but once they realised that all of the doors were closed, they started searching for water in the interior of the church. The latter is not completely a negative issue; however, it could be assessed if involving the exterior areas of the church to solve interior tasks could be part of the experience.
Not all of the participants noticed the omnipresent graphic cues within the game, such as the birds pointing to the correct grave or the maps and plans drawn on surfaces; therefore, it must be discussed if there is a need or not of these elements in the game; these could be removed or test the use another type of global reference guide of navigation within the game; however, the dimensions of the area are not large enough as to require a complex system such as a real-time navigation map.
4.3. Questionnaire results
At first glance, as shown in Figure 3, from the 14 items of the questionnaire, the responses of 10 of them were clustered among the ‘Strongly agree’ and in ‘Agree’, which led to conclude that most of the answers were positive.
The results in the chart below are represented in percentages to ease the reading of the proportion of the answers.
However, read it closely, the results of item 1, related to the statement of ‘There were distractions from the main task’, show that more participants were disagreeing, but still, the answers are spread, since these distractions may be perceived differently according to the personality of the users and their environments at the moment of testing the game. It is hard to identify from this statement what are those distractions or what caused the user to interpret distractions from the main task, so further enquiry may be made with a heuristic evaluation from part of the designers before assessing the experience again.
Moreover, the participants reported in item 2 that the task was indeed related to each other, which demonstrated coherence with the storyline, and the number of activities to undertake was adequate (item 3) in accordance with the playtime available and the narrative.
As shown in item 4, not all of the participants perceived that they received the overall goal initially when the game prompted; but still, in item 5, they reported that the intermediate goals, which were the middle task to achieve the final goal, were broadly well perceived. These results prove a gap of information in the initial phase of the experience that not necessary contributes negatively to the accomplishment of the subsequent tasks, which could be solved by preparing a landing screen showing a short description of the overall game before opening the virtual environment and this might help contextualise the user. Since this was not a recurrent problem, the solution of this issue might be a secondary task to undertake.
From the answers of items 6 and 7, which refers to the feedback received from the game about their progress and the next task to resolve, there were in average positive responses; it is possible to infer from these responses that this was a relatively well-covered feature throughout the whole experience.
In item 8 is observed that there was a certain lack of graphic hints delivered; this might be related to the comments pointed out earlier during the interview, concerning to the incoherence between the visual references such as lights in the interior of the church and the mismatch of the colour of the flowers in the cemetery and the woods. This sort of problems could be easily solved and likely improve the assertiveness of the user while completing the tasks.
However, the audio and text hints expressed on items 9 and 10 were broadly accepted and praised during the interview and also shown in the chart.
In item 11, the sense of control got in overall positive results, yet the responses slightly vary within the acceptance. Adding more animations, object collection, or the number of actions that an object exerts over the others; like the bottle of water empty at the beginning, filling it with water in the exterior fountain and then filling the interior fountain with it, for place an example, could also improve the feeling of control over the elements within the environment. Moreover, this item could gain more user approval if the experience is adapted to a virtual reality environment instead of desktop environments, where the feeling of control is higher.
In item 12, 5:6 of the participants stated that they were aware of the next step, which coincides with the responses received from the quality of text hints provided. The only negative response might come from the irritability felt for not being able to keep track of the next task made or to come due to the problems of the messages disappearing. Nevertheless, even though this item does not demonstrate a big impact on the survey, the solution to this problem should be addressed at first; since this frustration would not be tolerated among users in a real-world market situation.
Finally, even though it was reported that at least a third of the participants were still aware of their surroundings while playing (item 14), in item 13, more than half of the participants reported that they did not feel the time passing by while playing in the environment. As observed, the experience was completed in an average time of 20 minutes, which is a good indicator of immersion, but the uncontrolled environment where the experiments took place, along with the personality of each participant, might have played against not feeling completely immersed in the experience. Solutions to this might be testing the experience in controlled spaces or preparing a more engaging environment with high-quality textures, animations, sounds, and assets to draw more the user attention.
5. Conclusions
Overall, the experience received positive feedback from its first user experience evaluation; nevertheless, this report provides elements of discussion to improve the gaming experience.
The recommendations emerged during the interview, and the problems observed from elements that trigger negative emotions enabled the proposals of solutions to overcome some of the usability limitations.
After making the proposed improvements or the ones to surge during the design revision, it should also be prepared more in-depth measuring tools without time constraints, both by reformulating the questionnaire statements and defining a profile of participants according to an objective criterion that helps to measure other aspects of the experience.
Moreover, even though it is suggested the adaptation of the experience to a Virtual Reality system, once the circumstances concerning the COVID-19 improve, the gameplay and interface might face many changes, and a new first stage of evaluation of the user experience ought to be carried out from scratch since the conditions change drastically from the desktop to the lenses.
6. References
Ng, Y. Y., Khong, C. W., & Nathan, R. J. (2018). Evaluating Affective User-Centered Design of Video Games Using Qualitative Methods. International Journal of Computer Games Technology, 2018, 3757083. https://doi.org/10.1155/2018/3757083
Sullivan, G. M., & Artino, A. R., Jr. (2013). Analysing and interpreting data from likert-type scales. Journal of Graduate Medical Education, 5(4), 541–542. PubMed. https://doi.org/10.4300/JGME-5-4-18
Jamieson, S. (2004). Likert scales: How to (ab)use them. Medical Education, 38(12), 1217–1218. https://doi.org/10.1111/j.1365-2929.2004.02012.x
To download the PDF version click here
