Grandfathers, cogs and goop: learner choices for designs of companion agents

Published in July 2017

Abstract

“The Shift” project aimed to provide a participatory online learning environment, one aspect of which is an online embodied companion agent (or bot) to support the learner, provide advice, and direct the learners to relevant learning. This paper presents the findings of workshops with potential users of the system to identify the preferred appearance and functionality for the bot. The findings revealed a range of attitudes towards both appearance realism and behavioural realism in bots and the functionality that learners desire, and specifically do not want. The students showed high resistance to an anthropomorphic appearance; participants describing the nature of the agent in terms that match the literature’s discussions of “the uncanny”. Realistic behaviour such as personality produced a strongly positive response. The learners therefore preferred realism but not anthropomorphism and these are often blurred together in the literature.

Introduction

One aim of The Shift project (The Shift, 2012) was to develop a user interface for a learning platform. This development took a user needs approach (Kerly, Ellis and Bull, 2008; 100) to find out students’ preferences for the interface; the analysis in this paper focuses on the design of the agent, and uses these results to identify general design principles for agents for learning (ibid; 89), both in terms of its appearance and functionality. The intention for this research is to inform developers and teachers about the considerations when designing bots for education, specifically the degree of realism (both visual and behavioural) and the degree of anthropomorphism of the bot. The degree of anthropomorphism relates to the extent to which the representation relates to a human-like figure. Murray and Sixsmith (1999; 316) note that virtual representations run the spectrum from bodies that “ closely resemble … the body of the user (ibid; 325) to one that is very different, ie completely non-human “the represented body in VR does not have to closely map the person’s body in real life. In effect, it is envisaged that people could experience a radically reconfigured body, say from their usual anthropoid experience, to that of a lobster.” (ibid; 325 – 326). Though Murray and Sixsmith state that users can soon adapt to this representation, the findings of Bayne (2008; 201) are that many learners experience high degrees of unease with non-realistic elements.

“The Shift” project ran from 2012 to 2013 and developed a participatory online learning environment, consisting of two main elements. One of these aspects is a website designed around the concept of a learning journey; learners identify at what point they are now, where they want to be and the system will then map out a learning journey to facilitate their progress. The system also links to social networking platforms and provides links to local businesses. The modules are based around various learning materials for digital media production and are initially aimed at 16 to 19 year olds from the area of Greenwich in London. This study examined the initial interface design at the stage before the platform was populated with these modules.

The second aspect is the creation of an online embodied companion agent or “bot”, embedded in the learning environment, that supports the learner, provides advice, and directs the learners to relevant learning. The bot is the first point of contact for the learner, dealing with routine questions, but passing the learner on to a human adviser when the need arises. The Shift project was led by Ravensbourne College and Elzware Ltd and was funded by the Paul Hamlyn Foundation.

The element of the project outlined here is the development of the bot and the testing of the prototype, since these revealed most about the learners’ expectations of, and experience of, interacting with bots. In order to assess the needs of the learners, and ensure that the design process is as user-centred as possible, the features of the bot were ascertained over two workshops attended by approximately twenty 16 to 19 year olds, the majority of whom were in education or training. At the first of these workshops, the participants were asked to design their ideal companion agent, and suggest the types of functionality they would desire in an agent that was facilitating their education journey. The transcripts of these conversations were analysed to draw out 11 key pieces of functionality, and a range of nine different sorts of bots from the drawings produced at the previous workshop. To these, two additional bot designs were added from previous Elzware projects.

These workshops resulted in the selection of one preferred design for the bot, which then went into development. Four months later a final workshop was held, in which the prototype of the bot was tested with the students, and further feedback obtained. These activities took place during 2013.

Defining agents and avatars

Within the literature there is a very precise and generally accepted distinction between the concepts of agent and avatar. An avatar is any digital representation (either animated or static) which is driven by real-time human actions (Bailenson et al, 2006; 359). An agent is driven by computer algorithms and has

  1. Autonomy – operating “without the direct intervention of humans or others” with “some kind of control over their actions and internal state; (Allbeck and Badler, 2002; 314)
  2. Social ability – able to interact with other agents and humans (Allbeck and Badler, 2002; 314);
  3. Reactivity – able to “perceive their environment, and respond in a timely fashion to changes that occur in it” (Allbeck and Badler, 2002; 314), and
  4. Proactiveness – not only responding to their environment but “capable of goal-directed behavior by taking the initiative” (Allbeck and Badler, 2002; 314).

Agents that have digital models representing them are referred to as embodied agents (Bailenson et al, 2006; 359). Embodied agents interact within virtual worlds or on webpages and have their progenitors in the chatterbots that exist in MUDs, such as Julia in the mid-1990s (Murray, 1997; 215). The word “bot” still continues to be used as an alternative word for agent.

The literature review below examines the relationship between users and (specifically embodied) agents through three steps: the design of the agents, the perceptions of users with regard to these agents as a result of this design, and the range of responses that users have as a result of their perceptions.

Copresence and the perceptions of agents

Defining copresence

Copresence is variously defined in the literature, and is sometimes substituted by the term “social presence” which itself has alternative meaning of the ability to project oneself socially and emotionally in an online community (Arbaugh and Hwang, 2006: 10; Caspi and Blau, 2008; 324). However, most consensus is around the definition given by Bailenson et al (2005; 380):

copresence occurs when a user not only perceives or feels that he or she is “present” within a virtual environment, but that he or she perceives or feels that others are present there as well.

Since an essential part of the definition is about the participants’ perceptions, it is fundamentally a phenomenological phenomenon and research focuses upon the subjective experience of the participants (Bailenson et al, 2005; 380).

Bailenson et al (2005; 385) used four measures of participants’ responses to agents as a measure of the degree of copresence they experienced, these measures were:

  • embarrassment (the assumption being that if the participant was embarrassed to perform a task in front of an agent, this indicates that the agent has social presence)
  • likability (a social response to the agent)
  • memory (participants remember more information from entities displaying what Bailenson et al refer to as “sentience” (2005; 381), but context indicates they mean sapience.)
  • interpersonal distance (again the assumption being that if conventions around maintaining personal distance are maintained this indicates that the agent is being perceived as having a social presence).

Bailenson et al, (2005; 380) acknowledge that some researchers define copresence in terms of “a social, task-related, or physiological response to embodied agents … (which) occurs when people treat embodied agents as if they were other real people”. The difficulty with this as a definition is that it presupposes that participants need to feel an agent is actually an avatar in order to feel copresent with it. They may in fact be fully aware of the artificial nature of the entity with which (or perhaps whom) they are interacting, and still feel copresent, or may not choose to, or be able to, make the distinction (Childs, 2016).

Factors affecting copresence – appearance

The appearance of the agent will affect the perception of copresence, however this relationship is not a straightforward one; for example, a more realistic appearance may not result in a stronger sense of copresence. In fact, increasing realism has been found to make no improvement in copresence – a texture map from an actual photograph has the same effect on copresence as a cartoonlike face (Bailenson, et al, 2005; 381).

Other studies have indicated that the degree of anthropomorphism of the agents has an effect on perceptions of copresence. Nowak and Biocca (2003; 485) observed participants’ engagement with both agents and avatars, which were also both embodied and disembodied. They found that using an embodied agent with low anthropomorphic (i.e. cartoonlike) appearance increased the sense of presence in comparison to the agent not being embodied at all, but when the level of anthropomorphism was increased further (to look like a low-detailed representation of a human), the experience of copresence fell again to the same level as no embodiment at all (Nowak and Biocca, 2003; 490). Nowak and Biocca conclude:

at a certain level, increasing anthropomorphism may be less important to presence than exaggerating certain features of the image to enhance the experience. Also, increasing anthropomorphism may raise expectations and should be done only when the interface and system can meet higher expectations (2003; 492)

Their assumption is that the preference of users for cartoon-like agents rather than human-like is due to the constraints of the system in representing the images, though their findings do not have data to support this.

These are not contradictory findings, in that realism and anthropomorphism are two distinct characteristics, although Bailenson et al point out that the literature has repeatedly conflated the two, adding

We are unaware of studies that have systematically varied both photo-realism and anthropomorphism and, unfortunately, most studies that have been conducted on the effect of embodied-agent appearance have tended to confound photo-realism and anthropomorphism (Bailenson, et al, 2005; 382).

Factors affecting copresence – behaviour

In noting the lack of increase in copresence when the agent has a more realistic appearance, Bailenson et al note that it is the presence of realistic gaze behaviours that generate copresence. A more photorealistic face actually produces a decrease in copresence if the behaviours of the agent are not similarly realistic (2005; 381).

Allbeck and Badler refer to an agent being “believable”, a term which however they fail to define. From the context of their work this may mean that it is realistic, or that it may approximate human characteristics, or that it may be sufficiently complex that a user may be able to invest it with an individuality and/or form an affective relationship with it. Whatever their meaning of the word, they state that for the agent to be “believable” it must display the following in order to have the appearance of mental processes (2002; 319 – 321).

  • Personality: “A pattern of behavioral, temperamental, emotional and mental traits for an individual, and it describes the long-term tendencies of the agent. Personalities can affect the way an agent perceives, acts, and reacts and add to the believability of the character”. (Allbeck and Badler, 2002; 319).
  • Emotion: “Emotions are closely related to personality but have a shorter duration … (and) are generated based on three top-level constructs, namely the consequences of events, the actions of agents, and aspects of objects”. (Allbeck and Badler, 2002; 319).
  • Ethnicity and culture: Although some elements are common to all ethnicities and cultures (such as the basic facial expressions above), and some behaviours, (such as lowered posture for submission or greeting another with an open-handed gesture), culture strongly influences how communication is interpreted and influences social norms, for example distances for interacting, gestures and eye contact. Without cultural cues “communication becomes a trial and error process”. (Allbeck and Badler, 2002; 320). Conveying cultural information informs the interpretation of all of the other aspects of an agent’s actions.
  • Status and role: The relationship between the user and the agent will also influence the appearance and animation of the agent. The role created for the agent, and the status that it has, will need to be consistent with the other attributes in order for it to be “believable”. (Allbeck and Badler, 2002; 320 – 321).
  • Situation awareness: People’s actions are dependent on the environment, objects and other people around them, and similarly for agents to be believable, they must also be able to respond to their surroundings.

Situation awareness is a task-related understanding of the dynamic entities in an environment and their possible impact on agent actions. Situation awareness involves extracting useful information from the world while ignoring the overwhelming amounts of useless information. (Allbeck and Badler, 2002; 321).

The movement of the agent also helps with its believability; this movement is comprised of (Allbeck and Badler, 2002; 314 – 316):

  • Locomotion
  • Body actions
  • Facial expressions, the basic ones given as anger, disgust, fear, happiness, sadness and surprise and neutral (Tinwell et al, 2011; 741; Allbeck and Badler, 2002; 320)

Transitioning naturally from one action to the next enables the users to “better describe the agent’s cognitive state … (,) goals and activities” (Allbeck and Badler, 2002; 316).

Factors affecting copresence – uncanniness

Presence is not the key factor leading to engagement, however, since an agent may have high presence, but generate very low levels of affinity, or even repulsion. The degree of realism preferred by people interacting with any virtual character, whether avatar or agent, is influenced by where the appearance stands in relationship with the Uncanny Valley. The idea of the uncanny is not necessarily something that is frightening, but something that creates unease by mixing the familiar with unfamiliar, death and life, natural and unnatural (Bayne, 2008; 198). The concept of the Uncanny Valley was developed by Masahiro Mori and refers to the valley-shaped dip that appears on a graph of affinity (on the y-axis) against degree of anthropomorphism (on the x).

Mori observed that as a robot’s appearance became more human-like, a robot continued to be perceived as more familiar and likeable to a viewer, until a certain point was reached (between 80% and 85% human-likeness), where the robot was regarded as more strange than familiar. (Tinwell et al, 2011; 741).

The relationship between realism and positioning on the Uncanny Valley is complicated by a range of other factors. The degree of uncanniness varies depending on which emotion is being portrayed by a facial expression; anger and happiness appearing less uncanny than the other basic emotions (Tinwell et al, 2011; 741).

Methodology

The aim of the project was to be as inclusive and participative as possible both in terms of the design of the environment, and also in the way in which the research was conducted. To this end the selection of a bot was the subject of two linked workshops; the first being a collaborative process to identify a number of student-created bot designs and so place the students at the centre of the design process. The second stage was designed to select a bot from these designs through the completion of a survey.

The initial stage does encourage participation, but has a series of weaknesses akin to those of the focus group, i.e. unintended influence by the moderator, the process of engagement may influence the position of the participants, and that the group may act to accentuate some viewpoints more than others (Morgan, 1996; 139-141).

The survey in the second stage required the students to mark the bots in terms of their affective response to it and their requirements for its functionality. This was included because this information was required for the design process. However, the opportunity to conduct more wide-ranging research was seized on by inserting additional bot designs that were not part of the design process. The research questions were therefore opportunistic in that they were based on the requirements of the design process, so as to limit the impact on the students in terms of time and effort. These ancillary research questions were:

  • what factors in bot design support the experience of co-presence? and
  • what factors in bot design do students identify that are most effective in supporting their learning?

This study is therefore an opportunistic hypothesis testing, as part of a consultative design process; the hypotheses being that “affinity to a bot experienced by students will match the relationship with anthropomorphism as defined by Mori’s Uncanny Valley” and “students’ preferences for bot design will be greater the more affinity they experience”.

Using numerical responses as a basis for analysis both enabled a systematic way to rank the bots for the design requirements, and also enabled a numerical analysis to take place to address the research questions. However, these are still limited by being the students’ own perceptions of what would be useful for their learning, and thus are interpretivist data. A longer term study would gather more information about whether the students’ perceptions were accurate; however in the shorter term, meeting student expectations, and engaging them in the design process were the priorities. Similarly, giving students permission to alter the range of feedback ad hoc during the data gathering, limits the objectivity of the data but provides the students with the opportunity to participate in the research design and for the research to capture responses that would otherwise not be possible.

To explore potential explanations for the students’ numerical responses, the numerical data were used as a basis for a focus group discussion, providing qualitative data to expand upon the numerical data. Thus the overall research methodology was a mixed methods approach aiming to gather a wide range of student perceptions, and inclusive, but being limited in terms of objective data about the value of the findings for the learning experience.

Care was taken with the ethical design of the project, in that all students were informed about the nature of the study, the purpose to which the data were to be put, and were monitored throughout to observe whether the process was an uncomfortable one for any of the students. The purpose of the two workshops were to facilitate the students’ learning through the creation of The Shift platform, and so engaging the students and requiring commitment of time and effort was felt to be justifiable. Additional time and effort, to engage the students with a wider-ranging study into bots in learning, was kept to a minimum by including additional bots that were not part of the design process, but in not adding to the minimum required survey questions. Personalised data were not gathered at any time, and students were informed that they may leave the process at any time. All students continued throughout the two workshops, although some did not participate in the survey in the second workshop, which is discussed below.

In the first of the design workshops, approximately 20 participants were placed into groups of five, and were accompanied by a final year student from the animation course at the college. The participants made notes about potential appearance, style and personality of the bot, then discussed this as a group. After this the participants worked with the animators to make sketches of the ideas for comparison and discussion. The nine designs developed in the workshop were used in the follow-up workshop, together with two additional ones from previous Elzware projects (numbered 10 and 11 in table 1).

In the second workshop, participants were presented with all 11 bot designs and asked to respond on an answer sheet, marking their reactions on a Lykert scale from a range of -2 (for extremely negative reaction) to +2 (extremely positive) to the idea of this being used as the design of a companion agent to facilitate their learning. Participants were asked to rank the possibility functionalities of the bot from 1 to 4 representing a list of functionality prioritisation consisting of 4 = Must have, 3 = Should have, 2 = Could have and 1 = Wish list. The participants however insisted that some suggested functionalities were ones that they specifically did not want, and so the response protocol was adapted to include the possibility to respond “No” to any function.

After this, the participants were asked their opinion on the different designs of agents, and what reactions had led to their choices. As this was also an opportunity to uncover generic data on young people’s responses to bot designs, four images taken from pre-existing sources, and representing examples of both low and high anthropomorphism and low and high realism were presented to the participants and they were asked to choose the bot appearance that they preferred.

Results

The agent designs and sample comments are listed in table 1. The agents are ranked in order of the most popular and accompanied with examples of comments made.

Table 1: The ranking of the agents and sample comments made by the participants.

1st place
Bot 11
(20 points)
2nd place
Bot 1
(14 points)
3rd place
Bot 7
(12 points)
  Bot 1 Bot 7
 “It’s a simple likeable character.”
“He looks guiltily happy because he’s biting his lip.”
“The simple design of it.”
“Doesn’t get too much in the way or take too much attention.”
“You’d look at it and you’d be “oh that’s nice” but not too extrovert.”
“Cute.”

“He seems really confident as well.”
“From the picture he seems to have an interesting personality.”
“Morphable in terms of lots of different shapes.”

4th place
Bot 2
(11 points)
5th place
Bot 5
(6 points)
6th place
Bot 8
(5 points)
 Bot 2 Bot 5 Bot 8
 “He knows the answers”
“Fun”
“Mature”
“Skin colour should be non-human skin colour”
 “The egg evolves and then hatches into something else.”
“Lots of ideas going into the concept around interactivity.”
“It’s a good idea but you could end up spending time customising it rather than doing the work.”
7th place
Bot 3
(1 point)
8th place
Bot 9
(0 points)
3rd from bottom
Bot 6
(-2 points)
Bot 3  Bot 9  Bot 6
Too depressing.” “Quite similar to 2, he just looks intelligent.” “Where the cogs were laughing, joking maybe this one could have a wiser-looking face.”
Second lowest
Bot 10
(-8 points)
Lowest ranked
Bots 4
(-11 points)
 Bot 10  Bot 4
“It’s too human. It’s human but it’s not human, so that’s what freaked you out about it. It’s that sort of artificial … It’s really real, but it’s fake.” “I’m sure it would be brilliant with an explanation.”

Participants were also asked to choose their preferred bot from four images representing low detail and high detail and low anthropomorphic and high anthropomorphic bots. 16 participants chose a number; however, three respondents chose to give an answer of “No!”; a response that was not one given to them as a potential answer. The responses are shown in table 2.`

Table 2. Participants’ choices of different agent designs selected to represent high and low anthropomorphism and detail.

Low detail High detail
Low anthropomorphic 1 10
High anthropomorphic 2 3

With the series of questions about functionality, responses were tallied and added together, weighting was 1 for “would be nice”, 2 for “could be able”, etc. 5 marks were subtracted for each “no!”, the heavier weighting indicating the degree to which the participants felt strongly that the option should not be included, a post hoc decision to reflect the unanticipated responses to which the participants agreed. The functionalities that the participants felt the agent should display and the degree to which it should do so are given in table 3.

Table 3. The user preferences for bot functionality

“Must be able to …” Fn 8 Student tracking (the ability to give reminders about assignment deadlines etc)

Fn 6 Personality

56 points
“Must be able to …” Fn 4 The ability to change to reflect topical elements (such as marking specific days or seasonal changes, similar to the Google banner) 50 points.
“should be able to ..” Fn 5 Clickable .. i.e. able to interact with the bot by clicking on it, initiating small animations, dragging it around the screen etc. 41 points
“could be able to …” Fn 1 Move on screen 32 points
“could be able to …” Fn 3 Change shape

Fn 7 Evolve over time

31 points
wishlist Fn 9 Get annoyed 25 points
wishlist Fn 10 Personalise 17 points
wishlist Fn 11 Change size 15 points

The functionalities in the “wishlist” also scored highly in the “No! responses. The most negative reactions were to Fn 2, the ability of the bot to speak. Overall this scored -14, indicating that this was a particularly unwelcome option.

As mentioned above in the discussion of ethics, a priority of the research was that no student would feel uncomfortable at any time, could withdraw their participation at any time and would experience only limited detriment to their time or effort beyond what was required to support their own learning. Four participants, those with a lower academic orientation, felt that the answer sheet activity was too onerous and so submitted their preferences. Although this further limited the objectivity of the exercise, their comments are included within this research in order to represent their viewpoint, in order to broaden the range of input to the discussion and reduce the potential for bias towards those with a higher academic orientation and to value their contribution to the participative design process.

They saw the advantages of learning via a bot as:

  • A system that doesn’t judge.
  • That it gives you the chance to do things a second time.

Their requests were that the bot should:

  • Give advice on how to manage bullies and rude people.
  • Give advice on how to keep calm and express themselves
  • Give advice on finding ways to get your point across.
  • Provide information to get to lessons on time.
  • Re-direct you back to the things you needed to do again.

Analysis

Appearance of agent

The study above corresponds in a variety of different ways with the previous literature on reactions to the appearance of bots. Varying both anthropomorphism and level of detail (we have used the term “level of detail” rather than the “photorealism” used by Bailenson et al. [2005; 382] due to the contradiction in the concept of a non-anthropomorphic image being “photorealistic) indicates that when these two concepts are separated; it is the detailed but non-anthropomorphic images that are preferred, such as bot number 10. The findings tend to indicate that, contrary to the suggestion made by Nowak and Biocca in 2003 (2003; 492) that “increasing anthropomorphism may raise expectations and should be done only when the interface and system can meet higher expectations” even when the system can meet these expectations, anthropomorphic images are resisted by users. In fact, the response given by one participant to the high-detailed, high anthropomorphic image (bot number 11) exactly matches the definition of the “uncanny” given by Bayne (2008; 198), the student stating that “It’s too human. It’s human but it’s not human, so that’s what freaked you out about it. It’s that sort of artificial … It’s really real, but it’s fake.” When questioned about her statement, the participant revealed that she was not aware of this description having applied to photorealistic bots or of the Uncanny Valley (see figure 1), her response was completely independent of any prior knowledge of the literature. The three users who chose to answer “No!” presumably felt that none of the options were acceptable, perhaps preferring a non-embodied agent.

Although the Uncanny Valley accurately describes users’ reactions to agents, Uncanniness is not the only factor influencing the users’ preferences. Their resistance “valley” is wider and begins to dip at an earlier point than one solely due to Uncanniness. Bot 1 and Bot 7 (the cogs and the morphable goop) were preferred over the old man image (bot number 2) because, although the learners responded warmly to the senior grandfather image, the non-anthropomorphic bots had the advantage of being more simplistic, and therefore less attention-grabbing. The statements that the cogs and the goop weren’t too “extrovert” (from the context of the statement the participant is presumed to have meant that it wasn’t too intrusive) felt that in the role of supporting them with their learning, there was also a danger that the grandfather bot could be too distracting. Non-anthropomorphic agents fit the criterion of being present, but not dominating the attention.

This tension in the response to the grandfather bot is reflected in the comments of the group that developed the bot in the first workshop. In this group the negative statements about the bot were that it “could be annoying” and all agreed that they didn’t want it talking all the time and that “if its human it requires attention, like if there’s an old man just sitting there then you can’t ignore him”. On the positive side, a grandfather represented to the participants someone wise, trustworthy and inspirational. Also, because of its potential to be bad-tempered, this meant that the participants would not feel patronised by a bot instructing them. The behavioural realism of the bot possessing some negative behavioural characteristics was seen as promoting a more realistic and hence engaging relationship.

To the participants, one characteristic even more distracting than the idea of having a human in the corner of the screen was the idea of customisation, hence the lower marks for the customisable bot (bot 8). As one participant stated “It’s a good idea but you could end up spending time customising it rather than doing the work” and this concern is also reflected in the fact that in the survey of functionalities, the functionality for the agent to be able to be personalised (function # 10) scored highly in the “No!” responses (see table 3).

The preferred appearances of agents were also those that simply but effectively conveyed some sort of emotional state, echoing the conclusion of Nowak and Biocca (2003; 490) that “increasing anthropomorphism may be less important to presence than exaggerating certain features of the image to enhance the experience.” The top three bots preferred by the participants, although simple in design, and little more than faces with some minor additional design elements, all conveyed some personality through their appearance. Phrases such as “he looks guiltily happy” and “he seems really confident” suggest that the designs conveyed more sophisticated emotional states than the simple six basic ones of disgust, fear, happiness, sadness and surprise and neutral (Tinwell et al, 2011; 741; Allbeck and Badler, 2002; 320). Subsequent research indicates that people can perceive 21 of these compound emotions in facial expressions (Du, Tao and Martinez, 2014) and this suggests that this level of complexity is important for projecting authentic behavioural realism by a bot. The agent that scored lowest was the completely faceless scribble (bot 4), and although bot 6 scored low, the comment that it “could have a wiser-looking face” suggests that it was not the design itself that was not appreciated (since it matched the level of anthropomorphism and detail of the favoured ones) but that it failed to convey this more sophisticated level of personality. Bot 8 was similar to the grandfather image, but scored low because it did not have a well-defined character but “just looks intelligent”. Indeed, the most desired element of functionality of the agent (alongside the practical support of tracking the learner’s progress) was for it to have a personality. Furthermore, it is not only that the agent should have a personality, but that it should have a likeable character. Bot 3, although easily characterised, did not meet with approval because the idea of an emo robot was “too depressing”.

The various responses to the types of bots are summarised in table 4. Category 4 (anthropomorphic cartoon shape) normally corresponds to the peak in a graph of the Uncanny Valley, however here the peak is towards lower anthropomorphism (category 3).

Table 4: Summary of responses to bots

1. Amorphous Shape.
No facial features
No personality
Very low appearance realism.
Low behaviour realism.
Low presence.
Very low rated.
UNENGAGING
2. Non-anthropomorphic shape.
Facial features.
No or simple personality.
Low appearance realism.
Low behaviour realism.
Low presence.
Low rated.
UNENGAGING
3. Non-anthropomorphic shape.
Facial features.
Nuanced personality
Low appearance realism.
High behaviour realism.
Medium presence.
Very highly rated.
ENGAGING (BUT NOT TOO MUCH)
4. Anthropomorphic cartoon shape.
Facial features.
Nuanced personality.
Medium appearance realism.
High behaviour.
High presence.
Medium rated.
DISTRACTION.
5. Anthropomorphic realistic appearance.
Photorealistic facial features.
High appearance realism.
High behaviour realism.
High presence.
Very low rated.
UNCANNY

This difference is represented figuratively in figure 1, where the red line summarises the users’ responses in terms of affinity or otherwise to the bot (which matches a typical uncanny valley curve), peaking at a cartoon-like but quite anthropomorphic character (the cartoon grandfather), before dipping into the uncanny valley (bot 10). The blue line indicates the user declarations of preference, which peaks at the gloop agent.

Figure 1: depiction of the difference between user preferences and their affinity responses for bot representations (no numerical values intended).

Figure 1

Behaviour of agent

Analysing the functionality requested by the participants indicates a similar concern about intrusiveness and distraction. Changing shape and evolving over time were felt to be less important, changing size also received many “No!” votes. However, the ability to interact with the bot by clicking on it, and especially the feature of adapting occasionally to mirror topical or seasonal elements, were more highly rated by the  participants, the common element being that this gives them presence and are fun, but within constraints that means that they are not too distracting. The most negative response was for the functionality of speech. Again this could be because it is distracting, but the strength of the negative response indicates that this is particularly due to it being perceived as uncanny.

The functionalities that were most highly prized by the  participants were those that had a practical use in supporting their learning, so responding to learner metrics (which could be picked up from interactions with the bot, and from usage of the website) scored very highly. Other items of functionality which scored highly were those associated with behavioural realism, such as a personality, and being aware of calendar events, so marking specific days, or acknowledging the student’s birthday. Changing position and size were also useful aspects, in terms of making the interface user-configurable.

Feedback on prototype testing

The bot selected for further development was the highest scoring bot designed by the  participants, i.e. bot number 1. This went through a further design phase (figure 2) but these were still considered “too cartoony” according to  participants and the final designs settled on were in the “high realism/low anthropomorphism” category preferred by participants (figure 3).

Figure 2. The second iteration of the cog bot. Image: Aliyah Coreana

Figure 2

Figure 3. The third iteration of the cog bot. Image: Aliyah Coreana

Fig 3

The third version also meets the participants’ criteria for the bot in that it is less intrusive (the emotions are more understated), and yet has more behavioural realism in that it has a wider range of emotions, comprising neutral, confused, disapproving, excited, happy, optimistic, sad and unamused, and these are animated. It also has a sleep animation, to reduce the distraction it presents when not being used (in that, even with the reduced anthropomorphism there was still the potential for the students to “feel bad” for ignoring it).

By the time of the prototype testing, the bot also had a basic level of AI, enabling it to converse with students about random topics and make statements about thoughts and feeling and also guide them through learning. The learning path had also been narrowed to instruct students in how to create a website using WordPress, an activity that was an effective gateway to further learning, and would be a skill of practical value to all students. This enabled the AI of the bot to be more focused, and therefore possible to program to the required degree of realism.

The prototype testing did not produce any evidence conflicting with earlier parts of the study, but the hands-on experience with the working bot did enable the participants to consider the practical application of the bot to a greater extent. Suggestions by the participants were:

Accessibility: The participants stated the importance of being able to use it on their phones and tablets as well as PCs. The conversation brought up the ability to have the bot as a standalone app that transfers across platforms.

Fun v. Helpful: Although some participants liked the random comments made by the bot, others noted that this would be irritating if they were focused on work and it was drifting off-topic. The consensus was to be able to turn the humour on or off, either for all of the interactions with the bot, or for a single session. Where participants stated that they wanted the bot to be helpful, this was largely in response to the observation that the bot came up with random inappropriate things which could get in the way of learning. They wanted it to focus on providing information when that was important to the student.

Adaptive, plausible: The idea of adaptability refers to the idea of the bot developing a model of the students, enabling it to tailor learning to them. The non-linearity of the programme also appealed to the participants and the ability to control pace and revisit elements.

In essence then, the design of the bot was appropriate, and the balance between its presence and usefulness was approximately correct, but needed to be more adaptive to the needs and mode of the student. Making the bot platform-independent actually pre-empted the development cycle envisaged by the project team.

Conclusions

Bots can play an extremely valuable role in the learning process, particularly as the literature indicates that learners feel more self-confidence performing difficult tasks in the company of an artificial intelligence rather than a biological one. This is reflected in the responses of the more academically challenged students who responded with the comments that a bot has the advantages of being “A system that doesn’t judge” and “That it gives you the chance to do things a second time”. There is, however, potential for the interaction with a bot to feel uncanny, or be distracting, and so the design of the bot can be a deciding factor in its effectiveness.

When looking at the responses of the participants in this study, they correlate in some aspects to the previous literature on people’s perceptions of agents. The response to highly anthropomorphic appearance was one of high resistance, with the most anthropomorphic bot ranked second from bottom and participants describing the uncanny nature of the agent. The responses regarding complexity of design also confirm Nowak and Biocca’s observation (2003) that realism and anthropomorphism should be disentangled when gathering feedback on bot design. For example, the response to anthropomorphic behaviour such as speech was one of resistance, because of the closeness to human-like behaviour, however, realistic behaviour such as personality produced a strongly positive response. Furthermore, more nuanced behaviour, such as confidence, or “guiltily happy”, was particularly highly prized, rather than the six basic facial expressions. Conceptually the nature of realism and anthropomorphism are different; for amorphous goop to be confident is realistic, but not anthropomorphic, whereas for him to speak (and it is interesting too that the bot was described in gendered terms), is both real and anthropomorphic.

Mori’s Uncanny Valley (2012)  is also confirmed at the other end of non-anthropomorphism, in which the other least favoured agent design was the formless squiggle. It appears that to connect to a bot at all it requires a face, if not an identifiable form. More favoured, but still not popular are those bots with a face, but with no nuanced personality. This confirms the first of the hypotheses being tested.

However, the students’ responses did not support the second hypothesis, in that students did not want the bot for which they felt the most affinity, they instead preferred those that have some elements of anthropomorphism but are not too humanlike. When the participants were asked for their ideal tutor guide, the concern for the distracting nature of the agent dominated. The problem with the grandfather design was not that it did not have presence, or was uncanny, but that it had too much presence, and would therefore be distracting. This suggests that the responses of presence and preference also need to be disentangled and that the literature for preferences for designs of bots cannot be simply transferred to the design of bots for learning.

In summary, this research indicates the range of level of anthropomorphism and realism of appearance and behaviour that learners prefer from the design of an agent to guide them through their learning. In this study, the user-centred nature of this design process required that the intended recipients of the learning platform themselves would design the agents’ appearance. However, to disentangle the responses around presence, affinity and preference, and between anthropomorphism and realism, requires work with more nuanced designs for bots representing the entire possible range.

Furthermore, the learners’ position that an agent with too much presence may be counterproductive, in that it leads to distraction, is itself open to question. The literature indicates that greater affinity leads to more effective learning and it could prove that a distracting but highly engaging agent may actually be more effective as a guide despite the learners’ stated preferences. To establish the validity of the students’ assertions requires comparative evaluation of learning with a wide range of bots. Grandfathers may still prove to be better mentors than goop.

 

References

Author profiles

Dr Mark Childs

I’ve been an educator since 1988, when I studied a PGCE at Cardiff University, and taught in FE until 1994. I then got an administration job at the University of Wolverhampton, and then a research job, and then a research post working on an elearning project in 1997. Since then I’ve worked almost entirely in e-learning, doing a few more years at Wolverhampton, then seven at Warwick, then three as a Teaching Development Fellow at Coventry University. Adding up all the projects I worked on over that period gives a total of 38. I did a few years of staff development too, the three last years at Warwick and of course the three at Coventry.

Anna Childs

Anna Childs Consulting

Lizzie Jackson

Ravensbourne College

Phil Hall

Elzware