The Challenges of the Virtual Classroom—The Semiotics of Transmedial Literacy in VR Education

Kyle Davidson

University of Tartu, Estonia

Abstract In 2020, the global COVID-19 pandemic required a significant number of students to move their education online, via digital classrooms. The use of virtual learning environments, combined with virtual reality technology and avatars, has demonstrated the convergence of current technology and society. Identity, embodiment, and social reality construction are increasingly transmediated, with physical and virtual signs being present for young learners (and their teachers) in their classroom space—for example filters or avatars. These may enable a freedom of expression for some students, offering the ability to explore and express their inner selves without physical changes. Alternatively,there could be a bias inherent to such technology which exacerbates offline inequality and prejudice. For example, access to the technology, or the Anglo-centric nature of programming languages which necessitates some level of English language skill. While the ability to swap gender may be positive for transgender teens, the ability to change race or age is problematic. How these distinctions are made and understood requires an updated model of digital literacy, focusing on the sociology, and semiotic interpretation, of the signs within the virtual/physical transmediated space. What is understood as significant in the context of contemporary, online, young learners, is imperative for future identity construction.

Keywords: identity, online learning, gender, transmedia, avatar

1. Introduction

In many parts of the world, 2020/21 saw lockdowns and enforced quarantines as the COVID-19 pandemic spread. In the UK, many offices moved their operations to home based workstations and utilised the online tools available to continue their business.Schools similarly set up online classrooms that allowed students to work through the curriculum and submit work for their teachers to grade via the internet.

This move from the physical classroom or office in significant numbers has offered an opportunity to discuss and reflect on the future of communication online and the transmediality of identity with the increased use of virtual reality (VR) to help find solutions to the restrictions on physical gatherings.

Education and remote work could drive the breakdown of the threshold between modes (the online and the offline, the digital and the analogue) to create increasingly transmedial communication experiences. However, a non-exhaustive search suggests that education is intrinsic to the personal development of the student (Daniels &Brooker, 2014; Idris, Hassan, Ya’acob, Gill & Awal, 2012; Verhoeven, Poorthuis &Volman, 2019; Reed-Danahay, 1996) and the hardware/software of the online space can have an impact on the self-identity narrative the student subsequently forms (Beals,2010; Frederick, Hoppock, Liskey & Brown, 2014; Subrahmanyam & Smahel, 2011).Therefore, an understanding of the signs and spaces that exist to make up the VR narrative—and one’s own identity narrative within such virtual spaces—is imperative.

We will address the impact such “digital literacy” has on meaning in the narrative space of students-as-users. We will investigate the role of the underlying(technological) architectural systems in the creation and potential biasing of said narrative space, which will lead this article to opening the path to future applied research into key areas. Using a semiotic methodology enables a structured overview of the problem, identifying the flow of communication and the points of translation between addresser and addressee. Thus, identity, embodiment, and education become entwined as the three key points to analyse in the comprehension of virtual environments.

2. Previous Researc h

For our research, we are taking the gap between the virtual and the physical self,investigating how embodied representation reduces the threshold, and what an increasingly simulated presence does to education and identity formation. From this, a need for an updated digital literacy will be demonstrated, one that specifically focuses on the transmediality of the user’s present(ed) identity.

The definition of transmediality is discussed by Alison Gibbons as: “on the one hand, the creation and distribution of narrative worlds across media; on the other,the imaginative construction of these worlds by recipients” (Gibbons, 2017, p. 322).The reciprocal, dual, narrative construction and the collaborative nature of digital communication are clearly outlined, leaving us to consider the construction of identity as a synonym for the story telling narrative.

There is much research into transmedial, trans-reality, identity construction in young people (Weber & Mitchell, 2008; Alvermann & Sanders, 2020; Wängqvist &Frisén, 2016; Craig & McInroy, 2014; Junglas, Johnson, Steel, Abraham & Loughlin,2007; Larson, 2002; Meyers, 2009), and specifically within education too (Ragusa &Crampton, 2018; Harrison & Thomas, 2009; Phirangee & Malec, 2017; Cronin, 2014;Moss & Pittaway, 2013). We will review such research within the semiotic paradigm and highlight especially relevant examples below.

While significant research addresses the role of the individual as presenting an identity as an actor presents a role on stage—what Shanyang Zhao terms: “the Goffmanian tradition [of] focusing on the presentation of self to others in the online environment” (Zhao, 2005, p. 388)—we will, as Zhao does, take the role of the external space and influence that the digital signs have upon the formulation of one’s own identity as a reciprocal process.

Semiotics provides a model for meaning-making and interpretation of signs—for what constitutes a “sign” we will use C.S. Peirce’s definition, which in essence states a sign is anything that determines another sign:

A Sign is anything which is related to a Second thing, its Object, in respect to a Quality,in such a way as to bring a Third thing, its Interpretant, into relation to the same Object,and that in such a way as to bring a Fourth into relation to that Object in the same form, ad infinitum. If the series is broken off, the Sign, in so far, falls short of the perfect significant character. It is not necessary that the Interpretant should actually exist. (CP, 2.92)

Thus, it is not merely the signified interpretation that is addressed but additionally the potential Third thing—the Interpretant—that continues the series of signified/signifier relationships. The presented self does not stop at the presentation of the initial signs, but additionally the re-presented data from the signs of self-identity will continually add and form the notion of self. This is made clear by Fanny Georges’calculated identity—an identity formed by numeric data received on a user’s social media profile such as the number of “likes” a photo has or the quantity of “friends” one has in their network (Georges, 2009). The space around the user and the user’s presented identity are not separate elements but interpreted signs within a user’s experience—which is not limited to the online space.

Georges states the online self and offline self each use different signs—visual,oral, and so on—to signify one’s identity with differing immediacy. What Georges terms the “declarative” signs of identity alter when filtered through the interface of the computer screen. Georges highlights that the removal of the body as a primary identifying sign allows for a person to be taken on their intellectual (for example)signs alone:

The representation of identity in amixed realitychanges, at least partially, the problematic of self-representation: in it, reality is interfaced with a layer of digital information in which the bodya prioriresumes its place in the representation of the person, although it can also be augmented by information visible to a third party (for example, by visually associating facial recognition with information on that person’s centres of interest). The consequences are significant: in “on-screen” devices, users remotely dialogue through the intermediary of a graphic, auditory, and textual representation of their identity. (Georges, 2009, p. VI)

Whether this is a good or bad proposition is debatable. The freeing of the mind from the physical body could enable a fluidity of identity, a devaluing of such visual signs in the process of communication—this has been hinted at in at least one study that demonstrated women are considered better coders but only when their gender is disguised (Terrell, Kofink, Middleton, Rainear, Murphy-Hill, Parnin &Stallings, 2017). However, masking the issues endemic to human interaction rather than educating people and enabling a pride in one’s self-identity seems reductive.Alternatively, allowing gender fluidity in avatar construction and enabling people to explore their inner-identity without the restriction of their physical selves could potentially help LBGTQ+ and transgender users set their own signs of identity with a level of control that might not be possible in the physical world (Craig & McInroy, 2014; McInroy &Craig, 2017; Blodgett, Xu & Trauth, 2007) as well as potential, aspirational, identities(Wood & Szymanski, 2020). Of course, VR full-body tracking technology does reposition the physical body as an influential sign within the virtual space.

Clarisse de Souza introduces digital literacy to our article, stating that it will be essential for “citizenship”1. Such statements imply that our ability to coherently interact and manipulate the virtual signs—and our proficiency to do so—directly affect our place with society. Our relationship to the wider cultural narrative is therefore underpinned somewhat by the physical hardware and the underlying architecture of the virtual space that constructs the narrative of the online self-identity,with camera-phones forming an intrinsic part of a user’s declarative identity (Mirsarraf,Shairi & Ahmadpanah, 2017). Put simply, we believe the notion of self within a space of virtual signs has the potential to either bias or enable a freedom of expression, and digital literacy is a tool to help understand this semiosis.

A 2012 study has offered a model for describing the notion of digital literacy within the virtual learning environment: “Identifying different registers of digital literacy in virtual learning environments” (Knutsson, Blåsjö, Hållsten & Karlström,2012). Knutsson et al. record and analyse a series of interactions between teacher and student in order to identify a variety of linguistic nuances, and specifically identifying a discrepancy in register usage between teacher and student, (Knutsson et al., 2012, p.240) and in the failure of the environment designer to recognise the importance of the sign within the space (Knutsson et al., 2012, p. 245). This is a useful introduction to the research issues as Knutsson et al. outline the problem of defining digital literacy thus: “Digital literacy” is one of the problematic literacies because it could include any conceivable computer skill or any activity that takes place with a digital tool”(Knutsson et al., 2012, p. 238). Knutsson et al. outline several theorists’ work, and we included these among our own in the following paragraphs to address the question:“what is digital literacy?”

Lundström and Olin-Scheller write that the traditional notion of literacy as an exercise in encoding and decoding, with singular focus on modes like written media, no longer makes sense due to the rise of different, interactive, modes of communication:[This spread] challenge a narrow definition of literacy (i.e. being able to read and write).Instead, having developed in the last 20 years in particular multimodality and multimedia literacy is an approach which refers to different media that utilise several modes to communicate. (2014, p. 152)

Knutsson et al. concur with the notion of multimodality and non-traditional modes, highlighting that there exists many forms: “such as ‘musical literacy’,‘computer literacy’, ‘mathematical literacy’ and so on” (Knutsson et al., 2012, p. 237).Knutsson et al. take the various interpretations of “literacy” that Gunther Kress lists(2003, p. 23) before ultimately suggesting that such an expansion of the terminology renders the core term meaningless (Knutsson et al., 2012, p. 238). This is perhaps true,with our preference for a holistic, transmedial, multimodal, inclusive definition of“digital literacy” taken to mean any of the modes of communication that can inform a sign within the digital space.

Martin, cited by Knutsson et al., posits a three-stage definition of digital literacy,ranging from the fundamental ability to operate within a digital environment, to a more complex level of participation within an online community. The third level is described by Knutsson et al. as a creative process—it is a literacy capable of “digital transformation” (Martin, 2006, p. 156). The implication for a creative, transformative identity will be addressed shortly.

This perhaps suggests that rather than taking music or computer signs as a separate, individualised grammar, a literacy in isolation of all others, they should rather be addressed as part of a wider, multimodal grammar—a literacy that crosses multiple modes of encoding and decoding:

It is now widely accepted that literacy and literacy pedagogy can no longer be confined to the realm of language alone, and that reconceptualizing literacy and literacy education needs to account for the role of images (as well as other modes of meaning-making) in paper and electronic media texts. (Unsworth, 2006, p. 1165)

Unsworth’s introduction to their paper highlights a change to the current teaching methodologies that posits the need for a multimodal model of literacy. However,the implication that pictures, sounds, interactive media, gestures and so on are not languageper seis perhaps contentious, especially from a semiotic perspective where signs of any intelligible mode are meaningful.

M. A. K. Halliday—as cited by Knutsson et al.—describes language with a wider context, addressing the need for system that recogniseswhatthe communication is about (Field), who is interacting and what their relationship is (Tenor) and the method of communication (Mode). Additionally, Halliday highlights the systemic nature of the text as a reciprocal component of meaning in culture (and thus, by extension,identity):

[S]ystemic theory gives prominence to discourse, or ‘text’; not—or not only—as evidence for the system, but valued, rather, as constitutive of the culture. The mechanism proposed for this constitutive power of discourse has been referred to as the “metafunctional hookup”: the hypothesis that (a) social contexts are organic-dynamic configurations of three components, called “field”, “tenor”, and “mode”: respectively, the nature of the social activity, the relations among the interactants, and the status accorded to the language(what is going on, who are taking part, and what they are doing with their discourse); and (b)there is a relationship between these and the metafunctions such that these components are construed, respectively, as experiential, as interpersonal, and as textual meanings. Register,or functional variation in language, is then interpreted as systemic variation in the relative prominence (the probability of being taken up) of different options within these semantic components. (Halliday, 2006, p. 445)

The concept of multimodal communication transcends the visual, oral, and so on. Knutsson et al. pursue this multimodality in their discussion on register between student and teacher with the use of pictures and emoticons (or emoji).

We will not specifically address emoji in the article as it would detract from the core line of investigation, but we would be remiss to not discuss it as an example of digital signs forming an artificial language offline. We can draw on Kress and van Leeuwen who outline a theory that offers us a model for reading the elements of pictures, pictographic texts such as emoji, computer languages such as programming code and even visual elements of the physical and virtual worlds, as a comprehensible,meaningful whole:

To sum up: the opposition to the emergence of the visual as a full means of representation is not based on an opposition to the visual as such, but on an opposition in situations where it forms an alternative to writing and can therefore be seen as a potential threat to the present dominance of verbal literacy among elite groups. (Kress & van Leeuwan,2006, p. 17)

Scolari, Masanet, Guerrero-Pico and Estables (2018) extend the notion of literacy further in “Transmedia Literacy in the New Media Ecology: Teens’ Transmedia Skills and Informal Learning Strategies”. Scolari et al. posit a more collaborative learning experience across multiple media, including the term “prosumer”—a portmanteau of producer and consumer in this instance (Scolari et al., 2018, p. 810). As such, literacy in the digital environment will encompass different media and different modes of presentation via gesture, sound, images, and the avatar design. A user operates within a transmedial space, forming a narrative collaboratively: “transmedia literacy places digital networks and interactive media experiences at the centre of its analytical and practical experience” (Scolari et al., 2018, p. 810).

Knutsson et al. refer to social media—and we can support the discussion with research by Fanny Georges (2009, pp. 165-193)—to build a fuller picture of the online eco-system within which both teacher and student exist. As Knutsson et al.suggest in their introduction, social media is pervasive throughout a user’s experience of the online world—not just work or home but work and home (Knutsson et al.,2012, p. 237). This is the collaborative, transmedial eco-system that the digital learner moves within. Each contextual element is important—research by Knutsson et al. demonstrates the convergence of different registers within an online dialogue(Knutsson et al., 2012, p. 242). A digital literacy must include each register, each mode, of communication.

Knutsson et al. introduce Mondo as a virtual learning environment (VLE) for their study. Additionally, we want to bring in virtual reality (VR), augmented reality (AR),and mixed reality (MR). We will use VR+ for ease to mean virtual reality including augmented and/or mixed variants too. Such technology has become cheaper in recent months (the Oculus Quest 2 mobile VR headset was released within 2020 at $299) and the software has become significantly more mainstream for workplace scenarios—for example, for networking, conferences, or for education.

How we define reality is thus a fundamental question for this paper. The topic is extremely broad but we can opine a general definition for what makes a physical,virtual, augmented, or mixed reality scenario. We will use the Paul Milgram and Fumio Kishino virtuality continuum as the basis for defining our key terms—VR trends towards fully virtual environments, while AR augments the “real environment”(Milgram & Kishino, 1994, p. 3). Mixed reality is a combination of the two: “the most straightforward way to view a Mixed Reality environment, therefore, is one in which real world and virtual world objects are presented together within a single display, that is, anywhere between the extrema of the virtuality continuum” (Milgram& Kishino, 1994, p. 3). Viewing VR+ as a continuum suggests a transmediality to the environments at conceptions due to the lack of segmentation between the virtual and physical2as to what can be displayed within the single display.

VR+ adds an extra dimension—literally—to the VLE and subsequently creates a need to study avatar design and gesture tracking within the digital literacy/education model. While avatars could be a feature of the 2D windowed classroom of Zoom or Mondo, its use will be restricted and moderated by the teacher. We will study filter effects bleeding in from the social media and discuss whether such features create a new area of register divergence. However, the VR environment necessitates an avatar. The semiotics of learning via a communication model that is mediated by such dynamic avatars will be explored - for example, the efficacy of learning from a teacher presenting themselves as a virtual YouTuber (VTuber), in a similar style to the online entertainer Kizuna AI.

There is perhaps no difference to learning from a Sesame Street-style puppet character and a virtual avatar up to a certain age, but once the student is old enough they begin the process of understanding the sign for what it is. As a result, should the avatar be understood as a separate character, a mask, or an extension (or even online replacement) to the user’s physical/offline identities? Research is sparse but there are suggestions that unbiased avatars can be beneficial to young learners (Theng &Aung, 2012) although there is a need for digital literacy aimed at younger students(Wohlwend, 2010). Of course, the key term here is “unbiased” which, as we will address, is hard to achieve with current technology.

The VTuber sign augments reality rather than operates solely within VR, and as such it is relevant to study avatar use across the spectrum of VR+, rather than compartmentalise AR into the same research area as camera filters. Users can overlay an avatar while presenting a video within the physical space—see, for example,Kizuna AI interacting live at the release of the iPhone XR/XS at Softbank3. This offers an opportunity to discuss the term “Virtual YouTuber” as some examples—such as the online performer KimplE—have minimal presence on YouTube, preferring instead to invite audiences to watch live via the Twitch streaming platform. As we are not examining the conceptper sewe will continue to use the term “VTuber” as a placeholder term to denote not just the dynamic avatar, but rather the complete character that is constructed by a performer. While there are differences and nuances of the characterisation, the key point for us is that the physical and social identity of the user can be (potentially) hidden by the avatar, to varying degrees.

Such personas are an extension to the so-called ACG (Anime, Comic, Game)—or AMG (Anime, Manga, Game)—culture of Asia. This term is primarily used in China,while Japanese speakers use Nijigen, which literally translates to “two-dimensional space”. The recognition of the 2D as a multimodal, transmedial aspect to mainstream entertainment has been driven by the 100+ million potential audience members in China alone. Thus, in September 2020, the Chinese online entertainment company iQIYI released Dimension Nova4. This show combines the virtual Youtuber—or more specifically the VOCALOID style virtual idol (a virtual character that sings,dances, and is marketed as abona fidepopstar)—with the traditional TV show where people compete for a prize by demonstrating their music talent. Dimension Nova offers a glimpse at the cutting edge of the transmediality of identity offered by virtual personas, with characters interacting within a virtual space and—through real-time motion capture and rendering—with the audience and physical judges. The 30 or so virtual idols perform dances and sing, with their identities maintained as complete narratives—the contestants are the virtual characters, and between routines there are skits and scenes where the contestants interact casually. The role play is maintained,as with KimplE or Kizuna AI, and as such the notion of interacting with/as a virtual person is increasingly normalised. While the show is new, it does offer a hint at the potential of the technology (which could encroach in the educational space).

Games offer an opportunity to study the creation of an alternative identity within a virtual world that perhaps remains separate from the physical identity of the player who continues as an actor behind the avatar (Vella, 2016, pp. 79-121). While games have a ludic nature that perhaps separates them from the classroom experience,there is a crossover in how a user collaborates, role-plays, interacts, or otherwise demonstrates a presence within the game towards other users whose signs of identity are decentralised (to use Annabelle Klein’s vernacular—as referred to by Georges)from the physical self (Georges, 2009, p. IX). The agency of the signs of identity is within the user’s own control, rather than restricted via physical signs—the avatar in-game becomes the declarative identity, as well as the calculated identity—what Georges terms as identity formed from the quantifiable data of social media and other key indicators (likes high scores).

While the quantifiable information of how people react to the constructed identity should appear separate to the user’s offline identity, evidence suggests that the artificial identity can be as impactful and real to the user as one’s physical identity.The Proteus Effect is an example of this, coined by Yee and Bailenson to describe the actions of the offline users changing in response to the actions of their online persona(2007, pp. 271-290). The transmedial identity, as a collaborative concept rather than two distinct concepts, can be seen within fandom (Jenkins, 2010; Gibbons, 2017),as well as the critically under-researched impact of existing online behind the virtual avatar of a fully realised character such as Kizuna AI.

The semiotics of the interaction space is relevant to the process of education since it contextualises the avatar—and other signs—within an interaction process itself,one which starts with the designer as de Souzaet al. outline with an introduction to semiotic engineering in Human-Computer Interaction:

The role of a semiotic theory in this perspective is […] to tell designers how they should (a)make the communicative choices, so that the interactive system is good, and (b) convey their choice to users so that this communication does not get in the way of interaction, but serves as a secure scaffold for productive user activity. (de Souza, Barbosa & Prates, 2001, p. 463)

Key to our theory is that the architecture (that which translates the space from code and data and provides visual and aural feedback) has the potential to bias a student’s identity narrative during a formative educational stage. As de Souza et al.posit, the designer is always present in the interface (de Souza et al., 2001, p. 463).The work Georges has done using Facebook, as a spatial extension to the user’s selfrepresentation (Georges, 2009, p. VIII) is valuable, and the research by Dasgupta and Hill (2017) into efficacy of programming in English or localised to one’s native language will be very influential to our argument. Thus, we intend to update the research with specific focus on the interface within VR, AR, and MR.

The architecture of the virtual classroom can be uniform in aesthetics, but perhaps volumetrically scaled to the students’ individual room. Although this falls short of the physical classroom then, the overlay of the virtual aesthetic does create a sense of otherness, of distinction, between the home and the class.

Such separation alters the relationship to power (with teachers being an authority figure in the class) and hierarchy of language (with some language acceptable or taboo within the class space—as suggested by Knutsson et al.’s study on register).The power bias within the technology and its impact of the student/teacher dynamic has been studied by Laurel Jeris (2001) who looked at power relations in online, adult,learning environment. We will append the research with consideration to semiotics to power, gesture, and culture in VR.

Augmented reality or teaching within a computer screen does not always allow for the separation of space, instead relying on the inclusion and integration of signs within a room that already has signification and connotation to the student. Hence, a digital literacy model is not just an understanding of digital signs, but a comprehension of the semiotics of the sign within its wider context.

The contrast between the physical geographical spatiality of the user and the virtual space that exists seemingly beyond the geographic space enables the mixing of users from all over the world to join and engage within the community. The use of online classrooms actively makes communication between people of different cultures significantly easier than any previous technology, and more immersive. For example,the online communication tool, Skype, now has real-time translation5.

Avatar design, as with any artistic creation, can suggest a tribalism based on the cultural signification of the sign—fandom for Japanese anime for example. How this cultural tribalism translates into the virtual sphere, and whether it is a negative (like appropriation) or a positive (an exchange of signs) was researched extensively, preinternet, by John Fiske (1992, pp. 30-49) and updated with reference to contemporary“textual productivity” in internet fanfiction by Matt Hills (2013, pp. 130-153).

Taking these previous considerations on identity, literacy, and digital architecture,we will explore them within the context of the classroom. Education has an important formative impact on the young adult (as mentioned above) and Georges considers the young person’s development within a digital environment by expanding Annabelle Klein’s decentration process to include signs from the digital space and virtual media concepts such as “friend requests”, stating:

[T]he presence of quantified data (number of page visits, number of friends) constantly places users in a position of facing their reflection in the mirror of local cultural values,leading them to question, for example, the reasons for the growth or decrease in “friend requests”. These quantified evaluations […] can be destructive among adolescents, during an intense period of identity construction. (Georges, 2009, p. IX)

As such, the accumulation of virtual and physical signs form a transmedial identity that impacts on the ability of the user to be a citizen6, to formulate their adolescent identity (Georges, 2009), to understand their gender or sexuality—positively or negatively (Craig & McInroy, 2014), to collaboratively form meaning from the signs (Scolari et al., 2018), while such literacy of the digital signage has pedagogical repercussions on how educational resources are presented and responses interpreted(Chae, Lee & Seo, 2016, pp. 273-393). Improved modelling of the meaning of the digital sign will also enable understanding of the potential, transmedial, bias from the underlying architecture—such as via Anglo-centric programming languages (Idris &Ammar, 2018).

We will use this previous research to help analyse and understand the current situation within remote education and work, updating the conclusions with reference to the latest technology, and offer our own roadmap for future studies, using semiotics,to improve the efficacy and relevance of digital literacy as both a pedagogical concept,and a tool for pedagogical research.

3. Technology

3.1 Programming languages

The semiotician Juri Lotman stated that the systems of signs are hierarchical, with natural language as the primary modelling system from which the secondary systems of art and culture are made comprehensible (Lotman, 2011, pp. 249-270). Similarly,we can say the underlying syntax of the programming code is a foundational modelling system of the virtual space that directly translates to the topography of the environment, avatar, sound production and so on. It is the coding that translates the gestures of the user and, conversely, directs the user’s interactions with the interface.The argument we put forward is that computer code within the semiotic model is therefore comparable to the natural languages. Indeed, to cite Kumiko Tanaka-Ishii, the natural language layer of the programming codes is designed to be read by programmers to aid the development of the programme: “programmers are trained to choose and design meaningful identifiers from a natural language viewpoint”(Tanaka-Ishii, 2010, p. 20).

As an artificial language, computer syntax has several hierarchical libraries within each programme with machine-readable binary translated into human-readable code that attempts to simulate the natural languages7, with some libraries concerned with function and some concerned with reproducing content. This is where we encounter one of the key issues this author sees with the current VR education/identity architecture—the underlying potential for bias begins at the most fundamental level—the code.

There is a belief (which this author shares) that VR+ holds the potential to allow users to fully inhabit a variety of identities, using the virtual world as an escapism where one can fully express what they feel or dream:

Given its capacity to generate almost any conceivable experience, it might provide us with an escape not only from physical, biological, and social constraints, but also from a bleak or oppressive cultural landscape. Those who feel confined by their community’s social norms and expectations of conformity, for example, might replace them (for a brief time) with an alternative virtual world where they are free of such norms and pressures. (Blitz, 2009, p. 1145)

However, to fully create your own identity requires—at the most basic level—access. This includes access to the software and a stable internet infrastructure, but also to the education that can provide a student with the skill to read and write in code.It is assumed that 1.5 billion people globally speak English—with 527 million native speakers8—yet most programming languages use English syntax. Indeed, the cases of non-English programming languages are so unique that they can be listed individually and identified specifically by this notable fact9. Even assuming a disproportionate concentration of working age people speak English globally, the lack of nativity in the language presumably adds delay to when programming can be taught, delaying digital literacy education of this fundamental sign system (Raj, Ketsuriyonk, Patel &Halverson, 2018).

Digital literacy begins with the programming languages and with the increasingly transmedial nature of identity construction then. As such, the majority of the global population is going to learn how to manipulate the syntax required to represent themselves at a potentially slower rate than those who start with English as a first language. Research by Dasgupta and Hill using Scratch, a key educational environment for teaching programming, posited: “learners who code with localized programming language keywords and environments demonstrate new programming concepts at a faster rate relative to users from the same countries who use otherwise identical interfaces in English” (Dasgupta & Hill, 2017, p. 34).

It is not unrealistic to suggest a situation where users who are able to code betterper secan create an online persona of depth and complexity. Additionally, having to express selfhood in English can alter the identity narrative of the non-native speakers since the words available are translated via a degree of separation to the language.This might be a narrow view as the code terminology is a sign that is interpreted as meaningful by the user without the external object of the programme itself, as an example of Peirce’spansemiotictheory (which views the person and their thoughts as signs within the semiotics process too, not interpreters or overlords of the process):

[T]he pansemiotic viewpoint suggests that [external entities like physical objects] can be grasped in the mind only through representation by signs. […] Putting aside whether it applies to human thought, the pansemiotic view is taken […] because it allows comparison of computers with humans at the same level of the sign system. […] The computer world is a rare case in which the basic premise of pansemiotic philosophy holds. (Tanaka-Ishii,2010, pp. 20-21)

If the individual is a sign and signs are that which anything (organic) processes as information, then as long as it is intelligible the syntax constitutes a semiotic system.Ergo, the English syntax of the computer code could be understood as a sign within the computing paradigm without an additional understanding of its meaning within English itself—to construct an online identity requires an understanding of “integer” or “function”within the grammar of the code, not an understanding of their definitions within English as a natural language then. However, differences may arise with the familiarity users have between using pictographic script compared to phonetic scripts like English, as many of the cited studies suggest some benefit to native language localisation.

There is evidence of positives to bilingualism at a biological and cultural level with lower dropout rates and high attendance in school (Dasgupta & Hill, 2017, p. 33).Concerning digital literacy and identity in the classroom, any hurdle to understanding and comprehension of the signs will necessarily create a hierarchy of users that, given the ubiquity and importance of the architecture, we do not believe should be imposed.As Philip Guo identified in his introduction:

[Non-native learners] wanted instructional materials to use simplified English without culturally-specific slang, to use more visuals and multimedia, to use more culturallyagnostic code examples, and to embed inline dictionaries. […] Based on these findings, we recommend learner-centered design improvements to programming-related instructional resources and tools to make them more accessible to people around the world. (Guo, 2018, p. 1)

Dolittle is an object-oriented programming language designed to allow the use of Japanese—among others—that somewhat circumvents the bilingual issues mentioned above with the addition of a Graphical User Interface (GUI). Multimedia elements can be manipulated with mouse operations as well as by coded commands (Kanemune& Kuno, 2005, pp. 144-145). It was developed for schools to offer a head start in computer programming education, and although Dolittle is not a common language of avatar design, it does introduce the concepts of the programming “grammar”—objectorientation and the flexibility of a system that is potentially the basic building tool of a user’s online identity. The easy multilingualism of Dolittle demonstrates too that the Anglo-centric nature of programming languages is unnecessary at a technical level due to the required translation from human-legible to machine-legible code. Digital literacy does not need to include the machine-readable syntax, only our translation of it, and the subsequent hardware re-translation of the data into virtual artifacts.

Currently, the situation is to use tools provided by large corporations (such as Facebook) to create an online extension to their offline identity. Perhaps then,the bigger advantage of code literacy is to create a narrative free from potential censorship by such corporations. Of course, within the context of education and the online classroom, a more uniform level of coding ability could be expected, as well as more conservative expressions of identity within the virtual classroom (as is currently the case in bricks-and-mortar classes). Our point would be that the time spent teaching digital literacy and guiding the construction of a young learner’s identity online could perhaps be longer in non-native English speaking countries that have to teach English literacy first, and such a delay may be open to exploitation.

3.2 Virtual reality and presence

Virtual reality maps out the user’s physical presence, creating a representation—or simulacra perhaps (Ryan, 2001, p. 13, p. 29, p. 34)—of the user within a mapped representation of a virtual space. This space simulates distance without having physical properties. Thus, the programming syntax is a direct translation of the movement, the space, and the user’s self—their presence.

There is a warning to be made, though. The capacity to express oneself within the limits of the virtual plane, without a comprehensive social support structure—including digital literacy within the context of the wider sociological, ethical, moral,and legal paradigm of civilisation—could corrupt the positive potential of VR+.As Blitz writes, using a warning from Stanislaw Lem, there could be a potential escalation of deviant behaviours given the opportunity to enact all of humanities desires. Blitz proposes that the solo actions within VR should be protected under the First Amendment (of the American Constitution), giving the virtual space the same status as one’s own living room. Blitz continues:

Indeed, one of the greatest potential benefits of VR is that, like a number [of] other emerging technologies, it might allow us to obtain—in our own homes, where we are freest to shape our external environment—experiences that were once available only in the much more regulated world outside. […] a person may one day find that even in a small comer of a 400 square foot studio apartment, she has all the space she needs to visit and tour the city of her choice. (Blitz, 2009, p. 1151)

By making the virtual dream space protected by the same legal apparatus as art or speech, the suggestion is that similar rules and social norms will be applied to control the expression of such speech within certain institutions. For example, few students have the freedom to say, dress, draw, or write whatever they wish within the classroom, regardless of the First Amendment due to the age of the participants of the education system.

The transmediality of the underlying language is foundational to the architecture of the virtual space. Space itself crosses the modes of virtual reality and physical reality with the real-world spatial mapping and artifacts overlapping and interacting with the physical (Orel, 2017, pp. 123-125). The virtual avatar user may exist within live action windows, pre-recorded videos, or potentially another environment such as a live action concert, thus confirming the importance of context as part of the multimodality of identity in VR.

Volumetric capture creates a visual representation of the physical space-asdata, so there is a literacy of the physical area that can be applied to the linguistics of virtual identity. Take for example the physical sign of the school uniform. The goal of a uniform is to offer a student a degree of anonymity through conformity, a levelling of their background or socio-economic status with regards to their peers among the school, while marketing the school externally (Meadmore & Symes, 1996;Hertz, 2007). Additionally, it creates a threshold that signifies the school environment and its associated register of obedience. However, by bringing teaching online, the student is inviting their classmates and teacher into their very homes, offering the viewer a chance to incorporate a visual space into the identity they perceive with the student. In the case of VR, should space be required for physical movement within the classroom experience, then the uniformity of the school classroom becomes restricted by the student’s individual situation. The impact of the external space on not just VR education but all online education is somewhat under-studied, with only the merest hint at this problem, such as this single line which suggests a much larger issue: “The main negative aspects of the online course were ‘feeling lost in cyberspace’” (El Mansour & Mupinga, 2007, p. 246).

The core result of this is the uniform anonymity is removed as even a small encroachment of homelife into the classroom (a neighbour being loud, a parent walking past, poorly chosen camera placement, or a cat) can immediately append the student’s sign with some level of external detail that wouldn’t normally exist from the standardised classroom locale. The online classroom becomes reconstructed then where the social norms associated with online conversation crossover with the structured rigidity of the classroom:

Because social space and relations are radically disembedded within the virtual classroom,teachers and students can both import preestablished social conventions and rearticulate these conventions in ways that redistribute classroom power. Further, this rearticulation is engendered and constrained by the separation of time and space and the possibilities for recombining them afforded by the different forums within a virtual classroom.(Anagnostopoulos, Basmadjian & McCrory, 2005, p. 1726)

The multimodal classroom impacts the self-identity of the teacher as well as the student. The classroom creates space where identity narratives are collaboratively discussed, decentred from the normal power bias, and structured by the norms all users agree to establish. Thus, digital literacy is vital for both the student developing and expressing oneself within this space, and for the teacher to help understand and guide such rearticulation. This includes avatars and filters, with fashion as another sign that, adding to the spatial narrative within VR and online teaching, seemingly offers a different modality for the notion of identity (Bardzell, Pace & Terrell, 2010).

Understanding the framing of oneself within a chat window, or preparing an appropriate volumetrically mapped space for VR, requires an understanding of the signs around oneself when using the technology. Understanding the movement and focus of the camera, the limits and boundaries of the limb trackers, all affect the movement and presence of the student within the virtual teaching space. Literacy requires comprehension of the signs that we, as users, generate, as well as the signs established—or rearticulated—within the interface and avatar designs.

Gesture and presence within communication is important for conveying information, with presence in VR (and specifically immersive VR) being a focus of studies for decades:“desktop VR” (Lavroff, 1992) fails to provide first-person (Clancey, 1993) learning experiences and certainly does not engender presence. While non-immersive [virtual environments] are certainly effective for some kinds of learning (see Kozma et al., 1993)and are certainly more affordable than immersive systems, our work does not yet include them. (Winn, Hoffman & Osberg, 1995, p. 3)

Being literate of cross-cultural gestures is important in face-to-face communication (Kita, 2009)—and within the virtually bounded space which includes avatar expressions, it is equally vital (Koda, Ishida Rehm & André, 2009). It might even be more important considering the greater potential for mixing students from a variety of cultural backgrounds—it is, after all, easier to choose a university when geography is not a concern. This cross-cultural mix doesn’t just impact the student’s individual actions, but also what the digital space affords the student in terms of preformatted gestural proxies or signs of emotions—the most obvious simulacra being emoji and short, animated, gifs currently (Gawne & McCulloch, 2019).

The virtual classroom, both in VR and online, is a space of multimodal messages and transmedial identity narratives. The signs that construct, guide, or educate the register and codes within the space operate within multiple layers of programming and linguistics. Bias or other incomprehensibility will create hierarchies of ability within students, possibly leaving their identity open to manipulation at a formative age.Literacy as an understanding of the semiosis of these signs must be taught. We will now consider identity as perceived by others online, with regards to digital literacy.

4. Identity

4.1 Representations of self in social media

The representation of self, and the accompanying notion of presence, within the VR world begins at the fundamental level of the physical, architectural, and coded space,while the interpretation of the user-to-user message occurs within the virtual space,via the avatar or virtual room that is created from the physical boundary. Although younger students will, in all likelihood, be constrained by the technology to only present themselves using the live camera feed without filters or avatars,this doesn’t necessarily apply to the entities they will be learning from, or for older students. Additionally, digital literacy should be taught to prepare a student for life,not just within the classroom.

Within any of the main software applications someone may use to take and share photos or video of themselves—Instagram, TikTok, Snapchat, or even the native camera application available on the phone or camera—there are a myriad of filters and so-called “beautifying” effects that edit the photos automatically. These can range from subtle edits such as smoothing skin texture or adjusting the light levels, to actively manipulating the image to present the user as an animal (such as a racoon10),vegetable (such as a potato11) or even a different gender (using FaceApp12). These filters are so ubiquitous that users have started using them on their pets.

The culture of film editing and adding filter layers to one’s photographs has been the subject of many studies into the effects social media is having on self-image as a transmedial narrative construction (Mirsarraf et al., 2017). However, while it may seem that some have taken the editing too far, it is perhaps comparable to makeup. Social media without constraint seems to be detrimental to the development of a young user’s identity (Georges, 2009) and as such, digital literacy must include aspects of the virtual eco-system that exist externally to the classroom environment.

A student that has grown dependent on the filters to append and augment their appearance and identity is not a wholly fictional situation to encounter in the classroom at any age, this author would imagine. There are hints that the generation of users who are growing up fully ensconced with such tools are understanding the power and impact of them with better strategies than perhaps many realise from looking at the reality TV stars.The trend for real-life photos presented alongside the staged and edited Instagram shots highlights the illusion and falsity of the edit, leading to less body image dissatisfaction(Tiggemann & Anderberg, 2020). It seems that presentation of reality also detracts from the edit, presenting it as what it is: a staged shot and not real life. The conflict between the photograph and reality is not new, and the new technology is not particularly different to that which Roland Barthes theorised about decades before Instagram. A.S.P. Caldeira eloquently analyses the self-representation of Instagram images with a review of 12000 photos, ultimately concluding:

If digital technologies and in particular social networks such as Instagram brought with them a series of transformations to the photographic practices of the by-gone analogue era,they were nonetheless rooted on a quite solid foundation set by over a century and a half of photographic visual culture. (Caldeira, 2016, p. 154)

How the virtual space represents or translates reality has been the focus of study on multimodal, cross-channel, communication (Barricelli, Gadia, Rizzi & Marini,2016). As such, the physical and the virtual, the real and the unreal, are entwined in the formation of a narrative that relies on interpretation. The digital literacy of the teacher is as important as that of the student in the education setting to create appropriate strategies for the use of filters in the classroom. While the use of a filter may potentially detract from the student being fully present in the classroom, there is an alternative perspective that is less negative in its view.

The ability to apply a more masculine or feminine mask to the presentation of oneself can offer opportunity for users who are struggling with their gender identity.Such tools could offer the user the opportunity to express themselves outwardly with an identity that matches their own internal narrative. Research into the ethics and morality of this topic is woefully outdated when compared to the technology that is available. The issue is also fraught with misunderstanding about the nature of gender identity and sexuality that desperately requires researchers like Mary L. Gray13.

Such research is vital to create strategies for interpreting and reading identity signs from an education and digital literacy point of view. The interpreted gender of the avatar sign reflects a duality of identity—the transmediality of the virtual and physical. The anonymity afforded by the avatar or filter can impact greatly on the formation of an identity. While it can offer the potential for users to express an inner identity, the risk of users switching gender or race to “fit in” becomes an issue for digital literacy in the education context. Lisa Nakamura has extensively outlined the issues: “The illusion of diversity through digitally enabled racial passing and recombination produces a false feeling of diversity and tolerance born of entitlement”(Nakamura, 2008, p. 1674). She continues with reference to identity tourism and the notion of users wearing a skin that they justified as not being racist due to the ludic,fleeting, nature of identity within cyberspace. However, some studies also show that different race avatars reduce bias (Peck, Seinfeld, Aglioti & Slater, 2013).

Digital literacy and education must include discussion on the semiotics of signs that are offline as well. Additionally, semiotics, with its weighting on interpretation over intention behind the meaning enables a method for discussing how digital signs can alienate and harm regardless of the users’ original purpose.

4.2 Avatars and identity

A different extreme of the avatar presented identity spectrum is the use of non-human avatars. While there is evidence for the positive use of avatars on a gender spectrum for transgender or questioning users, some have expanded the discussion to include transspecies identities. This can range from those who believe themselves to be reincarnations of animals or otherwise embody a physical connection to an animals,to those who—similar to the VTuber models—believe that the aesthetic is cute or kawaii (a term meaning a specific aesthetic of cuteness in Japanese popular culture).Research into otherkin cultures online, (those who fall in the former category) has studied the relationship between species and identity, with furry culture being well established as a somewhat thriving online subculture. The identity of the human participant in the culture is often overridden with their curated furry persona when wearing the costume, avatar, or discussing topics within the fandom (Johnston, 2013,pp. 293-306)—again, somewhat similarly to the VTuber who embodies their character full time online.

With the move to online classrooms, what happens when a student wants to portray themselves as a non-human character? Computers-as-social-actors (CASA)is a theory that posits humans will treat non-human entities as human due to presence and functionality within social situations (Nowak, Fox & Ranjit, 2015, pp.554-569). It has been suggested that the human brain has not evolved to differentiate between organic and inorganic stimuli, so anthropomorphism seemingly leads to trust responses (Nowak et al., 2015, p. 555). As these responses are interpretations of signs, semiotics would allow a distinction to be made via context while enabling a meaningful reaction to the sign regardless of its overt humanity. Digital literacy is imperative as such responses can conversely dehumanise the user, with females being disproportionately dehumanised (Nowak et al., 2015, p. 556). While the transspecies avatar may be a long way from social acceptance in the classroom, it is already present in social games and forums—a literacy of these signs is already needed. The connection one has to their avatar can go both ways with people being hurt at actions committed against the avatar or finding pain relief in immersive VR (Matamala-Gomez, Donegan, Bottiroli, Sandrini, Sanchez-Vives & Tassorelli, 2019)—an understanding of the role of the avatar in the identity narrative within the pedagogical space needs to take the avatar sign as an intermediary in a two-way communication.The avatar is not a shield from external communications but a component within the transmedia identity of the student.

Dynamic avatars that overlay the physical user are commonplace with the virtual reality forums—such as VRChat and Alt-Space. Such spaces are still relatively avant-garde, maintaining a niche user base that has yet to cross into the mainstream.However, the pandemic demonstrated a potential use for these virtual forums as socially distanced, immersive, conference spaces. The simulation of the physical conference experience could potentially aid a transition from the static windows of the conference call setup like Zoom or Skype to the individual VR experience within a 3D space. Such a 3D space enables the use of models or objects within a lecturer that can be presented in real-time, offering an extra level of interactivity to the experience.

This suddenly requires a user to create an avatar for themselves as there is no ability to use a live representation of the physical self within the 3D virtual world.Within a virtual forum like VRChat, users will often apply models that are either specifically designed for them or that they have downloaded pre-made. A common aspect of VRChat is the prevalence of models from anime or popular culture. Where the line is drawn between the “Pygmalion” fantasy, to use Yuji Sone’s terminology for a desire by some to design a subservient woman (2017, p.23, p. 140), the furry subculture, and the school space, is going to take open discussion within the wider society. In terms of digital literacy and identity, we must consider whether there are some models that are “deviant” and where the distinction is drawn, taking race, age,religion, and so on into account.

The rights of artists who create models that are subsequently used by others are important—if a model becomes the sole indicator of a user’s identity, who owns the right to that identity? Recently there have been discussions on the representations of tattoos of real celebrities in video games, where the artist of the tattoo has claimed ownership of the image as copyrighted14. The implication is that the art of the tattoo remains the property of the artist, while simultaneously constituting a part of the identity of the celebrity.

Avatars allow for an agency of self-representation with few parameters. In the classroom, the teacher could easily present themselves as a character of varying authority—a cute bunny or a scary monster for example. Power roles and education have been extensively debated, (Anagnostopoulos et al., 2005; Read, 2008; Doubler,Harlen, Paget & Asbell-Clarke, 2003). However, Doubler et al. state: “Face-to-face instruction relies on oral communication, while online learning unfolds primarily through the written word” (2003). This seems outdated in today’s virtual learning environment where Zoom, Skype or a virtual space allows for oral communication in real-time. This suggests that the online teacher still maintains an authority regarding the dissemination of information, but the placement of the teacher has changed.

Within the conference call or VLE layout there is no “front”, with the digital windows situated around the classroom screen. Within the virtual classroom that uses virtual reality,there is the possibility to place the podium at the front, emulating the space of the physical spaces. Of course, just because wecancreate mirrors of traditional physical power structures, does not mean weshould(Ponti & Ryberg, 2004). How we contextualise the identity narrative within the education space requires comprehension of the signs, virtual and physical, and the relationships they create to each other.

Finally, we close our discussion on identity and digital literacy by asking, if the teacher themselves presents as an avatar, would that increase or decrease efficacy of the online teaching? As we have discussed, anthropomorphic avatars can be trusted,but they can be dehumanised. They lack sophisticated facial cues—currently. While there is research into learning within VR, there seems to be a lack of research into the role of teachers’ identity in teaching, and the effect avatars might have on such trust.Studies within the medical field show positive responses towards face-to-face contacts mediated via avatars (Shershneva, Kim, Kear, Heyden, Heyden, Lee & Mitchell,2014). However, they cite a lack of sophisticated facial cues as a negative, but recent developments in facial expression mapping could alleviate that issue. Although the reasons forwhya teacher (or a pupil) would use an avatar needs significant society-wide discourse, there is evidence—such as Shershnevaet al. and Nowak et al.—that suggests humans respond equally well to virtual or physical face-to-face communications. There may be positives to using an avatar—such as transgender identity—or negatives—such as “passing” to avoid racism.

There are two key aspects of communication—the presentation of signs and the interpretation of the signs. The interpretation imbues the sign with meaning, but the presentation of the sign influences this interpretation as signs are a part of a syntax that is read like any language. Each sign places itself in a structure with others,ultimately forming a text that can be read by the addressee. Ergo, the identity of the user, filtered through the virtual space, the editing software, or completely represented by the avatar, exists as a transmedial, multimodal, narrative.

5. Conclusion

The concerns we raise relate to student identity within the classroom space, and how understanding the semiotics of the signs used within such spaces—a model of digital literacy—enables the reading of signs via different modes and mediums in a holistic manner. The collaboration of image, sound, gesture, speech, and written text to create the space, the narrative, and the identity, is evidently unique within the emerging sphere of VR+ education, and the underlying architecture is a critically underexamined aspect of potential bias in a user’s identity narrative.

Understanding the signs and space as both student and teacher will help designers ensure future VLE’s allow for student-focused teaching, albeit one that is guided by the teacher at an individual level rather than biased by the programmers at a cultural (or commercial) level.

The surrounding ecosystem and additional programmes that operate external to the classroom, like dynamic avatars, cannot be ignored when considering digital literacy within the wider context. The signs within the virtual environment can be esoteric to specific online communities—as with natural language—and an awareness of the fandoms and societies can help provide valuable contextual information—meaning is a web, not an isolated, binary, relationship.

The virtual learning environment can be positive in allowing users to learn without restriction to their gender or sexuality, but it can also lead to abuse and stereotyping.The virtual presence can be both an aid to mental health and a hindrance. We have demonstrated how the interpretation and presentation of signs of identity within the online environment can lead to positive exploration or negative bias depending on the wider context (both online and offline). Additionally, the individual’s own awareness of the significance of the digital sign, on their own self-narrative formation and others,within the transmedial space of mixed reality education requires wider discussion as a global society. Further studies into the affects and effects of education in VR+environments, with focus on the semiotics, for both the student and teacher, must be performed to help advance such a digital literacy for the future.

Notes

1 https://www.interaction-design.org/literature/book/the-encyclopedia-of-human-computerinteraction-2nd-ed/semiotics

2 We prefer the term “physical” to “real” as a virtual is real in the sense that it exists and is viewable.

3 https://www.youtube.com/watch?v=blDfqqCDQfU

4 https://www.prnewswire.com/news-releases/iqiyi-to-launch-dimension-nova-chinas-firstvirtual-idol-variety-show-301127272.html and https://www.iq.com/play/1b41ci5v23s

5 https://www.skype.com/en/features/skype-translator/

6 https://www.interaction-design.org/literature/book/the-encyclopedia-of-human-computerinteraction-2nd-ed/semiotics

7 At least,somenatural languages.

8 https://www.washingtonpost.com/news/worldviews/wp/2015/04/23/the-worlds-languagesin-7-maps-and-charts/

9 https://enacademic.com/dic.nsf/enwiki/211978

10 https://wowfilters.com/91/furry-raccoon-face-mask/

11 https://www.pocket-lint.com/apps/news/151723-how-to-turn-yourself-into-a-potato-andother-filters-for-zoom-meetings-and-team-calls

12 https://www.faceapp.com/

13 https://www.microsoft.com/en-us/research/people/mlg/?ocid=msr_award_mgray_tw

14 https://www.hollywoodreporter.com/thr-esq/wwe-headed-to-trial-for-copying-wrestlerstattoos-for-video-game