COFI; A Framework for Modeling Interaction in Human-AI Co (PDF)

COFI: A Framework for Modeling Interaction in Human-AI Co-Creative Systems Jeba Rezwana, Mary Lou Maher Software and Information Systems University of North Carolina at Charlotte 9201 University City Blvd, Charlotte, NC 28223, US [email protected], [email protected] Abstract Human-AI co-creativity involves both humans and AI collaborating on a shared creative product as partners. In a creative collaboration, interaction dynamics, such as turn-taking, contribution type, and communication, are the driving forces of the co-creative process. There- fore an interaction model is an essential component for designing effective co-creative systems. There is rela- tively little research about interaction design in the co- creativity field, which is reflected in a lack of focus on interaction design in many existing co-creative systems. This paper focuses on the importance of interaction de- sign in co-creative systems with the development of the Co-Creative Framework for Interaction design (COFI) that describes the broad scope of possibilities for inter- action design in co-creative systems. Researchers can use COFI for modeling interaction in co-creative sys- tems by exploring the possible spaces of interaction. Introduction In human-AI co-creative systems, humans and AIs col- laborate in a creative process as creative colleagues, and the focus is on co-creative partnerships in contrast to cre- ativity support tools (Davis 2013). Creative collaboration involves interaction among collaborators, and the shared creative product is more than each individual alone could achieve (Sawyer and DeZutter 2009). Sonnenburg demon- strated communication as the driving force of collaborative creativity (Sonnenberg 1991). Interaction is a basic part of co-creative systems as both the human and the AI ac- tively participate and interact with each other. Designing co-creative systems have many challenges due to the open- ended nature of the interaction between the human and AI (Davis et al. 2016). Bown asserted that the success of a creative system’s collaborative role should be further inves- tigated concerning interaction design as interaction plays a key role in the creative process (Bown 2015). Interaction design is often an untended topic in the co- creativity literature despite being a fundamental property of co-creative systems. In recent years, researchers designed many co-creative systems that are very intriguing and cre- ative, yet sometimes users fail to maintain their interest and engagement while collaborating with the AI due to the unim- pressive quality of collaboration. An adequate interaction model dramatically improves the quality of the collabora- tion and usability of a system. Therefore, as a young field, a holistic framework that captures the scope of interaction de- sign is necessary. A good starting point to investigate ques- tions about interaction modeling is studying creative collab- oration in humans (Davis et al. 2015). Understanding the factors of human collaboration can be a tool to build the foundation for the development of systems that can augment or enhance creativity in humans (Mamykina, Candy, and Edmonds 2002). The literature on computational creativ- ity and computer-supported collaborative work (CSCW) can also help identify interaction components related to human- AI co-creativity. In this paper, we present the Co-Creative Framework for Interaction design (COFI) that defines interaction compo- nents in a co-creation to describe the broad scope of possi- bilities for interaction design in co-creative systems. These interaction components represent various aspects of a co- creation, such as participation style, contribution type, and communication between humans and the AI. COFI is in- formed by the literature on human collaboration, CSCW, computational creativity and human-computer co-creativity. We adopted interaction components based on a literature re- view and adapted the components to concepts relevant to co- creativity. We argue that COFI can be used as a guide when modeling interaction in co-creative systems as researchers can use COFI for exploring the possible spaces of interac- tion. COFI can also be useful while analyzing and interpret- ing the interaction design of existing co-creative systems. Related Work In co-creative systems, humans and AI both contribute as creative colleagues in the creative process (Davis 2013). Creativity that emerges from human-AI interaction cannot be credited either to the human or to the AI alone and sur- passes both contributors’ original intentions as novel ideas arise in the process (Liapis, Yannakakis, and Togelius 2014). Designing interaction in co-creative systems has unique challenges due to the spontaneity of the interaction between the human and the AI (Davis et al. 2016). A co-creative AI agent needs continual adjustment and adaptation to cope with human strategies. Mamykina et al. argued that by un- derstanding the factors of collaborative creativity among hu- mans, methods can be devised to build the foundation for the Proceedings of the 12th International Conference on Computational Creativity (ICCC ’21) ISBN: 978-989-54160-3-5 444  development of systems that can augment or enhance collab- orative creativity (Mamykina, Candy, and Edmonds 2002). In the field of co-creativity, interaction design includes various parts and pieces of the interaction dynamics between users and the AI, such as - participation style, communi- cation, contribution type etc. Bown argued that the most practiced form of evaluating creative systems is mostly theo- retical and not empirically well-grounded and suggested in- teraction design as a way to ground empirical evaluations of computational creativity (Bown 2014). Yee-King and d’Inverno also suggested a need for integration of interac- tion design practice into co-creativity research (Yee-King and d’Inverno 2016). There is a lack of a framework for in- teraction design, which is necessary to explain and explore the possible interaction spaces and compare and evaluate the interaction design of existing co-creative systems to improve the practice of interaction modeling. Interaction among the individuals in collaboration makes the process emergent and complex. For investigating human collaboration, many researchers stressed the importance of understanding the process of interaction. Fantasia et al. pro- posed an embodied approach of collaboration that considers collaboration as a property and intrinsic part of interaction processes (Fantasia, De Jaegher, and Fasulo 2014). Schmidt defined CSCW as an endeavor to understand the nature and characteristics of human collaboration to design adequate computer-based collaborative technologies (Schmidt 2008). Co-Creative Framework for Interaction Design (COFI) We develop and present Co-Creative Framework for Interac- tion design (COFI) as a guiding tool that describes the broad scope of possible spaces for interaction design in co-creative systems. This framework describes various aspects involved in the interaction between the human and the AI. COFI is informed by research on human collaboration, CSCW, com- putational creativity, and human-computer co-creativity. The primary categories of COFI are based on two types of interactional sensemaking of collaboration as described by Kellas and Trees (Kellas and Trees 2005): interaction between collaborators and interaction with the shared prod- uct. Interaction with the shared product, in the context of co-creative systems, describes interaction aspects related to the creation of the creative content. Interaction between col- laborators explains how the interaction between the human and the AI is unfolding through time which includes turn- taking, roles, timing of initiative, communication, etc. Thus, COFI characterizes relational interaction dynamics between the collaborators (human and AI) as well as functional as- pects of interacting with the shared creative product. We choose the Kellas and Trees framework for the primary cat- egories of COFI because they used their framework as a tool for explaining and evaluating the interaction dynamics in hu- man creative collaboration in joint storytelling. As shown in Figure 1, we further divide the two primary categories of COFI, interaction with collaborators and in- teraction with the shared product, into four subcategories. Interaction between collaborators is divided into collabora- tion style and communication style, inspired by research in CSCW and HCI. Interaction with the shared product is di- vided into the creative process and creative product, inspired by research in creativity and, more specifically, computa- tional creativity. CSCW literature discusses collaboration mechanics among humans to make effective CSCW sys- tems, whereas creativity literature talks about creative pro- cess and product. The interaction components are child cat- egories of the four main subcategories and are adapted to the context of human-AI co-creativity. COFI is not a com- plete ontology and can continue to expand as new interaction components emerge. Interaction between Collaborators (Human and AI) This section presents components related to the relational interaction dynamics between the human and the AI as co- creators. As shown in Figure 1(a), interaction between col- laborators is divided into two subcategories which are col- laboration style and communication style. Collaboration Style Collaboration style is about different parts and pieces of interaction between humans and the AI, related to the nature of the co-creation. The following sub- sections describe each interaction component in this cate- gory. Participation Style: Participation style in COFI refers to whether the collaborators can participate and contribute si- multaneously, or one collaborator has to wait until the part- ner finishes a turn. Therefore, participation style in COFI is categorized as parallel and turn-taking. Categorization of participation stems from work on cooperation in the litera- ture (Johnson and Johnson 2005; Liu, Saito, and Oi 2015). Task Distribution: Task distribution refers to the distribu- tion of tasks among the collaborators in a co-creative sys- tem. In COFI, there are two types of task distribution, same task and task divided. When it is same task, there is no division of tasks among collaborators and the collaborators take part in the same task. For example, in a human-AI co- creative drawing, both co-creators do the same task, gener- ating the drawing. In a task-divided distribution, the main task is divided into specific sub-tasks and the sub-tasks are distributed among the collaborators. For example, in co- creative poetry, the user can generate a poem while the AI agent can evaluate the poetry. This component of COFI emerged from discussions of the two interaction modes pre- sented by Kantosalo and Toivonen: alternating and task di- vided co-creativity (Kantosalo and Toivonen 2016). Timing of Initiative: In a co-creative setting, the timing of both parties’ initiative taking can be scheduled beforehand, or it can happen naturally in real-time. If the timing of the initiative is fixed in advance, in COFI, it will be addressed as planned. If collaborators initiate their contribution naturally without any prior plan or fixed rules, it will be addressed as spontaneous. Timing of initiative should be chosen based on the motivation. Spontaneous timing is suitable for in- creased emergent results, whereas planned timing is suitable for systems where users want inspiration or help in a specific way for a particular aspect. Salvador et al. discussed tim- Proceedings of the 12th International Conference on Computational Creativity (ICCC ’21) ISBN: 978-989-54160-3-5 445  Figure 1: Co-Creative Framework for Interaction Design (COFI): On the left (a) Components of Interaction between the collaborators. On the right (b) Components of Interaction with the Shared Product ing of initiative in their framework for evaluating groupware for supporting collaboration (Salvador, Scholtz, and Larson 1996). Communication Style Communication is a vital compo- nent in any collaboration for the co-regulation between the collaborators and helps the AI agent make the proper deci- sion in a creative process. Communication style includes different kinds of channels to communicate between users and the AI. Human to AI Intentional Communication: Human to AI intentional communication channels represent the possible ways a human agent can intentionally communicate to the AI agent to provide feedback and convey important infor- mation to each other. Gutwin and Greenburg proposed a framework for groupware that discusses the mechanics of collaboration and it includes intentional communication as a major element of collaboration mechanics (Gutwin, Green- berg, and Roseman 1996). In COFI, human to AI communi- cation channel includes direct manipulation, voice, text and embodied communication. The user can directly manipulate the co-creative system by clicking buttons for giving instruc- tions or feedback or inputs and providing user preferences by selecting from AI-provided options. Using the whole body or gestures for communicating with the computer will be referred to as embodied. According to HCI modalities, intentional communication from human to AI includes di- rect manipulation, gesture, text, and voice (Nigay 2004). Human to AI Consequential Communication: In COFI, human to AI consequential communication channels rep- resent the ways a co-creative AI agent can track and col- lect unintentional or consequential information from the hu- man user, such as eye-tracking, facial expression tracking and embodied movements. Tracking consequential details from the human is essential to perceive user preference, user agency and engagement. Gutwin and Greenburg reported consequential or unintentional communication as a major el- ement of collaboration mechanics, in addition to intentional communication (Gutwin, Greenberg, and Roseman 1996). AI to Human Communication: Humans expect feed- back and evaluation of their contribution from collaborators. Therefore, if the AI agent can communicate their status and feedback for a contribution, it would make the co-creation more balanced as the AI agent will be perceived as an equal partner rather than a mere tool. The AI to user communi- cation can include text, voice, visuals (icons, image, ani- mation), haptic and embodied communication according to HCI modalities (Nigay 2004). Proceedings of the 12th International Conference on Computational Creativity (ICCC ’21) ISBN: 978-989-54160-3-5 446  Interaction with the Shared Product Interaction components related to the shared creative prod- uct in a co-creative setting are discussed in this section and illustrated in Figure 1(b). Interaction with the shared prod- uct is divided into two subcategories, creative contribution to the product and creative process. Creative Process Creative process characterizes the se- quence of actions that lead to a novel, and creative produc- tion (Lubart 2001). In COFI, there are three types of cre- ative processes that describe the interaction with the shared product: generation, evaluation, and definition. During a creative generation, creative artifacts and ideas are produced in a specific conceptual description. In a creative evaluation, produced ideas, artifacts or concepts get assessed to be more refined and appropriate for the creative objective. In a cre- ative definition process, collaborators determine and prepare the creative conceptual space. For example, a co-creative AI agent can define the attributes of a fictional character before a writer starts to write about the character. The basis of this categorization is the work of Kantosalo et al. that defines the roles of the AI as generator, evaluator, and concept definer (Kantosalo and Toivonen 2016). Creative Product The creative product is the idea or con- cept that is being created. Creative product has two interac- tion components, contribution type and contribution similar- ity. for example (Boden 1992) Contribution Type: In a co-creation, an individual can contribute in different ways to the shared product. Co- creators can generate new elements, extend the existing con- tribution, modify or refine the existing contribution. The pri- mary contribution types in COFI are: ‘create new’, ‘extend’, ‘transform’ and ‘refine’. ‘Extend’ refers to extending the contribution of the partner or adding on to the partner’s con- tribution. Generating something new or creating new objects is represented by ‘create new’, whereas ‘transform’ conveys turning the partner’s contribution into something totally dif- ferent. ‘Refine’ is evaluating and correcting the partner’s contributions with a similar type of contribution. Contribu- tion types are adopted and adapted from Boden’s categories of computational creativity based on different types of con- tribution: combinatorial, exploratory, and transformational (Boden 1998). Contribution Similarity: Contribution similarity refers to the degree of similarity or association in terms of the con- tribution compared to the partner’s contribution. Both con- vergent and divergent exploration have value in a creative process. Basadur et al. asserted that divergent thinking is re- lated to the ideation phase and convergent thinking is related to the evaluation phase (Basadur and Hausdorf 1996). Conclusions As a growing field, human-computer co-creativity lacks sig- nificant research on interaction models and their implica- tions in co-creative systems. Human-AI co-creativity re- search needs a holistic framework that captures aspects and components of interaction to design effective co-creative systems. In recent years, a few frameworks have been devel- oped about interaction design in co-creative systems. How- ever, they lack a focus on interaction components related to the interaction between collaborators as distinct from inter- action components related to the shared product. We de- velop and describe COFI as a new framework to provide a way for researchers to explore the design space of interac- tion for a specific system. COFI will provide useful guide- lines for interaction modeling while developing co-creative systems. COFI can also be beneficial while investigating and interpreting the interaction design of existing co-creative systems. As a framework, COFI is expandable as other in- teraction components can be added to it in the future. By establishing COFI, we can look for the relationship between different interaction designs in co-creative systems and dif- ferent creative outcomes. COFI can be also used to develop co-creative systems that can improve user engagement with effective collaboration strategies through adequate human- AI interaction. References Basadur, M., and Hausdorf, P. A. 1996. Measuring diver- gent thinking attitudes related to creative problem solving and innovation management. Creativity Research Journal 9(1):21–32. Boden, M. 1992. The Creative Mind. London: Abacus. Boden, M. A. 1998. Creativity and artificial intelligence. Artificial Intelligence 103(1-2):347–356. Bown, O. 2014. Empirically grounding the evaluation of creative systems: Incorporating interaction design. In ICCC, 112–119. Bown, O. 2015. Player responses to a live algorithm: Conceptualising computational creativity without recourse to human comparisons? In ICCC, 126–133. Davis, N.; Hsiao, C.-P.; Popova, Y.; and Magerko, B. 2015. An enactive model of creativity for computational collab- oration and co-creation. In Creativity in the Digital Age. Springer. 109–133. Davis, N.; Hsiao, C.-P.; Yashraj Singh, K.; Li, L.; and Magerko, B. 2016. Empirically studying participatory sense-making in abstract drawing with a co-creative cogni- tive agent. In Proceedings of the 21st International Confer- ence on Intelligent User Interfaces, 196–207. Davis, N. M. 2013. Human-computer co-creativity: Blend- ing human and computational creativity. In Ninth Artificial Intelligence and Interactive Digital Entertainment Confer- ence. Fantasia, V.; De Jaegher, H.; and Fasulo, A. 2014. We can work it out: an enactive look at cooperation. Frontiers in psychology 5:874. Gutwin, C.; Greenberg, S.; and Roseman, M. 1996. Workspace awareness in real-time distributed groupware: Framework, widgets, and evaluation. In People and Com- puters XI. Springer. 281–298. Johnson, D. W., and Johnson, R. T. 2005. New develop- ments in social interdependence theory. Genetic, social, and general psychology monographs 131(4):285–358. Proceedings of the 12th International Conference on Computational Creativity (ICCC ’21) ISBN: 978-989-54160-3-5 447  Kantosalo, A., and Toivonen, H. 2016. Modes for cre- ative human-computer collaboration: Alternating and task- divided co-creativity. In Proceedings of the seventh interna- tional conference on computational creativity, 77–84. Kellas, J. K., and Trees, A. R. 2005. Rating interac- tional sense-making in the process of joint storytelling. The sourcebook of nonverbal measures: Going beyond words 281. Liapis, A.; Yannakakis, G. N.; and Togelius, J. 2014. Com- putational game creativity. ICCC. Liu, T.; Saito, H.; and Oi, M. 2015. Role of the right in- ferior frontal gyrus in turn-based cooperation and competi- tion: a near-infrared spectroscopy study. Brain and cogni- tion 99:17–23. Lubart, T. I. 2001. Models of the creative process: Past, present and future. Creativity research journal 13(3-4):295– 308. Mamykina, L.; Candy, L.; and Edmonds, E. 2002. Collabo- rative creativity. Communications of the ACM 45(10):96–99. Nigay, L. 2004. Design space for multimodal interaction. In Building the Information Society. Springer. 403–408. Salvador, T.; Scholtz, J.; and Larson, J. 1996. The den- ver model for groupware design. ACM SIGCHI Bulletin 28(1):52–58. Sawyer, R. K., and DeZutter, S. 2009. Distributed creativity: How collective creations emerge from collaboration. Psy- chology of aesthetics, creativity, and the arts 3(2):81. Schmidt, K. 2008. Cooperative work and coordinative practices. In Cooperative Work and Coordinative Practices. Springer. 3–27. Sonnenberg, F. K. 1991. Strategies for creativity. Journal of Business Strategy. Yee-King, M., and d’Inverno, M. 2016. Experience driven design of creative systems. Proceedings of the 12th International Conference on Computational Creativity (ICCC ’21) ISBN: 978-989-54160-3-5 448