Adaptable User Interface Based on the Ecological Interface Design Concept for Multiple Robots Operating Works with Uncertainty

Problem statement: When supervising a team of semi-autonomous robots, the operator’s task is not only the manipulation of each robot but also achievement of the top goal that is assigned to the entire team of humans and robots. The operators are demanded to comprehend highly complex states and make appropriate decisions in dynamic environment using the limited cognitive resources. The Ecological Interface Design (EID) is a design approach based on visualization of constraints in study environment onto the interface to reduce the cognitive workload. Our research question was “How should we design Human-Robot Interface (HRI) that allows operators to understand states of multiple robots systems working in environment with uncertainty?” Approach: Our proposed method was Adaptable User Interface (AUI) with EID. The interface is a framework that allowed the operators to take selectable use of displays indicating different types of functional information. Two types of expressions of functions were defined for indicating states of functions: indication of functional disposition and of effectiveness. Three cognitive experiments were conducted to reveal potential efficacy of the method using an experimental test-bed simulation. Results: The results of the first and second experiments showed that usefulness of functional indications is different from different works, tasks and operators. The third experiment demonstrated that operators could select and use appropriate information among a set of indicators with different types and levels of functional information, which are adaptive for tasks, their strategies, their skills and available information and knowledge about the tasks. Conclusion: From the remarks, it can be concluded that the proposed method (AUI+EID) is a useful and feasible HRI method for multiple robot operations, even knowledge about the work is not sufficient to build a set of dedicated indicators each has only necessary information for a situation.


INTRODUCTION
Human-robot interface for multiple robot operations: Multiple robot systems are one of effective solutions to implement robust, flexible and adaptable systems that can be used in various conditions. The systems are increasingly being used in environments that are inaccessible or dangerous to humans (Stoeter et al., 2002;Chaimowicz et al., 2005).
When supervising a team of semi-autonomous robots, the operator's task is not only the manipulation of each robot but also achievement of the top goal that is assigned to the entire team of humans and robots. Clearly, support of the operator's skill-based behaviors is important (Kaber et al., 2006;Nam et al., 2009). Equally, it is important to support human operators in their understanding of the overall state of a study-inprogress and the situation around it using a systemcentered view. Although cognitive resources of humans are limited, operators are demanded to understand highly complex states and make appropriate decisions in dynamic environment (Calhoun et al., 2005;Goodrich et al., 2005). Fong and Thorpe (2001) claim that Human-Robot Interface (HRI) "must provide mechanism for the operator and the robot to exchange information at different levels of detail or abstraction" because levels of operator's task are different at different levels. When a task of an operator is manipulation of each robot, he or she should comprehend local situation of the robots. When achievement of top goals is an operator's task, he or she should consider overall situation of a whole human-robot system.
A main goal of our project is the development of an HRI design method for human supervision of multiple robot systems, providing information about states of functions that are necessary to achieve the top goal of a human-robot system and elementary techniques necessary for real implementation of the HRI.
An HRI design method with the ecological interface design approach: Ecological Interface Design (EID) was adopted as a basic concept for the method to solve the problem with mental workload of operators. The EID is a design approach based on visualization of constraints in study environment onto the interface to reduce the cognitive workload during state comprehension (Vicente, 1999;2002;Vicente and Rasmussen, 1992). Information on function is identified using the Abstraction-Decomposition Space (ADS). An ADS is a framework for representing the functional structures of work in a human-machine system that describes hierarchical relationships between the top goal and physical components with multiple viewpoints, such as abstraction and aggregation (Rasmussen, 1986). Since the operator's comprehension of the functional states based on the ADS is an essential view for the study, supporting the view is crucial for them to make operational plans and execute the plans appropriately under high-workload conditions (Miller and Vicente, 2001).
Several attempts have been made to apply the design approach to HRI. Sawaragi et al. (2000) applied the EID concept to HRI to support naturalistic collaboration between a human and a robot at the skill level via a 3D display. Nielsen et al. (2007) proposed an ecological interface paradigm for teleportation of robots, which combines video, map and robot-position information into a 3-D mixed reality display. In both studies, the target level of operation is limited to skillbased control on a robot. The methods can be used for direct operation of individual robot in a multi-robots system. Jin and Rothrock (2005) propose a functionbased interface display which indicates the state of communication between human operators and multiple robots in which the target function is limited to one. This indication can be used as a display design for representation of a function of a multi-robots system. Although these methods have the promising usages in designing of HRI for multi-robots systems, further studies are necessary to develop methods those achieve the requirements stated by Fong and Thorpe (2001).

Research question:
This study reviews three experimental studies conducted (Furukawa, 2008;2010) and discusses a research question about the human-robot interface design for multiple robot operations in environment with uncertainty. The question is "How should we design HRI that allow operators to understand states of multiple robots systems working in environment with uncertainty?" An HRI designer may confront this question when he or she finds out that the basic EID concept assumes they do have enough knowledge to construct ADS models and design displays for every situation (Lee et al., 2000).
When designers do have proper quantitative and qualitative models representing causes and effects in target system, it must be possible for them to design HRI indicating accurate status of functions in the system. On the other hand, their ADS models may not be mature enough to construct suitable HRI for the operations when they have missing links in knowledge about the functional conditions.

Methods:
Our proposed method is Adaptable User Interface (AUI) with EID. The interface is a framework that allows the operators to take selectable use of displays, each of them indicating different types of functional information.
As elementary techniques, two types of expressions of functions are defined for indicating states of functions. The one is indication of functional disposition and the other is indication of effectiveness. The indication of functional disposition indicates characteristics of means and ways of use to users. When we use a pair of scissors to cut a sheet of study, we may read an instruction book and check the specifications and the use of scissors to select an appropriate type. The second functional indication describes degrees of achievement of target goals. During we are cutting along a line, we may check an error from the line or smoothness of the edge to evaluate the achievement.
If operators do not have sufficient knowledge, that is an adequate mental model and cannot get enough information to estimate quality of the result using the means, the selection must be done by tryand-error manner.
Three cognitive experiments were conducted to reveal potential efficacy of the method using an experimental test-bed simulation. The two types of indications were examined to support the cognitive study of operators and reduce their workloads.

MATERIALS AND METHODS
A target task, HRI implemented in this study and three cognitive experiments conducted are given below.

The RoboFlag simulation platform:
This study uses the RoboFlag simulation, which is an experimental testbed modeled on real robotic hardware (Campbell et al., 2003). The chief goal of an operator' job is to take flags using semi-autonomous robots and to return to the home zone. One operator directs a team of robots to enter an opponent's territory capture the flag and return to their home zone without losing the flag. Defensive action takes the following form: A rival robot will be inactivated if it is hit by a friendly robot while in friendly territory. Because time constraints are severe, human operators need gain an understanding of the situation as rapidly as possible. Figure 1 shows the original display used for an operator to monitor and control his or her own team of robots (Campbell et al., 2003). Eight robots were used in this study. A circle around a robot indicates the detection range within which the robot can detect opponents. The current situation is unknown at most of the field. This is a cause of uncertainty in the target task.
The simulation provides two types of operations that operators can select: manual controls and automatic controls. In manual control mode, an operator indicates a waypoint to a robot by clicking the point on a display. Two types of automatic controls were implemented in this study. When Rush and Back (R and B) mode is assigned, the robot tries to reach the flag and returns home after it captures the flag. In Stop or Guard (S/G) mode, the robot tries to inactivate the opponent if an opponent robot comes into detection range.
Another cause of the uncertainty in the study is lack of knowledge about strategies of the opponent robots. Offensive and defensive tasks of the rival robots were fully automated by using the two types of automatic controls implemented in this study. The number of defensive robots was five and three for offensive robots. The offensive robots had six different courses to enter the player's territory and three types of the timings. A pair of settings was randomly selected from the alternatives. Because of the randomness, the participants had to operate their robots adaptively.
Functional modeling with the abstractiondecomposition space: Figure 2 shows outline of the ADS whose top is the Offence function (Furukawa, 2007). Two functions capture flag and take flag home, depicted below the function are the means of achieving the top function. To capture the flag, the robot needs to reach the flag (reach flag). At the same time, the robot should be in active mode (stay active) and avoid opponents: Avoid opponents is one of the necessary functions to achieve the goal.

Implementation of functional indications:
The following are descriptions of function-based indication designs. To specify each state of the function, expressions that graphically showed the state in the physical relations between each robot and the object was used. This has aimed to enable operators intuitive to understand the state of the functions and relationships between the functions. Functions were selected because results from the previous studies (Parasuraman et al., 2005) indicated that it was difficult to comprehend the states of the functions during plays.  • Defensive sectors: Cooperation between defensive robots is a type of defensive function realized by a team of robots. The picture illustrated in Fig. 3 is the functional indication designed for enabling an operator to be clearly aware of the state of the function. A fan-shaped sector, a defensive sector is where a robot in S/G mode has a high ability to intercept opponent robots coming through. Outside the defensive sector, the possibility of catching opponents is lower than within the sector. An operator can use spaces between the sectors as an indication of the defensive ability of the defensive robot team in the position  • Group effectiveness: The functional indicator is designed to show state of two functions which are higher than those in the Individual effectiveness. The first function is Defensive ability of the defensive robot team and the second is offensive ability of the offensive robot team. In this study, a team is formed based on their physical positions. Robots which are in same area are grouped into a team. Degree of achievement of the first function is evaluated by taking an average of defensive robot's degree for Inactivate opponents and degree of the second function is for shun opponents. Each calculated degree is indicated by changing color of the area (Fig. 6) Procedures of the experiments: The sets of these indications used in the three experiments are summarized in Table 1: • Experiment 1 (indications of functional disposition): The aim of the first experiment was to evaluate the basic efficacy of the indication of functional disposition under the condition that the wide variety of strategies was used for operations (Furukawa, 2008). The task was a sudden-death type game. A game ends when the player or the opponent team reach the goal  • Experiment 2 (indications of effectiveness): The second experiment was conducted to analyze usage and usefulness of the indications of effectiveness with higher and lower levels of abstraction, the individual effectiveness and the group effectiveness, under two typical tasks in which the goals and cognitive workloads were different (Furukawa, 2009) The first task was the sudden-death type game and the second was a time-limit type game. Players play two games in same time. The goal of players is to take more flags to the home zone than the opponents. The time limit was set 3 min. It is expected that mental workload of the players in relatively high:

Subjective evaluation on usefulness of functional indications:
The results about subjective evaluation on the functional indicators by the participants are illustrated. In an interview immediately after the main experiments, they were asked to explain the strategies they used during the main experiments, their usages of the indications of functions during the experiments and the usefulness of the information in completing their missions. Table 2 shows the numbers of participants who evaluated "useful" or "little useful" for each indication.
The ratio of participants who considered the indication of functional disposition, that is, the Defensive sector, to be useful is much larger at tasks with higher workload than with lower workload. All participants used the indicator at the time-limit type game.
In the case of the indication of effectiveness with the lower task, the ratio of participants who used the individual effectiveness, namely two thirds, was as same as that of who used the group effectiveness. At the higher workload task, the ratio for the individual effectiveness was a third of the participants. On the other hand, the ratio for the group effectiveness was 75%, which is larger than that at the lower workload. Table 3 shows the numbers of won games with the indications of effectiveness for the sudden-death and time-limit type games. Through T-test, there was no significant difference between the types of display in the lower workload condition. In higher workload situation, the test shows that the number with Field display was significantly larger than with the Individual display (t = -3.079, df = 11, p = 0.010*).

Performance data about indication of effectiveness:
Performance data at single-use and combined-use of indications: Table 4 shows the numbers of won games using a type of indications or two types simultaneously and the ratio where forty games were conducted. The numbers are sum of all participants' results. Statistical analysis on the parameters could not detect any significant differences between types of displays. However, at least, the results show no sign of any ill effects caused by the combined use.      (Table 2). When operators have enough knowledge and information possible to comprehend states of target functions, the necessity of the indication is not high. In higher workload tasks, functional indication with higher level of abstraction (the group effectiveness) was necessary and should be indicated in the display. In this case, evaluation on achievement of target functions is difficult tasks for many of operators. The information about effectiveness of means is important for players to make up appropriate plan. According to the research results, usefulness of functional information depends on characteristics of target works (e.g., goals, constraints and mental workload necessary to achieve) and characteristics of operators (e.g., mental models, skills, cognitive abilities and strategies). From this remark, it can be said that selection of appropriate information for each situation is a big issue in display designing and therefore the need of the AUI is high at designing HRI for real world tasks.

Efficacy of combined use of indications:
According to results of the interviews, operators could select and use appropriate information among a set of indicators with different types and levels of functional information, which are adaptive for tasks, their strategies, their skills and available information and knowledge about the tasks. Furthermore, some participants took jointly uses of different types of functional indications to overcome the uncertainty in the study. These results may suggest that the proposed HRI is appropriate for providing necessary information to operators of the multiple robot system, even if the necessary functional information depends on individual differences.

Summary:
These three experimental studies provided empirical evidences that the proposed adaptable interface which is built up with indications of functional disposition and the indications of effectiveness has basic efficacy to support operators in supervision on multiple robot systems with uncertainty. Combined use of functional indicators "Adaptable User Interface with the Ecological Interface Design approach (AUI+EID)" must be a useful design for human-robot interface CONCLUSION A main goal of our project is the development of a human-robot interface design method for supervision of multiple robot systems, based on the Ecological Interface design approach using multi-level representations of functions necessary to achieve top goals of human-robot systems.
This study describes experimental studies conducted to discuss efficacy of an Adaptable User Interface (AUI) with EID for multiple robot work with uncertainty. Two types of indications are used for representing states of functions. The one is indication of functional disposition and the other is indication of effectiveness. The adaptable interface with the two types of functional indications was examined through a cognitive experiment with experimental test-bed simulation in which knowledge and information about situations are limited. The results demonstrate that operators might be able to select and use appropriate indication (s) among a set of displays indicating functional information with different approaches and levels of abstraction. The use is modified adaptively depending on availability of information and knowledge about situations. From the remarks, it can be concluded that the proposed method (AUI+EID) is a useful and feasible HRI method for multiple robot operations, even knowledge about the study is not sufficient to build a set of dedicated indicators each has only necessary information for a situation and an operator.
In this study, states of functions were defined using available information without any estimation. Some participants wanted to use indications about future states of functions. Possible approaches are techniques using qualitative and quantitative models to predict future states of functions, or to estimate current states of functions which are unknown. In the development of the techniques, "uncertainty" must be a high hurdle to overcome.