Towards a Virtual Operating Environment
Exploring Immersive Virtual Interface Design Using a Simple VR Image Viewer


Appendices

1. Original Testbed Application Specification

Required: A small application to test various aspects of interface design on a variety of users.

A SIMPLE DISK-SPACE NAVIGATING IMAGE VIEWER:

This application should provide a good range of interface principles. The program will take two forms: a realworld form and an abstract form.

1.1 The realworld form:

The user is immersed into an environment that models everyday surroundings. A shelf will hold various books which, when selected, can be paged to quickly browse the images, which can be selected for display in the picture frame (see Figure A1 below).



\includegraphics{images/rwSketch2.ps}



Figure A1: Sketch of the realworld application

1.1.1 Selection

It is planned to keep things simple to avoid confusion about what part of the interface is effective. To select a book or image the hand is used to grab the object, and then it is moved to the relevant place. Paging through the book is done by grabbing a blank part of the page.

1.2 The abstract form:

This form provides the user with a symbolic representation of disk space. This can either be an amalgamated view (a 3D grid landscape) or a network view (connected nodes with 3D grid landscapes alongside). Figure A2 shows a sketch of the network view. The amalgamated view would have a large single disk space block. Clicking on a block of the grid brings up a container which is used to browse the images or select an image to be displayed in the frame (which is similar to the frame in the realworld application).



\includegraphics{images/abSketch.ps}



Figure A2: Sketch of the abstract application

1.2.1 Selection

As with the realworld version simple selection of images, blocks in grids, nodes and containers is done by grabbing with the hand.

1.2.2 Proportionality

The picture frame is envisaged as a fixed structure, whereas the user can move around the collection of nodes to get to different disk spaces (by grabbing a node or possibly using a separate movement widget or hand action).

1.3 What we get out of all this:

Hopefully, a better understanding of the 3D human computer link. The basic interactions - grabbing and placing an object, or moving around the network - should give an idea of the best ways to do the simple operations in a 3D environment. The two different interfaces will allow comparison of the effectiveness of representations which are drawn from the real world with those that are drawn from imagination and more abstract associations.

Switching between these forms (which will be added afterwards) will provide a mechanism to test meta-level controls, which are an important part of the operating environment.

Insight into control within an application may be gained from the mechanism for moving between different views in the abstract version.

1.4 Possible extensions:


Matthew Mundell, 23 March 1999

2. Screenshots

\resizebox*{0.8\textwidth}{!}{\includegraphics{images/rwpress.ps}}

Figure B3: Pressing an image on the book

\resizebox*{0.8\textwidth}{!}{\includegraphics{images/rwpturn.ps}}

Figure B4: Pointing to turn the book page

\resizebox*{0.8\textwidth}{!}{\includegraphics{images/rwheld.ps}}

Figure B5: Grabbing the book

\resizebox*{0.8\textwidth}{!}{\includegraphics{images/rwincase.ps}}

Figure B6: The book in the case

\resizebox*{0.8\textwidth}{!}{\includegraphics{images/abpress.ps}}

Figure B7: Pressing an image on the board

\resizebox*{0.8\textwidth}{!}{\includegraphics{images/abpoint.ps}}

Figure B8: Pointing a new set of images onto the board

\resizebox*{0.8\textwidth}{!}{\includegraphics{images/abgrab.ps}}

Figure B9: Grabbing the ring

\resizebox*{0.8\textwidth}{!}{\includegraphics{images/aboncyl.ps}}

Figure B10: The ring sitting on the cylinder

3. Questionnaires

3.1 Realworld Image Viewer

VIRTUAL REALITY QUESTIONNAIRE

IMAGEVIEWER USABILITY EVALUATION

The purpose of this questionnaire is to use the ImageViewer program to test various aspects of virtual reality systems.

Please answer the questions after going through the ImageViewer test run. Where there is a choice mark the block with the option that best suits the question. Don't feel obliged to answer any of them. All information will be treated with complete confidentiality.

Thanks for your assistance, and enjoy the virtual experience.



Age         



Sex   M     F  
     



Please give a short description of your computer skill and experience: (e.g. no experience, Computer Science masters student)



                                                                                                                                                





Have you been exposed to any virtual reality (VR) systems before?   Y     N  
     



If so, please give a brief description:



                                                                                                                                                
 



Feedback Section

How much did you enjoy using the image viewer?



YAWN... NOT MUCH FUN 1 2 3 4 5 COOLEST THING I'VE DONE ALL WEEK





What was the letter displayed in one of the images from the collection?       


\begin{spacing}{1.00}
\vspace{0.375cm}
{\raggedright\begin{tabular}{c\vert c\ver...
... letter? }&
~~~~~~\\
\cline{2-2}
\end{tabular}\par }
\medskip {}
\end{spacing}

Do you like the way the scene is presented to you when you first start the program?



DISLIKED 1 2 3 4 5 LIKED



Could you suggest a better way to introduce someone to the environment?



                                                                                                                                                



How easily did you forget that your environment was not a real one? (i.e. how easy was it to feel like you were immersed in the environment?)



NOT VERY EASILY 1 2 3 4 5 VERY EASILY



What could have been done to improve the feeling of immersion?



                                                                                                                                                
 



How much do you feel the use of objects (things) from every day life contributed towards any feeling of immersion?



LITTLE CONTRIBUTION 1 2 3 4 5 MUCH CONTRIBUTION



Did you experience any problems or discomfort with the physical equipment?



Headgear  
                                                                                                                         
   



Stick II                                                                                                                              
   
   



Kennel                                                                                                                            
   
   

What was the purpose of each of the following objects?



Hand                                                                                                                                            
   
   



File-like Menu                                                                                                                        
   
   



Bookcase                                                                                                                                 
   
   



Picture Frame                                                                                                                        
   
   



The Room                                                                                                                                
   
   

How long did it take you to figure each of these purposes out? (roughly, in minutes)



Hand          
File-like Menu  
Bookcase  
Picture Frame  
The Room  



What specific problems did you experience while using the different objects?



Hand          
File-like Menu  
Bookcase  



Do you have any suggestions for better ways that the objects could be represented?



Hand          
File-like Menu  
Bookcase  
Picture Frame  
Room  



What do you think would be a good way to provide help for the system?



                                                                                                                                                
 



From your experience of the image viewer, what other ways could you suggest for viewing images in a 3D system?



                                                                                                                                                



Did you find it strange to interact with a computer imitating a 3D perspective, as opposed to the usual 2D

desktop?



QUITE STRANGE 1 2 3 4 5 EASY TRANSITION



How transparent was your interaction with the computer?

ie Did the absence of a visible computer and monitor distance you from the computer, or did the closeness of the headset display serve as a constant reminder that your interaction was with a computer?



NORMAL SETUP = MORE TRANSPARENT 1 2 3 4 5 VR SETUP = MORE TRANSPARENT



What object was floating behind you?



                                                                                                                                                




General Feedback Section


Do you have any other comments about the system?



                                                                                                                                                
 



Do you have any other comments about the testing or questionnaire?



                                                                                                                                                
 



Thanks again for your input !!

Evaluation organiser:

Matthew Mundell

Email: g95m4965@cs.ru.ac.za


3.2 Abstract Image Viewer

VIRTUAL REALITY QUESTIONNAIRE

IMAGEVIEWER II USABILITY EVALUATION

The purpose of this questionnaire is to use the ImageViewer program to test various aspects of virtual reality systems.

Please answer the questions after going through the ImageViewer test run. Where there is a choice mark the block with the option that best suits the question. Don't feel obliged to answer any of them. All information will be treated with complete confidentiality.

Thanks for your assistance, and enjoy the virtual experience.



Age         



Sex   M     F  
     



Please give a short description of your computer skill and experience: (e.g. no experience, Computer Science masters student)



                                                                                                                                                





Have you been exposed to any virtual reality (VR) systems before?   Y     N  
     



If so, please give a brief description:



                                                                                                                                                
 



Feedback Section

How much did you enjoy using the image viewer?



YAWN... NOT MUCH FUN 1 2 3 4 5 COOLEST THING I'VE DONE ALL WEEK





What was the letter displayed in one of the images from the collection?       


\begin{spacing}{1.00}
\vspace{0.375cm}
{\raggedright\begin{tabular}{c\vert c\ver...
... letter? }&
~~~~~~\\
\cline{2-2}
\end{tabular}\par }
\medskip {}
\end{spacing}

Do you like the way the scene is presented to you when you first start the program?



DISLIKED 1 2 3 4 5 LIKED



Could you suggest a better way to introduce someone to the environment?



                                                                                                                                                



How easily did you forget that your environment was not a real one? (i.e. how easy was it to feel like you were immersed in the environment?)



NOT VERY EASILY 1 2 3 4 5 VERY EASILY



What could have been done to improve the feeling of immersion?



                                                                                                                                                
 



Would you have preferred to have used objects that mimic the things we use in everyday life?



PRESENT OBJECTS 1 2 3 4 5 EVERYDAY OBJECTS



Did you experience any problems or discomfort with the physical equipment?



Headgear                                                                                                                        
   
   



Stick II                                                                                                                            
   
   



Kennel                                                                                                                            
   
   

What was the purpose of each of the following objects?



Hand                                                                                                                                        
   
   



Red Board                                                                                                                        
   
   



Ring Tube                                                                                                                                 
   
   



Picture Frame                                                                                                                        
   
   

How long did it take you to figure each of these purposes out? (roughly, in minutes)



Hand          
Red Board  
Ring Tube  
Picture Frame  



What specific problems did you experience while using the different objects?



Hand          
Red Board  
Ring Tube  



Do you have any suggestions for better ways that the objects could be represented?



Hand          
Red Board  
Ring Tube  
Picture Frame  
The Room  



What do you think would be a good way to provide help for the system?



                                                                                                                                                
 



From your experience of the image viewer, what other ways could you suggest for viewing images in a 3D system?



                                                                                                                                                



Did you find it strange to interact with a computer imitating a 3D perspective, as opposed to the usual 2D

desktop?



QUITE STRANGE 1 2 3 4 5 EASY TRANSITION



How transparent was your interaction with the computer?

ie Did the absence of a visible computer and monitor distance you from the computer, or did the closeness of the headset display serve as a constant reminder that your interaction was with a computer?



NORMAL SETUP = MORE TRANSPARENT 1 2 3 4 5 VR SETUP = MORE TRANSPARENT



Was there an object floating behind you?



                                                                                                                                                




General Feedback Section


Do you have any other comments about the system?



                                                                                                                                                
 



Do you have any other comments about the testing or questionnaire?



                                                                                                                                                
 



Thanks again for your input !!

Evaluation organiser:

Matthew Mundell

Email: g95m4965@cs.ru.ac.za

4. Data Collected

4.1 Keys



Heading Meaning
Cexp Computer Experience
VRexp VR Experience
Grab, Pnt, Prss Times for grab, point 'n press actions
Pos Time taken positioning object
Tot Total time taken
Init Satisfaction with initial environment
Imm Level of immersion experienced
RWcont Contribution of realworld objects to immersion
Mimic Prefer mimicking of realworld objects
2vs3 2D/3D has stranger interaction?
Trn 2D/3D more distant from computer?
Saw Noticed the rocket in the background



Table D1: Key to the headings used in the performance data (see questionnaires for more detail)



Value Computer Experience
-3 Not Given
-2 None
-1 Little
0 Literate
1 Equiv of CS1
2 Equiv of CS2
3 Equiv of CS3
4 Honours
5 Masters
6 PreProf (Masters complete)
7 Prof



Table D2: Key to Computer Experience (Cexp)



Value Virtual Reality Experience
0 None
1 Exposure to 1 partial VR system (eg imageviewer)
2 Exposure to more than 1 partial VR system
3 Exposure to a full VR system (eg CAVE)
4 VR Development



Table D3: Key to Virtual Reality Experience (VRexp)

4.2 Realworld Viewer Data



# Age Sex Cexp VRexp Grab Pnt Prss Pos Tot Init Imm RWCont 2vs3 Trn Saw
1 22 M 4 2 0 30 60 90 180 3 4 5 3 4 Y
2 23 M 4 4 0 30 60 90 180 4 4 3 4 2 N
3 22 M 4 1 30 30 60 120 240 4 4 5 5 5 N
4 23 M 5 4 0 30 60 90 180 4 4 5 2 4 N
5 22 M 4 0 10 90 139 0 239 3 4 4 4 2 N
6 21 M 3 0 41 79 90 0 210 3 3 3 4 3 N
7 20 M 5 2 0 30 30 60 120 3 3 5 4 5 N
8 20 M -3 0 15 165 60 0 240 5 4 4 5 4 N
9 20 M 2 0 0 60 60 0 120 4 5 4 5 1 N
10 20 M 1 0 60 50 130 0 240 4 4 4 5 1 N
11 19 M 1 0 60 120 60 0 240 4 4 1 2 4 N
12 18 M -2 0 3 60 120 60 243 5 5 5 3 5 N
13 17 M -1 0 300 240 60 0 600 3 2 3 4 3 N
14 20 M 1 0 20 140 180 0 340 3 5 3 5 5 N
15 19 M 2 1 240 60 240 300 840 4 5 5 5 5 N



Table D4: Realworld Viewer Data

4.3 Abstract Viewer Data



# Age Sex Cexp VRexp Grab Pnt Prss Pos Tot Init Imm Mimic 2vs3 Trn Saw
1 21 M 3 1 97 22 106 104 329 4 4 4 2 3 ?
3 24 M 5 4 113 55 30 54 252 3 5 4 5 5 ?
4 24 M 5 1 77 83 178 0 338 4 3 4 4 4 N
5 24 M 5 1 152 511 125 0 788 3 4 3 3 4 ?
6 21 F 3 0 150 132 91 0 373 5 2 2 5 4 N
7 21 F 3 0 244 168 62 0 474 4 3 1 1 2 N
8 22 M 1 1 170 76 8 79 333 2 2 4 2 3 N
9 22 M 2 0 327 23 111 0 461 4 3 1 4 4 ?
10 21 M 3 0 122 97 45 0 264 4 2 4 2 4 ?
11 21 M 0 0 106 50 63 11 334 5 3 3 3 4 N
12 21 M 4 4 30 60 3 0 93 4 4 3 4 4 N
13 25 M 4 4 30 118 39 30 217 4 2 2 5 4 ?
14 20 F 0 0 195 418 163 0 776 4 4 4 2 4 N
15 24 F 4 0 250 165 361 0 776 1 5 5 2 5 N
16 22 M 3 0 337 354 205 0 896 4 4 5 3 4 N



Table D5: Abstract Viewer Data

5. Realworld Result Capture

Fifteen people with VR experience levels ranging from very little experience to CoRgi developers helped with the testing of the image viewer. Some new ideas and a lot of problems and room for improvement for the program came out the questionnaire. These are summarised in Section 1 below. Section 2 gives numeric results recorded during testing. This is followed by Section 3 - issues which became apparent from observation of the testers during the test.

Program bugs can really make gathering results difficult, but at the same time they help to uncover issues you normally wouldn't notice. Section 4 deals with how this affected the testing.

5.1 Tester Feedback

A number in brackets alongside a comment shows agreement amongst that number of testers.

5.1.1 Objects

Problems with the objects:

Suggestions:

Problems with movement of the objects:

5.1.2 Help

Problems with the help provided during the test:

Suggestions for providing help for the system:

5.1.3 Improvements

Suggestions for improvements to the program:

Suggestions for better ways of doing things:

Suggestions for new ways of doing things

5.1.4 Equipment

Some of the testers experienced problems with the equipment:

Head mounted display (HMD)

The Stick II (a five-button finger controller)

The Kennel (a darkroom)

Suggestions for improvement

5.2 Tester Observation Feedback

This is the feedback gleaned from observation of the testers.

One of the testers was quite nervous and asked how you do things with the hand straight away, and once he had grabbed the book he wanted to know how to open it. Perhaps a new VR user would be uncertain and unkeen to experiment, like this tester was.

Another interesting situation was when the tester assumed that the hand couldn't pass through the book, so he spent a while trying to get to the pictures after the book opened very near to his perspective.

Here are the summaries of other observations that were made:

5.2.1 Testers wanted

5.2.2 Testers tended to

5.2.3 Testers tried

5.2.4 Testers expected

5.2.5 Looking around

Some of the testers tried to move their heads forward to get a closer look at the objects and images. Very few of the users (2, I think) looked around to see if the picture was anywhere in the room.

5.3 Problems with the Program

Collision Detection

Really sensitive collision detection meant that a collision only occurred when two specific parts of two objects collided. This really makes it difficult for the tester, who'll assume that the gesture or action has no effect and try something else.

With good collision detection it should be really easy to select an object from the book or grab the book. Also the book will be able to be placed anywhere on (or around) the bookcase's shelves.

Book Selecting and Positioning

The difficult positioning of the book caused hassles for many of the testers and made timing of the action discovery harder. The program needs to be fixed to position the open book according to the position that the closed book was held at when released. Also the position of the closed book in the hand when grabbed must be tidied by setting it explicitly.
Drop Bug

A few times the book was dropped but it didn't open. Mostly the tester just picked it up and carried on normally but twice the program had to be restarted - I'm not sure what's causing this.
Broken Thumb

The thumb movement in the wrong direction on the hand was mentioned by many of the tester, and one guy found it quite irritating.

6. Abstract Result Capture

These are the full feedback and observation recordings for the abstract image viewer. As in the realworld results above a number in brackets alongside a point indicates agreement among that number of testers.

6.1 Tester Feedback

Summarisation of the tester feedback is divided into several sections, which follow.

6.1.1 General

An interesting comment from a tester with no VR experience was that she expected to feel like she was inside computer, interacting directly with the files and disks.

Problems experienced with the test:

One tester thought that depth perception was made difficult by having the hand able to move through objects, and that other types of feedback are necessary to give depth clues. It was also commented that 3D visualization and the judging of distance (2) was difficult. This may all have stemmed from the lack of stereoscopic display. (shifted but not rotated)

6.1.2 Introduction to the system

Suggestions for improvement:

6.1.3 Immersion

6.1.4 Objects

Problems with

Suggestions for improvements

Problems with object movement and manipulation

Interesting perceptions of the objects:

Issues surrounding the hand:

6.1.5 Help

Some of the users' immediate reaction was to ask questions about what they should and how, while others were happy to explore without being told to do so.

Possible improvement to the help during testing:

Suggestions for ways of providing help in the viewer:

6.1.6 Improvements

Suggestions for ways to improve the program or system:

Suggestions for better ways of doing things:

Suggestions for new ways of doing things:

6.1.7 Equipment

Feedback about the equipment:

Head mounted display (HMD)

The Stick II (a five-button finger controller)

The Kennel (a darkroom)

The Polhemus Tracker

Suggestions for improvement

6.2 Tester Observation

Here follows summary of tester observation during the testing process.

One tester looked around a bit, while 2 others took a good look around the room.

A few tester were lost at first, while others went straight to the cylinder of rings. Many of the testers thought the frame itself was important, and started there. It was almost out of reach, which caused problems for some.

Often it seems like the tester ignores approaching the situation logically, and rather expects the system to be illogical.

Every tester works at his own pace, sometime the guy who takes it really slow finishes faster than the guy who rushes things.

The hand passing through the object really confused one of the testers, who couldn't seem to come terms with this.

One tester was unsure of what to do with the grabbed ring and had to be prompted, but the others pretty quickly assumed that they should try drop it somewhere.

Almost always the board was pointed at in an series of attempts at getting something to happen, and this would reveal the picture of the D to tester. A few times the tester would not notice the change for a while (perhaps because of the angle of the board, or because the point is effective at a distance), and continue with the attempts to get something to happen.

6.2.1 Testers experienced

6.2.2 Testers wanted

6.2.3 Testers tended to

6.2.4 Testers tried

6.2.5 Testers expected

6.2.6 Testers asked

6.2.7 The observer noticed that