The ongoing migration of computing and information access from stationary environments to mobile computing devices for eventual use in mobile environments, such as Personal Digital Assistants (PDAs), tablet PCs, next generation mobile phones, and in-car driver assistance systems, poses critical challenges for natural human-computer interaction. Spoken dialogue is a key factor in ensuring natural and user-friendly interaction with such devices which are meant not only for computer specialists, but also for everyday users. Speech supports hands-free and eyes-free operation, and becomes a key alternative interaction mode in mobile environments, e.g. in cars where driver distraction by manually operated devices may be a significant problem. On the other hand, the use of mobile devices in public places, may make the possibility of using alternative modalities possibly in combination with speech, such as graphics output and gesture input, preferable due to e.g. privacy issues. Researchers’ interest is progressively turning to the integration of speech with other modalities such as gesture input and graphics output, partly to accommodate more efficient interaction and partly to accommodate different user preferences. This book: combines overview chapters of key areas in spoken multimodal dialogue (systems and components, architectures, and evaluation) with chapters focussed on particular applications or problems in the field. focusses on the influence of the environment when building and evaluating an application. Audience: Computer scientists, engineers, and others who work in the area of spoken multimodal dialogue systems in academia and in the industry. Graduate students and Ph.D. students specialising in spoken multimodal dialogue systems in general, or focusing on issues in these systems in mobile environments in particular
Author(s): W. Minker, Dirk Buhler,Laila Dybkjaer
Edition: 1
Year: 2005
Language: English
Pages: 403
Contents......Page 6
Preface......Page 12
Contributing Authors......Page 14
Introduction......Page 22
Part I Issues in Multimodal Spoken Dialogue Systems and Components......Page 30
1. Introduction......Page 31
2. Varieties of Multimodal Dialogue......Page 32
3. Detecting Intentional User Inputs......Page 34
5. History and Context......Page 35
6. Domain Reasoning......Page 36
8. Dialogue Management......Page 37
9. Conclusion......Page 38
References......Page 39
1. Ubiquitous/Wearable Computing Environment......Page 40
2. State-of-the-Art Speech Recognition Technology......Page 41
3. Ubiquitous Speech Recognition......Page 43
4. Robust Speech Recognition......Page 45
5. Conversational Systems for Information Access......Page 48
6. Systems for Transcribing, Understanding and Summarising Ubiquitous Speech Documents......Page 51
7. Conclusion......Page 59
References......Page 60
3 A Robust Multimodal Speech Recognition Method using Optical Flow Analysis......Page 64
1. Introduction......Page 65
2. Optical Flow Analysis......Page 66
3. A Multimodal Speech Recognition System......Page 67
4. Experiments for Noise-Added Data......Page 70
5. Experiments for Real-World Data......Page 75
6. Conclusion and Future Work......Page 76
References......Page 79
1. Introduction......Page 81
2. Basic Dialogue Framework......Page 82
3. Feature Functions......Page 85
4. Computing Dialogue Costs......Page 89
5. Selection of Dialogue State/Action Pairs......Page 90
6. XML-based Data Structures......Page 91
7. Usability in Mobile Environments......Page 94
8. Results......Page 95
References......Page 100
1. Introduction......Page 103
2. State-of-the-Art in Problem Solving Dialogues......Page 106
3. Reasoning Architecture......Page 107
4. Application to Calendar Planning......Page 111
5. Conclusion......Page 114
References......Page 116
1. Introduction......Page 118
2. Specification......Page 122
3. Interpretation......Page 128
4. Realisation in an Animated Talking Head......Page 130
5. Discussion and Future Work......Page 134
References......Page 136
Part II System Architecture and Example Implementations......Page 139
1. Introduction......Page 140
2. Towards Personal Multimodal Conversational User Interface......Page 141
3. System Architectures for Multimodal Dialogue Systems......Page 145
4. Standardisation of Application Representation......Page 149
5. Conclusion......Page 152
References......Page 153
1. Introduction......Page 156
2. XISL Execution System......Page 157
3. Extensible Interaction Scenario Language......Page 159
4. Three Types of Front-Ends and XISL Descriptions......Page 163
5. XISL and Other Languages......Page 169
6. Discussion......Page 170
References......Page 171
1. Introduction......Page 172
2. Application Considerations, Technologies and Mobile Terminals......Page 173
3. Projects and Commercial Developments......Page 177
4. Three Multimodal Demonstrators......Page 179
5. Roadmap for Successful Versatile Interfaces in Telecommunications......Page 184
6. Conclusion......Page 186
References......Page 187
1. Introduction......Page 191
2. Why Multimodal Wireless?......Page 193
3. Walking Direction Application......Page 194
4. Speech Technology for Multimodal Wireless......Page 195
5. User Interface Issues......Page 196
6. Multimodal Architecture Issues......Page 201
7. Conclusion......Page 204
References......Page 206
1. Introduction......Page 207
2. Related Work......Page 208
3. SmartKom - Intuitive Human-Machine Interaction......Page 211
4. Scenarios for Mobile Use......Page 213
5. Demonstrator Architecture......Page 215
6. Dialogue Design......Page 216
7. Outlook - Towards Flexible Modality Control......Page 219
8. Conclusion......Page 221
References......Page 222
1. Introduction......Page 225
2. LARRI - System Description......Page 226
3. LARRI - Hardware and Software Architecture......Page 230
4. Experiments and Results......Page 235
5. Conclusion......Page 237
References......Page 239
Part III Evaluation and Usability......Page 241
1. Introduction......Page 242
2. State-of-the-Art......Page 244
3. Empirical Generalisations......Page 248
4. Frameworks......Page 255
5. Multimodal SDSs Usability, Generalisations and Theory......Page 257
6. Discussion and Outlook......Page 259
References......Page 262
1. Introduction......Page 268
2. Wizard-of-Oz Experiment......Page 272
3. Overhearer Experiment......Page 283
4. Discussion......Page 287
References......Page 288
1. Introduction......Page 290
2. The VICO System......Page 292
3. VICO Haptics - How and When to Make VICO Listen?......Page 293
4. VICO Graphics - When might the Driver Look?......Page 295
5. Who is Driving this Time?......Page 299
6. Modelling the Driver......Page 301
7. Conclusion and Future Work......Page 305
References......Page 306
16 Design, Implementation and Evaluation of the SENECA Spoken Language Dialogue System......Page 307
1. Introduction......Page 308
2. The SENECA SLDS......Page 310
3. Evaluation of the SENECA SLDS Demonstrator......Page 321
4. Conclusion......Page 328
References......Page 329
1. Introduction......Page 331
3. Techniques......Page 335
4. Evaluation......Page 342
5. Conclusion......Page 346
References......Page 347
1. Introduction......Page 349
2. The Matis System......Page 352
3. Methods......Page 355
4. Results and Discussion......Page 357
5. Conclusion......Page 365
References......Page 366
19 User Multitasking with Mobile Multimodal Systems......Page 369
1. The Challenge of Multitasking......Page 370
3. Analyses of Single Tasks......Page 374
4. Analyses of Task Combinations......Page 379
5. Studies with Users......Page 384
6. The Central Issues Revisited......Page 391
References......Page 395
1. Introduction to Conversational Interfaces......Page 398
2. Research Goals......Page 401
3. Method......Page 402
4. Results......Page 406
5. Discussion......Page 410
6. Conclusion......Page 412
References......Page 413
C......Page 417
I......Page 418
N......Page 419
W......Page 420
X......Page 421
