Systems Engineering for Ethical Autonomous Systems

Cover
Contents
List of figures
List of tables
Foreword
References
List of acronyms
1 The art of the acceptable, not the art of the possible
1.1 Introduction
1.2 Technologies and their acceptance
1.3 Machines that think
1.4 What is autonomy?
1.5 Maintaining control
1.6 Responsibilities
1.7 Autonomous Weapon Systems (AWS)
1.8 Concept to product
1.9 Future regulations
1.10 Principles applicable to non-military systems
References
2 Decision-making
2.1 Freedom of action
2.2 Skills, behaviours and automation
2.3 Situational awareness
2.4 Human workload
2.5 Decision-making
2.6 Multiple humans and autonomous systems
2.7 The Observe Orient, Decide and Act (OODA) loop
2.8 Authority to act
2.8.1 Authority and responsibility
2.8.2 Military Command and Control (C2)
References
3 Automated control and autonomy
3.1 Introduction
3.2 Automatic or autonomy – does the choice of word matter?
3.3 Definitions of autonomy and automatic
3.4 Automated and Autonomous Weapon Systems (AWS)
3.5 Autonomy levels
3.5.1 The need for autonomy levels
3.5.2 Autonomy levels for non-military systems
3.5.3 Autonomy levels for military systems
3.6 Autonomy, trust and work-sharing
3.7 Control system developments
3.7.1 Intelligently designed mechanisms
3.7.2 Intelligently designed control systems
3.7.3 Intelligent control systems
3.8 Models and control systems
3.8.1 Models of the process under control
3.8.2 Models of the control system
3.9 Control and the targeting process
Appendix A3 Definitions of autonomy and autonomy levels
A3.1 Autonomy-related definitions from published civilian standards
A3.2 Military definitions of autonomy
A3.2.1 NATO discussion of terms
A3.2.2 UK Joint Doctrine Note JDN 3/10
A3.2.3 US DOD Directive of 2012
A3.3 Non-military definitions of autonomy levels
A3.3.1 Barber and Martin [31]
A3.3.2 Parasuraman et al. [32]
A3.3.3 Space industry
A3.3.4 Road vehicle industry
A3.3.5 Maritime industry
A3.4 Military definitions of autonomy level
A3.4.1 NATO-suggested definitions
A3.4.2 The US Department of Defense levels
A3.4.3 Autonomy Levels For Unmanned Systems (ALFUS)
References
4
Operational analysis to systems engineering
4.1 Introduction
4.2 Terms and tools
4.2.1 Systems of systems
4.2.2 Emergent behaviours and wicked problems
4.2.3 Architectures
4.2.4 Architecture frameworks
4.2.5 UML and SysML
4.3 Contradictions in technology developments for military use
4.4 Defining and delivering military requirements
4.5 Capability-based planning
4.6 The capability-based Vee diagram
4.7 Establishing solutions by operational analysis
4.8 Operational analysis for autonomous systems
4.9 New types of engineering for military systems
4.10 Capability engineering
4.11 Wider capabilities
4.12 Systems engineering
4.12.1 Overview
4.12.2 Inputs to the systems engineering process
4.12.3 The systems engineering process
4.12.3.1 Post-contract reviews
4.12.3.2 First iteration of solution
4.12.3.3 Final system design
4.12.4 Spiral development
4.13 Validation of the system design
4.14 Post-contract award changes
4.14.1 Pre-delivery changes
4.14.2 Setting-to-work changes for military systems
Appendix A4 Through-life-capability-management terminology
References
5 Engineering design process
5.1 Introduction and overview
5.2 Management control
5.2.1 General principles
5.2.2 Project management
5.2.3 Technical management
5.2.4 Technology Readiness Levels (TRLs)
5.2.5 Initial critical review
5.2.5.1 Customer need
5.2.5.2 Delivery criteria
5.2.5.3 Design requirements
5.2.5.4 Project timescales
5.2.5.5 Project costs
5.2.6 Make-or-buy decisions
5.2.7 Risk registers
5.3 Project organisation
5.3.1 Work breakdown structure
5.3.2 Project monitoring
5.4 Autonomous (sub)system design
5.4.1 General principles for weapon systems
5.4.2 The CADMID cycle
5.4.3 Design validation and verification (V&V)
5.4.3.1 The reasons for V&V
5.4.4 V&V of autonomous systems
5.4.4.1 Validation of an autonomous system
5.4.4.2 Verification of an autonomous system
5.4.5 Integration and test
5.5 Module design
5.5.1 Setting the requirements
5.5.2 Detailed design
5.5.3 Fitted for, but not with
Appendix A5 Technology, integration and system readiness levels from different sources
References
6 Ethics, civil law and engineering
6.1 Introduction
6.2 Ethical background
6.3 Regulations, standards and certification
6.3.1 Regulations
6.3.2 Standards
6.3.3 Drones – an illustration of the extremes of regulatory problems
6.3.4 Certification
6.4 Current standards for autonomous systems
6.4.1 ISO 8373-2012 robots and robotic devices – vocabulary
6.4.2 IEEE 1872–2015, IEEE standard ontologies for robotics and automation
6.4.3 BS8611: 2016, robots and robotic devices. Guide to the ethical design and application of robots and robotic systems
6.5 Future regulations and standards for autonomous systems
6.6 Safety
6.7 Liabilities
6.8 Risks, benefits and ALARP
6.9 Safety cases for autonomous systems
References
7 Introduction to military legal context and its relevance to engineering
7.1 Introduction
7.2 The need for laws of war
7.3 Legality and legitimacy
7.4 International Humanitarian Law
7.5 The Geneva Conventions, their protocols and subsequent agreements
7.6 Customary principles and rules
7.7 Judicial decisions
7.8 Expert opinions
7.9 Military manuals
7.10 Engineering requirements from selected conventions in Section 7.5
7.10.1 General
7.10.2 1949 Geneva Convention 1. For the amelioration of the condition of the wounded and sick in armed forces in the field
7.10.3 1949 Geneva Convention II. For the amelioration of the condition of wounded, sick and shipwrecked members of armed forces at sea
7.10.4 1949 Geneva Convention III. Relative to the treatment of prisoners of war
7.10.5 1949 Geneva Convention IV. Relative to the protection of civilian persons in time of war
7.10.6 1976 Convention and 1994 Guidance on environmental modification techniques
7.10.7 1980 UN convention on prohibitions or restrictions on the use of certain conventional weapons which may be deemed to be excessively injurious or to have indiscriminate effects
7.10.8 2008 Convention on cluster munitions
7.11 The additional protocols to the 1949 Geneva conventions
7.11.1 Introduction
7.11.2 General requirements from API and APII
7.11.3 More specific requirements from API and APII
7.12 The law at sea
7.13 Rules-of-engagement
7.14 An example – design changes to move Phalanx from sea to land
Appendix A7 Extracts from Additional Protocol I (API)
References
8 Targeting
8.1 Introduction to targeting
8.2 Types of weapon used in attack
8.2.1 Classes of attack
8.2.2 Ballistic projectiles
8.2.3 Externally guided projectiles
8.2.4 Self-guided projectiles
8.3 Targeting law
8.4 Targeting processes and cycles
8.4.1 A range of targeting processes
8.4.2 The F2T2EA phases
8.4.2.1 Find
8.4.2.2 Fix
8.4.2.3 Track
8.4.2.4 Target
8.4.2.5 Engagement
8.4.2.6 Assessment
8.5 Automating targeting processes
8.5.1 General considerations
8.5.2 Assumptions
8.5.3 Observe
8.5.3.1 Electro-optical systems
8.5.3.2 Radar and sonar sensors
8.5.3.3 Other sensor types
8.5.4 Orient
8.5.5 Decide
8.5.6 Act
8.6 Issues for autonomous targeting and Article 36 reviews
8.6.1 The general problem
8.6.2 A projectile as an Autonomous Weapon System (AWS)
8.6.3 Capability restrictions on the AWS
8.6.4 Article 36 reviews of projectile AWSs
References
9 Influences on future military autonomous systems
9.1 Introduction
9.2 Recent and current campaigns
9.3 The military operating environment
9.3.1 The three Ds
9.3.2 The five Cs
9.3.2.1 Congested
9.3.2.2 Cluttered
9.3.2.3 Contested
9.3.2.4 Connected
9.3.2.5 Constrained
9.4 Societal changes
9.4.1 Political changes
9.4.2 Economic changes
9.5 Technology changes
9.5.1 Connectivity
9.5.2 Artificial Intelligence
9.5.3 Exploiting commercial technologies
9.5.4 Commercial autonomous systems
9.5.5 Decision aids or decision-makers?
9.6 Urgent requirements for operations
9.7 Countering autonomous systems
9.7.1 Physical attack
9.7.2 Cyber-attack
9.8 Non-kinetic AWS
9.9 Future operational analysis
9.9.1 Problems for traditional operational analysis
9.9.2 Operational analysis and system concepts with COTS modules
9.9.2.1 Questions about algorithms
9.9.2.2 Questions about the software
9.10 Future changes in International Humanitarian Law (IHL)
9.10.1 Introduction
9.10.2 The Martens Clause
9.10.3 Potential new treaties
References
10 Systems engineering applied to International Humanitarian Law (IHL)
10.1 Introduction
10.2 Context
10.2.1 Bespoke solutions
10.2.2 Automate each function
10.3 Requirements derived from the four IHL principles
10.3.1 General principles
10.3.2 Military necessity
10.3.3 Humanity
10.3.4 Proportionality
10.3.5 Distinction
10.4 Developing the architecture framework
10.5 Generic weapon system architecture
10.5.1 Architecture requirements
10.5.2 The 4D/RCS reference model (NSTIR 6910)
10.5.3 Approach and definitions
10.6 Architecture requirements
10.6.1 Roles and functions during each phase
10.6.2 Surveillance
10.6.3 Monitor localised scene
10.6.4 Confirm authority to act
10.6.5 Release and guide weapon
10.6.6 Knowledge database contents
10.7 Cognitive function requirements
10.8 Issues in moving from automation to autonomy
10.8.1 Approach
10.8.2 Automating Observe
10.8.2.1 Target and non-target types
10.8.2.2 Authorised power for Observe phase
10.8.3 Automating Orient
10.8.3.1 Automating Monitor Localised Area Role
10.8.3.2 Predict scene at impact time function
10.8.3.3 Authorised power for Orient phase
10.8.4 Automating Decide and Act
10.9 Authorised power for a Level 1 node
10.10 Learning systems
References
11 Systems engineering for a new military system
11.1 Introduction
11.2 Analysis of the problem using Chapter 4
11.2.1 The state's problem
11.2.2 Required capability
11.2.3 Initial operational analysis
11.2.3.1 Current Blue assets
11.2.3.2 Initial analysis results
11.2.4 Refining the operational analysis
11.2.5 Unmanned air system capabilities
11.2.6 Architectural analysis
11.2.7 System-of-systems effectiveness
11.2.8 Pre-competition activities by the procurement agency
11.2.8.1 Contract review
11.2.8.2 Analysis
11.2.8.3 Technical advice
11.3 The system-of-systems delivering the capability
11.3.1 The C2 infrastructure
11.3.2 Changes to UAV ground stations
11.3.3 Changes to UAVs
11.3.4 Development of tactics
11.4 Delivering the capability
11.4.1 The procurement problem
11.4.2 System-of-systems requirements
11.4.3 The procurement process
11.4.4 Integration plans
11.4.5 Work packages and contract award
11.4.6 Work Packages 1 and 2, integration into C2 structure
11.4.7 Radar work packages
11.4.8 WP3 (Part), upgrade surveillance UAV autopilot
11.5 Legal review and guidance
11.5.1 Is a review needed?
11.5.2 Testing assumptions
11.5.3 The Article 36 Review of the initial capability
References
12 Making military capabilities autonomous
12.1 Approach
12.2 Introducing autonomy into systems
12.3 Article 36 review questions for the system of systems
12.4 An example capability with upgrades for autonomous operation
12.4.1 Top-down approach
12.4.2 Functionality in the top-down approach
12.4.3 Incremental approach to autonomy
12.5 Configuration and control during the OODA process
12.5.1 Configurations during OODA
12.5.2 Command and control (C2)
12.6 Technical evidence required for top-down approach
12.6.1 Sensory processing
12.6.1.1 Defensive Aids Suite (DAS)
12.6.1.2 Non-DAS sensors during the Observe phase
12.6.1.3 Non-DAS sensors during Orient, Decide and Act phases
12.6.2 Value judgement, world modelling and behaviour generator
12.6.3 Knowledge database
12.7 Article 36 reviews for incremental upgrades
12.7.1 A possible architecture
12.7.2 Sensory processing
12.7.3 World modelling
12.7.4 Value judgement
12.7.5 Behaviour generator
12.7.6 Knowledge database
12.8 Wider implications – limits to autonomy
References
13 Design of civilian autonomous systems using military methodologies
13.1 Introduction
13.2 Autonomous systems and design methodologies
13.2.1 Design methodologies
13.2.2 Technology push or demand pull?
13.3 General principles for autonomous systems
13.3.1 Ethical principles in design
13.3.2 Value-based design
13.3.3 United Nations Declaration of Human Rights
13.3.4 The five 'EPSRC' rules3
13.3.5 Principles from assertions
13.3.6 Asimov's laws of robotics
13.4 A physical system that works with humans
13.4.1 Assumptions
13.4.2 How an autonomous system operates in a complex environment
13.4.3 Legal issues for the autonomous system
13.4.4 Insurance and litigation issues for the autonomous system
13.4.5 Functional requirements and architecture
13.5 Completing the task
13.5.1 Identifying the legal framework
13.5.2 The autonomous system is given an instruction
13.5.3 The autonomous system plans how to complete the task
13.5.4 The autonomous system completes its tasks
13.5.5 Post-goal actions
13.6 Functions for complex environments
13.6.1 Need for identifying legal functions
13.6.2 The necessary functions for legal operation
13.7 System architecture
13.7.1 Node architecture
13.7.1.1 Knowledge database
13.7.1.2 Sensory processing
13.7.1.3 World modelling
13.7.1.4 Value judgement
13.7.1.5 Behaviour generation
13.7.2 Task management activities
13.8 Requirements for functional elements in a generic autonomous system
13.8.1 Overarching requirements
13.8.2 Operator interface requirements
13.8.3 Knowledge database requirements
13.8.4 Sensor and sensory-processing requirements
13.8.5 World model requirements
13.8.6 Value judgement requirements
13.8.7 Behaviour generator requirements
13.8.8 Legal criteria
13.8.9 Test, Validation and Verification (V&V)
13.9 Transferable methods
References
14 Final considerations for ethical autonomous systems
14.1 Timescales for acceptability of autonomous systems
14.2 Identifying ethical guidelines
14.3 Authority and responsibility
14.4 Cyber threats
14.5 Misuse of autonomous system
14.6 Future regulations
14.7 Intelligent swarms
14.8 Human–machine interface
14.9 How issues in this book may help solve future problems
References
Appendix 1 Red Cross Guide to Article 36 Reviews
Executive summary
Introduction
Structure
1 Material scope of application of the review mechanism
1.1 Types of weapons to be subjected to legal review
1.2 Legal framework: rules to be applied to new weapons, means and methods of warfare
1.2.1 Prohibitions or restrictions on specific weapons
1.2.1.1 Prohibitions or restrictions on specific weapons underinternational treaty law
1.2.1.2 Prohibitions or restrictions on specific weapons under customary international law
1.2.2 General prohibitions or restrictions on weapons, means and methods of warfare
1.2.2.1 General prohibitions or restrictions on weapons, means and methods of warfare under international treaty law
1.2.2.2 General prohibitions or restrictions on weapons, means and methods of warfare under customary international law
1.2.2.3 Prohibitions or restrictions based on the principles of humanity and the dictates of public conscience (the ‘Martens clause’)
1.3 Empirical data to be considered by the review
1.3.1 Technical description of the weapon
1.3.2 Technical performance of the weapon
1.3.3 Health-related considerations
1.3.4 Environment-related considerations
2 Functional aspects of the review mechanism
2.1 How should the review mechanism be established?
2.1.1 By legislation, regulation, administrative order, instruction or guidelines?
2.1.2 Under which authority should the review mechanism be established?
2.2 Structure and composition of the review mechanism
2.2.1 Who should be responsible for carrying out the review?
2.2.2 What departments or sectors should be involved in the review? What kinds of experts should participate in the review?
2.3 Review process
2.3.1 At what stage should the review of the new weapon take place?
2.3.2 How and by whom is the legal review mechanism triggered?
2.3.3 How does the review mechanism obtain information on the weapon in question, and from what sources?
2.4 Decision-making
2.4.1 How does the review mechanism reach decisions?
2.4.2 Should the reviewing authority's decision be binding or should it be treated only as a recommendation?
2.4.3 May the reviewing authority attach conditions to its approval of a new weapon?
2.4.4 Should the reviewing authority's decision be final or should it be subject to appeal or review?
2.5 Record-keeping
2.5.1 Should records be kept of the decisions of the review mechanism?
2.5.2 To whom and under what conditions should these records be accessible?
CONTACTS
ANNEX I
ANNEX II
ANNEX III
Index
Back Cover