Development and deployment of Multiplayer Online Games Vol I

CONTENTS
Chapter 1. Game-Design Document from an MOG perspective 25
Your Game As Your Baby ........................................................................................................................................26
3500-Word Crash Course for First-Time Game Developers .......................................................27
On the GDD .......................................................................................................................................................................................................................................27
Subject to Change, Seven Days a Week ......................................................................................................................................................28
Being Agile and Writing It Down ....................................................................................................................................................................29
The Overly Generic Fallacy ........................................................................................................................................................................................29
On Project Stakeholders .....................................................................................................................................................................................................31
On Focus Testing and Playtesting.......................................................................................................................................................................32
On Marketing and Monetization: Critical Mass .....................................................................................................................................33
On Stakeholder Availability .......................................................................................................................................................................................34
TL;DR on Project Stakeholders ...............................................................................................................................................................................34
On a Typical Non-MOG Team Structure ............................................................................................................................................................35
Time-to-Market, MVP, and Planning .......................................................................................................................................................................35
Dealing with Time-to-Market ...................................................................................................................................................................................36
On the Importance of Holding Your Horses .................................................................................................................................................37
TL;DR On a Crash Course for First-Time Developers ............................................................................................................................38
Three All-Important GDD Rules ........................................................................................................................39
On Separating GDD and Implementation Details ..................................................................................................................................40
Dealing with “Difficult” Stakeholders and (Jobs Forbid) Managers .......................................................................................41
Limited-Lifespan vs. Undefined-Lifespan Games ................................................................................42
Client-Driven vs. Server-Driven Development Workflow .............................................................44
Server-Driven Workflow .......................................................................................................................................................................................................44
Client-Driven Workflow ........................................................................................................................................................................................................44
Dealing with Client-Driven Workflow..............................................................................................................................................................45
Option 1. “Continuous Conversion”...............................................................................................................................................................46
Option 2. “Integrate Server into Toolchain” .........................................................................................................................................47
On Matchmaking and the Social Aspect of Your MOG .................................................................47
Matchmaking That Doesn’t Work (As a Rule of Thumb) ..................................................................................................................48
Matchmaking That Works ..................................................................................................................................................................................................49
On Support for Smaller Groups of Players ..............................................................................................52
Technical Issues Affecting Marketing and Monetization .............................................................54
Your GDD Requirements List ...............................................................................................................................59
viii • Development and Deployment of Multiplayer Online Games
On MOG-Specific Teams ..........................................................................................................................................66
Network Team .................................................................................................................................................................................................................................66
Server Team .......................................................................................................................................................................................................................................67
Database Team ..............................................................................................................................................................................................................................68
Back-End Team...............................................................................................................................................................................................................................68
Timeline for the Back-End Team...........................................................................................................................................................................69
All MOG-Specific Teams Must Be First-Class Citizens .........................................................................................................................71
Running Costs Breakdown .....................................................................................................................................73
Common GDD Pitfall: Just Throw In a Multiplayer for Free .......................................................77
Game Entities and Interactions .........................................................................................................................79
Game Entities: What Are You Dealing With?.................................................................................................................................................80
Interactions between Game Entities......................................................................................................................................................................80
What Should You Get? Entities-and-Interactions Diagram ...........................................................................................................80
Examples of Entities and Interactions ..................................................................................................................................................................81
Social Farming and Farming-Like Games ....................................................................................................................................................82
Casino Multiplayer Games ...........................................................................................................................................................................................83
Stock Exchanges, Sports Betting, and Auction Sites ......................................................................................................................84
Large Virtual-World Games (MMOTBS/MMORTS/MMORPG/MMOFPS) ......................................................................86
Team Competitions/eSports .......................................................................................................................................................................................88
Entities and Interactions Diagram As a Starting Point to Architect Your Game....................................................89
Chapter 1 Summary ....................................................................................................................................................90
Bibliography ......................................................................................................................................................................................................................................91
Chapter 2. On Cheating, P2P, and [Non-]Authoritative Servers 93
If You’re Popular Enough, They Will Find Reasons to Cheat .....................................................94
The Big Fat Hairy Difference from E-commerce ..................................................................................96
Dealing with Cheaters ..............................................................................................................................................97
Gameplay .............................................................................................................................................................................................................................................98
Architecture .......................................................................................................................................................................................................................................98
Bot Fighting .......................................................................................................................................................................................................................................99
Attacks: The Really Big Advantage of the Home Turf ....................................................................99
Home Game ......................................................................................................................................................................................................................................99
Road Game .....................................................................................................................................................................................................................................100
Published vs Unpublished Attacks ..............................................................................................................102
Published Attacks: Higher Impact, But Home-Turf Advantage Is Regained ...........................................................102
Unpublished Attacks ...........................................................................................................................................................................................................103
Contents • ix
Attack Types ..................................................................................................................................................................104
Legal Stuff and Problems Banning ......................................................................................................................................................................104
Game Cheats ................................................................................................................................................................................................................................104
Game-Rule Violations ...................................................................................................................................................................................................105
Information Exposure ....................................................................................................................................................................................................105
Reflex Augmentation ....................................................................................................................................................................................................106
Abuses of Disconnect Handling ........................................................................................................................................................................107
Grinding Bots ..........................................................................................................................................................................................................................108
Multiple Accounts .............................................................................................................................................................................................................109
Classical Attacks ......................................................................................................................................................................................................................109
DB Attacks .................................................................................................................................................................................................................................109
Stealing Your Source Code .....................................................................................................................................................................................110
Password Phishing.............................................................................................................................................................................................................111
Keyloggers/Trojans/Backdoors on Another Player’s Device ...............................................................................................112
DDoS ................................................................................................................................................................................................................................................113
MOG Attack Type Summary ........................................................................................................................................................................................114
Authoritative Client: Pretty Much Hopeless Against Cheaters (Except for Console-
Only Games)...................................................................................................................................................................116
Code Signing—Doesn’t Really Work in a Hostile Environment (Except for Consoles) ................................117
Theoretical Protections ....................................................................................................................................................................................................119
Cross-Checks—Undetectable Attacks, Taking Over the World, and Latencies .............................................119
Consensus (Actually, Majority Vote)—
Even More Latencies ................................................................................................................................................................................................121
Trusted Nodes—Who Is the One We Trust? ..........................................................................................................................................121
Homomorphic Encryption—Doesn’t Even Start to Fly ..............................................................................................................122
Authoritative Client MOG Summary ..................................................................................................................................................................123
Deterministic Lockstep: No Game-Rules Violations, But Wide-Open to Information
Leaks ....................................................................................................................................................................................124
Authoritative Server: As Cheater-Proof as They Get ....................................................................126
Authoritative Servers: Scalability Is Imperfect But Workable ................................................................................................128
An Example Calculation .............................................................................................................................................................................................130
Summary: Authoritative Server Is Not Ideal, But Is the Only Thing Workable ....................................................131
Think Positive! Or, Maybe There’s Still Hope... ....................................................................................132
Every Bit Counts: Multi-Layer Protection .......................................................................................................................................................133
Bottom Line for Chapter 2: Yes, It Is Going to Be an Authoritative Server .................135
Bibliography ..................................................................................................................................................................................................................................136
x • Development and Deployment of Multiplayer Online Games
Chapter 3. Communications 139
Client-2-Server and Server-2-Client Communications ................................................................140
RTT, Input Lag, and How to Mitigate Them ..........................................................................................141
Data-Flow Diagram, Take 1 ..........................................................................................................................................................................................141
Input Lag: The Worst Nightmare of an MOG Developer .............................................................................................................144
Input Lag: User Expectations ................................................................................................................................................................................144
Input Lag: How Much We Have Left for MOG ...................................................................................................................................145
Accounting for Packet Losses and Jitter ..................................................................................................................................................147
Internet is Packet-Based, and Packets Can Be Lost ..............................................................................................................147
Cutting Overhead ........................................................................................................................................................................................................148
Client-Side and Server-Side Buffering on Receipt ....................................................................................................................150
Time Synchronization ..............................................................................................................................................................................................152
Sync-Once .................................................................................................................................................................................................................................................. 152
Sync-Once with Subsequent Adjustments ............................................................................................................................................................ 153
NTP-Like Protocol .............................................................................................................................................................................................................................. 154
Phase-Locked Loop (PLL) .......................................................................................................................................................................................................... 154
TL;DR on Time Sync ......................................................................................................................................................................................................................... 156
On TCP .....................................................................................................................................................................................................................................156
Input Lag: Taking a Bit Back ..................................................................................................................................................................................157
Data-Flow Diagram, Take 2: Fast-Paced Game Specifics .............................................................................................................158
RTT ...........................................................................................................................................................................................................................................................160
LAN RTT vs Internet RTT ...........................................................................................................................................................................................162
On CDNs and Geo Server Distribution .......................................................................................................................................................162
RTT and Players ...................................................................................................................................................................................................................163
Back to Input Lag ....................................................................................................................................................................................................................163
Data-Flow Diagram, Take 3: Client-Side Prediction and Interpolation .........................................................................165
Client-Side Interpolation ...........................................................................................................................................................................................165
Client-Side Extrapolation, a.k.a. Dead Reckoning ...........................................................................................................................166
Running into the Wall, and Server Reconciliation ...................................................................................................................166
Client-Side Prediction ...................................................................................................................................................................................................168
Client-Side Prediction: Dealing with Discrepancies ................................................................................................................169
On Distributed Authority .....................................................................................................................................................................................170
Take-3 Diagram ....................................................................................................................................................................................................................171
Lag Compensation—Potential for Cheating vs. Player Happiness...............................................................................174
Server Rewind ..................................................................................................................................................................................................................174
Subtracting Client RTT on the Server-Side ......................................................................................................................................175
Lag Compensation Is Inherently Open to Cheating... ...........................................................................................................175
...OTOH, Player Happiness Is Much More Important ..............................................................................................................176
There Are So Many Options! Which Ones Do I Need? .............................................................................................................178
Contents • xi
Game-World States and Reducing Traffic ..............................................................................................179
Server-Side, Publishable, and Client-Side Game-World States ..............................................................................................180
Limit on Bandwidth .........................................................................................................................................................................................................180
Additional Reasons to Optimize Bandwidth ........................................................................................................................................181
Triangles and Bandwidth ...........................................................................................................................................................................................183
Three Different States of MOG ..........................................................................................................................................................................184
Client-Side State ...........................................................................................................................................................................................................184
Server-Side State ..........................................................................................................................................................................................................184
Publishable State .........................................................................................................................................................................................................186
Why Not Keep Them the Same?.................................................................................................................................................................189
Non-Sim Games and Summary .....................................................................................................................................................................190
Publishable State: Delivery, Updates, Interest Management, and Compression...............................................191
Interest Management: Traffic Optimization and Preventing Cheating ...................................................................191
Implementing Interest Management .....................................................................................................................................................193
Grid-Based Interest Management ................................................................................................................................................................................... 193
Interest Management as a Way to Prevent Information-Leak Cheating .......................................................194
On Frustum-Based Interest Management ........................................................................................................................................195
Before Compression: Minimizing Data ........................................................................................................................................................197
Compression ............................................................................................................................................................................................................................200
What Exactly Is “Compression”? ...................................................................................................................................................................201
On “Reference Bases” for Compression...................................................................................................................................................................... 203
“Reference Base” for Unreliable Communications Low-Latency Compressible State Sync.............................. 203
Advantage of Low-Latency Compressible State Sync over TCP and Reliable UDP ........................................... 206
Delta Compression .....................................................................................................................................................................................................206
Two Flavors of Delta Compression ................................................................................................................................................................................. 206
Delta Compression—Generalization to Arbitrary Trees ........................................................................................................................... 207
Delta Compression of Arbitrary Trees—Collecting Updates “On the Fly” ............................................................................ 208
Dead Reckoning As Compression ..............................................................................................................................................................209
Dead Reckoning As Compression: Variations ...................................................................................................................................................... 211
Classical Compression ............................................................................................................................................................................................212
Compressing Reliable Streams ............................................................................................................................................................................................ 212
Compressing Independent Packets ............................................................................................................................................................................... 213
Compression Using Acknowledged Packets ........................................................................................................................................................ 214
Combining Different Compression Mechanisms and the Law of Diminishing Returns ...................214
On Using Doubles with Lossless Compression .................................................................................................................................................. 215
On Adaptive Traffic Management ...................................................................................................................................................................216
Adaptive Traffic Management—UDP ......................................................................................................................................................217
Adaptive Traffic Management—TCP ........................................................................................................................................................218
Adaptive Traffic Management in the Context of Authoritative Servers ..........................................................219
Traffic and Real-Time Strategies .............................................................................................................................................................................220
Traffic Optimization: TL;DR and Recommendations ..........................................................................................................................222
xii • Development and Deployment of Multiplayer Online Games
MMOG and Scalability ...........................................................................................................................................224
On Shared-Nothing Architecture ...........................................................................................................................................................................225
An Obvious One: Separate NPC/AI ......................................................................................................................................................................226
Splitting into Areas ................................................................................................................................................................................................................227
Seamless Worlds: Overlap! ............................................................................................................................................................................................227
On Server-Side Uncertainty .........................................................................................................................................................................................229
Eliminating Uncertainty Completely: Time Sync .............................................................................................................................230
Synchronization without Rewind: CMS/LBTS. Lockstep ......................................................................................................230
Synchronization via Server Rewind: “Value Date” .....................................................................................................................231
Transient Events, Forwarded Inputs, and (Kinda-)Broadcasted Messages ...................232
Transient Events ........................................................................................................................................................................................................................232
Forwarded Inputs.....................................................................................................................................................................................................................233
Potential for Information Leaks .........................................................................................................................................................................233
Implementation ...................................................................................................................................................................................................................234
(Kinda-)Broadcasted Messages (Broadcast with Interest Management) ....................................................................234
Point-to-Point Communications and Non-Blocking RPCs .........................................................235
RPCs .......................................................................................................................................................................................................................................................236
Implementing Non-Blocking RPCs .................................................................................................................................................................237
Void vs Non-Void Non-Blocking RPCs .........................................................................................................................................................237
Non-Void RPCs ................................................................................................................................................................................................................238
Client-to-Server and Server-to-Client Point-to-Point Communications ......................................................................240
Inputs ..............................................................................................................................................................................................................................................241
Input Timestamping ..................................................................................................................................................................................................242
“Macroscopic” Client Actions .................................................................................................................................................................................243
Server-to-Client ...................................................................................................................................................................................................................244
Server-to-Server Communications ........................................................................................................................................................................245
Seamless Handling of Transient Disconnects ....................................................................................................................................245
Option 1. Separate Caller/Callee Handling .....................................................................................................................................246
Option 2. Two Guaranteed Delivery Streams ...............................................................................................................................247
Going Further: Inter-DB Async Transfer with Transactional Integrity .......................................................................248
Server-Side Entity Addressing ..............................................................................................................................................................................251
Server-Side: TCP Often Wins Over UDP ....................................................................................................................................................252
Using Message Queues for Server-to-Server Communications ......................................................................................254
MQs and Transactional Integrity .................................................................................................................................................................254
On Transactions in AMQP ......................................................................................................................................................................................................... 256
Brokered vs Brokerless MQ...............................................................................................................................................................................256
Broker as Directory Service ..................................................................................................................................................................................................... 256
Brokers and Transactional Integrity ............................................................................................................................................................................... 257
Using MQ on the Server-Side: Summary ............................................................................................................................................258
Contents • xiii
On Protocol Changes .............................................................................................................................................259
IDL: Encodings, Mappings, and Protocol Changes ..........................................................................262
Intra-Language vs Standalone .................................................................................................................................................................................263
IDL Development Flow .....................................................................................................................................................................................................264
IDL + Encoding + Mapping ............................................................................................................................................................................................265
Example: IDL ..................................................................................................................................................................................................................................267
On Sanitizing Input Data ...........................................................................................................................................................................................268
Test Case Generation ..............................................................................................................................................................................................269
Example: Mapping ...................................................................................................................................................................................................................269
Mapping to Existing Classes ..................................................................................................................................................................................272
Example: Encoding .................................................................................................................................................................................................................273
Protocol Changes and Backward Compatibility ....................................................................................................................................276
Field Identifiers ....................................................................................................................................................................................................................276
Growing Messages. Fences .....................................................................................................................................................................................277
Versioning ..................................................................................................................................................................................................................................280
Versions for Replies ...................................................................................................................................................................................................281
Merits of Versioning ..................................................................................................................................................................................................283
Which One to Choose? ................................................................................................................................................................................................283
Implementing IDL and Specific Encodings ..................................................................................................................................................284
Summary for Chapter 3 .........................................................................................................................................285
Bibliography ..................................................................................................................................................................................................................................287
Volume I Summary 291
What’s Next .....................................................................................................................................................................292
Index .....................................................................................................................................................................................295
Letter from the Author ..........................................................................................................................................307