Development and deployment of Multiplayer Online Games Vol II

Introduction ........................................................................................................................... 17
The Hare and the People Behind... ..................................................................................... 17
On Real-World Experiences ................................................................................................ 18
Is This Book for You? .......................................................................................................... 19
How to Read This Book ...................................................................................................... 21
Setting the Context. Vol. I Summary ...................................................................................... 25
Chapter 4. DIY vs. Re-Use: In Search of Balance ..................................................................... 27
Dependencies and Vendor Lock-Ins ................................................................................... 28
Impact of Vendor Lock-Ins .............................................................................................. 28
Weakening/Isolating Dependencies ............................................................................... 29
On Platform Lock-Ins and Cross-Platform Code ......................................................... 30
Non-Working Cross-Platform: Rewriting Low-Level APIs ....................................... 30
Working Cross-Platform: Isolation Layer ................................................................ 31
Working Cross-Platform: Isolation Layer/Lowest Common Denominator ......... 31
Working Cross-Platform: Isolation Layer/In Terms of App ................................. 32
Working Cross-Platform: Summary ........................................................................ 33
Isolating Arbitrary Libraries. Isolation APIs in terms of App ....................................... 33
On “Wrapping, Then Wrapping Some More” ......................................................... 34
Vigilance, and More Vigilance .................................................................................... 35
“Two Platforms” Approach ..................................................................................... 36
TL;DR on Vendor Lock-Ins and Isolating Dependencies ................................................. 36
Business Perspective: DIY Your Added Value ..................................................................... 37
Engine-Centric Approach: Game Engine as an Inherent Vendor Lock-In ........................... 38
Engine-Centric Approach: Pretty Much Inevitable for Indie RPG/FPS games ................ 38
Engine-Centric Approach: You Still Need to Understand How It Works ........................ 39
Engine-Centric Approach: On “Temporary” Dependencies ........................................... 40
“Re-Use Everything in Sight” Approach: An Integration Nightmare .................................. 42
“DIY Everything”: The Risk of a Never-Ending Story ........................................................... 44
“Responsible Re-Use” Approach: In Search of Balance ...................................................... 45
“Responsible Re-Use” Examples ..................................................................................... 48
“Responsible Re-Use”: on “Temporary” Dependencies ................................................. 50
Chapter 4 Summary ............................................................................................................ 51
Bibliography ........................................................................................................................... 51
Chapter 5. (Re)Actor-fest Architecture. It Just Works. ........................................................... 52
To React or Not to React? That is (Almost) No Question ................................................... 53
“Almost” Before “No Question” ..................................................................................... 55
Reactors or Not—Stay Away from Thread Sync in your Game Logic ......................... 55
On Terminology. Actor, Reactor, Event-Driven Program, Game Loop, or Ad-Hoc Finite
State Machine? We’ll Name it (Re)Actor. .......................................................................... 58
Game Loop: Game Programming Classic ........................................................................... 59
(Re)Actor as a Generalization of Game Loop ..................................................................... 61
Other Event-Driven Systems: GUI, Erlang, Node.js, and Java Reactor ........................... 63
On Separating Infrastructure Code from Logic Code ..................................................... 64
Advantages of (Re)Actors ............................................................................................... 66(Re)Actors in Game Engines ........................................................................................... 67
Two All-Important Improvements to Classical Event-Driven Programming: Mostly-Non-
Blocking Processing and Determinism ............................................................................... 69
Non-Blocking Processing .................................................................................................... 69
What NOT to Use—“OO” RPC Frameworks ................................................................... 70
To Block, or Not to Block, That Is the Question. Mostly-Non-Blocking (Re)Actors ........ 73
Case 1. Processing Input Events While Call Is in Progress Is Required at Logic Level . 74
Case 2. No Processing at Logic level While Call Is In Progress .................................... 74
Blocking or Non-Blocking? Mostly-Non-Blocking ....................................................... 76
Implementing Non-Blocking Processing for Games ....................................................... 77
Waits/Timers .............................................................................................................. 78
Non-Blocking State Publishing and (Kinda-)Broadcast Messages .............................. 78
Point-to-Point Communications and Other Request-Response Stuff ........................ 79
Request-Response .................................................................................................. 79
Request-Response Complication: What to Do When the Call is Completed? .... 80
Handling Returns in Non-Blocking Way in (Re)Actors ................................................ 82
Take 1. Naïve Approach: Plain Messages (Will Work, But Is Plain Ugly) ................ 84
Take 2. Void-Only RPCs (A Tiny Bit Better, Still Really Ugly) ................................... 89
Take 3. OO-Style: Less Error-Prone, But Still Way Too Much Boilerplate .............. 92
Exceptions .............................................................................................................. 95
Cascading Exception Handlers ............................................................................ 98
Take 4. Lambda Pyramid ...................................................................................... 100
Cascaded Exception Handling, Lambda Style ................................................... 103
Take 5. (Re)Actor Futures ..................................................................................... 104
Similarities and Differences from Existing Futures/Promises ........................... 107
Take 5 Summary ............................................................................................... 108
Take 6. Code Builder ............................................................................................. 109
Take 6a. Enter C++ Preprocessor ...................................................................... 110
Offloading ......................................................................................................... 112
Offloading Caveat #1: Deep Copy Required ................................................. 112
Offloading Caveat #2: Keeping Portions Large ............................................. 113
Offloading Caveat #3: DON’T Offload Unless Proven Necessary .................. 114
Yet Another Offloading Caveat: Flow Control .............................................. 115
Really Serious Stuff: HPX .............................................................................. 116
Last But Not Least: Implementing Offloading Is Not in Scope Now ............. 116
Take 7. Fibers/Stackful Coroutines ....................................................................... 117
On boost:: coroutines and boost::context ....................................................... 119
On Using goroutines-for-callbacks: BEWARE THREAD SYNC! ........................... 119
Take 8. async/await (.NET, Node.js, and not-standard-yet C++) .......................... 120
Across-the-Board Generalizations ........................................................................ 122
Surprise: All the Different Takes are Functionally Equivalent, and Very Close
Performance-Wise Too ..................................................................................... 122
Similarities to Node.js ....................................................................................... 122
Handling Non-Blocking Returns in Different Programming Languages ............ 123
Serializing (Re)Actor State ................................................................................ 123
Serializing Lambda Closures and co_await frames in C++ ............................ 124
Why So Much Discussion of This One Thing? ....................................................... 127TL;DR for Non-Blocking Communications in (Re)Actors ....................................... 127
Determinism ..................................................................................................................... 128
Distributed Systems: Debugging Nightmare ................................................................ 128
Non-Deterministic Tests are Pointless ..................................................................... 130
The Holy Grail of Post-Factum .................................................................................. 130
Portability: Platform-Independent Logic as “Nothing but Moving Bits Around” ......... 131
Stronger than Platform-Independent: Determinism .................................................... 132
Deterministic Logic: Benefits ........................................................................................ 132
On Replay-Based Regression Testing and Patches ....................................................... 136
On (Re)Actors and Fuzz Testing .................................................................................... 138
On Determinism and User Replay ................................................................................ 139
Implementing Deterministic Logic ............................................................................... 140
Deterministic System: Modes of Operation ............................................................. 140
Implementing Inputs-Log ......................................................................................... 141
Going Circular ....................................................................................................... 142
Recordable/Replayable (Re)Actor ............................................................................ 143
Implementing Deterministic Logic: Non-Determinism Due to System Calls ............ 144
Dealing with System Calls: Original Non-Deterministic Code ............................... 144
Dealing with System Calls: Call Wrapping ............................................................ 145
Dealing with System Calls: Pure Logic .................................................................. 148
Dealing with System Calls: TLS Compromise ........................................................ 149
Dealing with System Calls: Pools of On-Demand Data ......................................... 151
Dealing with System Calls: On-Demand Data via Exceptions ............................... 151
Dealing with System Calls: RPC-like Handling ...................................................... 153
Dealing with System Calls: allocations ................................................................. 153
Dealing with System Calls: Which System Functions We’re Speaking About and
What to Do About Them? .................................................................................... 155
Implementing Deterministic Logic: Other Sources of Non-Determinism ................. 158
On Undefined Behavior ........................................................................................ 158
No Access to Non-const Globals and TLS ............................................................. 158
On Threads ........................................................................................................... 160
On Determinism of the Whole Distributed Systems ............................................ 160
Implementing Deterministic Logic: Non-Issues ........................................................ 160
PRNG .................................................................................................................... 161
Logging/Tracing .................................................................................................... 161
Caching ................................................................................................................. 162
On Isolation Perimeters ........................................................................................ 162
Implementing Deterministic Logic: Cross-Platform Determinism ............................ 163
Achieving Cross-Platform Determinism ................................................................ 164
Implementing Deterministic Logic: Summary .......................................................... 166
Types of Determinism vs Deterministic Goodies ......................................................... 167
Relation of Deterministic (Re)Actors to Deterministic Finite Automata ...................... 169
Deterministic Finite State Machines: Nothing New—Let’s Just Start Using Them ... 171
TL;DR for Determinism Section .................................................................................... 172
Divide et Impera, or How to Split the Hare the Hair the (Re)Actor .................................. 172
On Importance of the DIY IDL Compiler ....................................................................... 173
Big-n-Ugly-Switch ..................................................................................................... 174Generated Switch ..................................................................................................... 175
Stream-based Unmarshalling ................................................................................... 175
Composition: (Re)Actor-within-(Re)Actor .................................................................... 175
State Pattern ................................................................................................................ 177
State-Specific Handling ............................................................................................. 177
Common Data Members ...................................................................................... 178
Potentially Expensive Allocations ......................................................................... 179
Hierarchical States .................................................................................................... 179
Stack-of-States ......................................................................................................... 181
VALIDATE-CALCULATE-MODIFY-SIMULATE Pattern ..................................................... 181
VALIDATE-CALCULATE-MODIFY-SIMULATE and Exceptions ..................................... 182
Exceptions Before MODIFY Stage Are Safe, Including CPU Exceptions ................ 183
RAII Equivalents in Different Programming Languages ........................................ 183
Posting Messages (calling RPCs, etc.) Within VALIDATE/CALCULATE ................... 184
Exception Safety for MODIFY and SIMULATE Stages ............................................ 185
Divide et Impera Summary ........................................................................................... 185
(Kinda-)Scaling Individual (Re)Actors ............................................................................... 186
Splitting and Offloading ................................................................................................ 187
(Re)Actor-with-Mirrored-State—Limited Relief ........................................................... 187
(Re)Actor-with-Extractors ............................................................................................ 188
(Re)Actor-fest Architecture: Putting It All Together ......................................................... 192
Philosophy of (Re)Actor-fest ........................................................................................ 193
(Re)Actor-fest and Conway’s Law ............................................................................. 193
Implementing (Re)Actor-fest ........................................................................................ 194
(Re)Actor Factories ................................................................................................... 194
That’s Pretty Much It ................................................................................................ 195
(Re)Actor-fest and Programming Languages ................................................................ 195
Relation of (Re)Actor-Fest to Other Systems ............................................................... 196
Relation to Actor Concurrency ................................................................................. 196
Relation to Erlang Concurrency, Akka Actors, and Node.js ...................................... 197
(Re)Actors and Microservices as Close Cousins ........................................................ 198
Physical Server—VM Docker—(Re)Actor as a Spectrum of Tradeoffs Between
Isolation and Flexibility ......................................................................................... 199
Summary of Chapter 5 ..................................................................................................... 200
Bibliography ................................................................................................................. 201
Appendix 5.A. C++-Specific Examples and Comments for Chapter 5 ............................... 205
Avoiding Expensive Allocations .................................................................................... 227
C++: Enforcing const-ness for VALIDATE and CALCULATE stages in VALIDATE-
CALCULATE-MODIFY-SIMULATE pattern ...................................................................... 228
On Posting messages from VALIDATE/CALCULATE in C++ ....................................... 230
Chapter 6. Client-Side Architecture ...................................................................................... 231
Graphics 101 ..................................................................................................................... 231
On Developers, Game Designers, and Artists ............................................................... 232
On Using Game Engines and Vendor Lock-In ............................................................... 233
Types of Graphics ......................................................................................................... 234
Games with Rudimentary Graphics .......................................................................... 235
Games with 2D Graphics .......................................................................................... 235On Pre-rendered 3D ................................................................................................. 237
Games with 3D Graphics .......................................................................................... 239
Generic Client Architecture .............................................................................................. 240
Logic-to-Graphics API ................................................................................................... 241
Naïve vs Logic-to-Graphics for Cross-Platform Development .................................. 244
Logic-to-Graphics Benefits ....................................................................................... 245
Logic-to-Graphics Costs ............................................................................................ 246
Dual Graphics, Including 2D+3D Graphics ................................................................ 246
Modules and Their Relationships ................................................................................. 247
Game Logic Module .................................................................................................. 247
Game Logic Module & Graphics ........................................................................... 248
Game Logic Module: Client-Side Prediction and Simulation ................................ 249
Game Logic Module: Game Loop ......................................................................... 249
Game Logic Module: Keeping it Cross-Platform ................................................... 250
Animation&Rendering Module ................................................................................ 251
Communications Module ......................................................................................... 251
Sound Module .......................................................................................................... 252
Relation to MVC ........................................................................................................... 252
Differences from Classical 3D Single-Player Game ....................................................... 253
Interaction Examples in 3D World: Single-Player vs MOG ....................................... 254
MMOFPS Interaction Example (Shooting) ............................................................ 254
MMORPG Interaction Example (Ragdoll) ............................................................. 255
UI Interaction Example ......................................................................................... 256
Pre-alloc Everything ................................................................................................. 256
Progressive Downloading ......................................................................................... 257
(Re)Actor-fest Client-Side Architecture ............................................................................ 258
(Re)Actor-fest Client Architecture ................................................................................ 259
(Re)Actor Specifics ........................................................................................................ 262
Animation&Rendering (Re)Actor and Game Loop ................................................... 263
Communications (Re)Actor and Blocking/Non-Blocking Sockets ............................. 265
Other (Re)Actors....................................................................................................... 266
On (Re)Actors and Latencies ........................................................................................ 266
(Re)Actor-fest Variations .............................................................................................. 267
On Code Bases for Different Platforms ........................................................................ 270
Scaling (Re)Actor-fest Architecture on the Client ........................................................ 271
Parallelizing Client-Side (Re)Actors .......................................................................... 272
Summary of (Re)Actor-fest Architecture on the Client-Side ........................................ 275
Programming Language for Game Client ......................................................................... 275
One Language for Programmers, Another for Game Designers (MMORPG/MMOFPS
etc.) .............................................................................................................................. 276
A Word on CUDA and OpenCL ...................................................................................... 277
Different Languages Provide Different Protection from Bot Writers ........................... 278
Resilience to Reverse-Engineering of Different Programming Languages ............... 279
Compiled Languages ............................................................................................. 279
Languages That Compile to Byte-Code ................................................................. 281
Scripting/Interpreted Languages .......................................................................... 282
On asm.js, wasm, and Emscripten ........................................................................ 282Summary .............................................................................................................. 283
Language Availability for Client-Side Platforms ........................................................... 284
On Garbage Collection ................................................................................................. 285
On “Stop-the-World” Problem ................................................................................. 286
To GC or Not to GC? ................................................................................................. 290
On Consistency Between Client-Side and Server-Side Languages ................................ 291
Sprinkle with All the Usual Considerations ................................................................... 292
C++ as a Default Game Programming Language .......................................................... 293
Big Fat Browser Problem .............................................................................................. 294
Client-on-Server Trick ............................................................................................... 296
On UI Dialogs .................................................................................................................... 297
Fixed layouts—Way to Disaster .................................................................................... 298
Way Out—HTML-Like Flexible Layouts ........................................................................ 298
On Modal Dialogs and MOGs ....................................................................................... 299
On Using Browser Window Alongside the Client ............................................................. 299
Downloadable Client + Completely Separate Web Site ............................................... 300
Downloadable Client with a System-Provided In-App Browser ................................... 301
Downloadable Client with an Embedded Web Engine ................................................. 301
Embedded HTML Engine but Own Communications ............................................... 302
On Embedding and emscripten ................................................................................ 303
Integrating with 3rd-party Websites. 3rd-party Logins ................................................ 304
Bottom Line for Chapter 6 ................................................................................................ 305
Bibliography ................................................................................................................. 307
Chapter 7. Client-Driven Development: Unity, UE, Lumberyard, Urho3D, and 3rd-Party
Network Libraries ................................................................................................................. 309
On Client-Driven vs. Server-Driven Development Workflows .......................................... 309
On Server-Driven Development Workflow .................................................................. 309
Client-Driven Development Flow ................................................................................. 310
Implementing Client-Driven Workflows ....................................................................... 310
Single-player Prototype and “Continuous Conversion” ........................................... 311
Engine-Provided Server vs. Standalone Server ......................................................... 312
Important Clarification: Development Workflow vs Data Flow ............................... 312
Most Popular 3rd-party Game Engines ............................................................................ 313
Unity 5 .......................................................................................................................... 313
Event-Driven Programming/Reactors ...................................................................... 314
Built-In Communications: HLAPI (for Engine-Provided Server) ................................ 315
State Synchronization ........................................................................................... 316
RPCs (a.k.a. “Remote Actions”) ............................................................................ 317
HLAPI Summary .................................................................................................... 319
Built-In Communications: LLAPI (Both for Engine-Provided Server and Standalone
Server) ...................................................................................................................... 319
3rd-party Communications for Unity: Photon Server .............................................. 320
Photon Server SDK ................................................................................................ 322
Photon Cloud (PUN and Realtime) ....................................................................... 323
3rd-party Communications for Unity: SmartFoxServer ............................................ 324
3rd-party Communications for Unity: uLink ............................................................. 326
3rd-party Communications for Unity: DarkRift ........................................................ 3273rd-party Communications for Unity: Lower-Level Libraries ................................... 328
Unity 5 Summary ...................................................................................................... 329
Engine-Provided Server. HLAPI Now, Probably LLAPI Later ................................. 330
Standalone Server with Export from Unity ........................................................... 332
Engine-Provided vs Standalone: Which One Is Better? ........................................ 333
Unreal Engine 4 ............................................................................................................ 333
Event-Driven Programming/Reactors ...................................................................... 333
UE for MOG .............................................................................................................. 334
UE Communications: Very Close to Unity 5 HLAPI ................................................... 334
UE Communications: Lower-Level C/C++ Libraries .................................................. 335
Reliable UDP Libraries .......................................................................................... 335
Socket Wrapper Libraries ..................................................................................... 337
UE4 Summary: Engine-Provided and Standalone Servers ........................................ 337
Amazon Lumberyard .................................................................................................... 338
A Choice Between Amazon-Only Hosting—and (Hopefully) Co-Location ................ 338
Amazon Lumberyard: General ................................................................................. 340
Amazon Lumberyard: Platforms, Programming Languages, and Event-Driven
Programming ............................................................................................................ 340
Amazon Lumberyard Networking: GridMate ........................................................... 341
AWS Integration: Amazon GameLift ..................................................................... 341
Amazon Lumberyard: Summary and Engine-Provided/Standalone Servers ............ 342
Urho3D ......................................................................................................................... 342
Urho3D: Supported Platforms, Programming Languages, and Event-Driven
Programming ............................................................................................................ 343
Urho3D Networking ................................................................................................. 343
Urho3D: Summary and Engine-Provided/Standalone Servers ................................. 344
The Engine That Didn’t Make It—Source ..................................................................... 344
Comparison Table ......................................................................................................... 344
Summary for Chapter 7 .................................................................................................... 352
Bibliography ................................................................................................................. 353
Vol. II Summary .................................................................................................................... 355
What’s Next ...................................................................................................................... 355
Letter from the Author ..................................................................................................... 358