Reverse Engineering for Beginners

Code Patterns
The method
Some basics
A short introduction to the CPU
Numeral Systems
Converting From One Radix To Another
An Empty Function
x86
ARM
MIPS
Empty Functions in Practice
Returning Values
x86
ARM
MIPS
Hello, world!
x86
x86-64
ARM
MIPS
Conclusion
Exercises
Function prologue and epilogue
Recursion
An Empty Function: redux
Returning Values: redux
Stack
Why does the stack grow backwards?
What is the stack used for?
A typical stack layout
Noise in stack
Exercises
Almost empty function
printf() with several arguments
x86
ARM
MIPS
Conclusion
By the way
scanf()
Simple example
The classic mistake
Global variables
scanf()
Exercise
Worth noting: global vs. local variables
Accessing passed arguments
x86
x64
ARM
MIPS
More about results returning
Attempt to use the result of a function returning void
What if we do not use the function result?
Returning a structure
Pointers
Returning values
Swap input values
GOTO operator
Dead code
Exercise
Conditional jumps
Simple example
Calculating absolute value
Ternary conditional operator
Getting minimal and maximal values
Conclusion
Exercise
Software cracking
Impossible shutdown practical joke (Windows 7)
switch()/case/default
Small number of cases
A lot of cases
When there are several case statements in one block
Fall-through
Exercises
Loops
Simple example
Memory blocks copying routine
Condition check
Conclusion
Exercises
More about strings
strlen()
Boundaries of strings
Replacing arithmetic instructions to other ones
Multiplication
Division
Exercise
Floating-point unit
IEEE 754
x86
ARM, MIPS, x86/x64 SIMD
C/C++
Simple example
Passing floating point numbers via arguments
Comparison example
Some constants
Copying
Stack, calculators and reverse Polish notation
80 bits?
x64
Exercises
Arrays
Simple example
Buffer overflow
Buffer overflow protection methods
One more word about arrays
Array of pointers to strings
Multidimensional arrays
Pack of strings as a two-dimensional array
Conclusion
Exercises
Example: a bug in Angband
Manipulating specific bit(s)
Specific bit checking
Setting and clearing specific bits
Shifts
Setting and clearing specific bits: FPU example
Counting bits set to 1
Conclusion
Exercises
Linear congruential generator
x86
x64
32-bit ARM
MIPS
Thread-safe version of the example
Structures
MSVC: SYSTEMTIME example
Let's allocate space for a structure using malloc()
UNIX: struct tm
Fields packing in structure
Nested structures
Bit fields in a structure
Exercises
The classic struct bug
Unions
Pseudo-random number generator example
Calculating machine epsilon
FSCALE instruction replacement
Fast square root calculation
Pointers to functions
MSVC
GCC
Danger of pointers to functions
64-bit values in 32-bit environment
Returning of 64-bit value
Arguments passing, addition, subtraction
Multiplication, division
Shifting right
Converting 32-bit value into 64-bit one
LARGE_INTEGER structure case
SIMD
Vectorization
SIMD strlen() implementation
64 bits
x86-64
ARM
Float point numbers
64-bit architecture criticism
Working with floating point numbers using SIMD
Simple example
Passing floating point number via arguments
Comparison example
Calculating machine epsilon: x64 and SIMD
Pseudo-random number generator example revisited
Summary
ARM-specific details
Number sign (#) before number
Addressing modes
Loading a constant into a register
Relocs in ARM64
MIPS-specific details
Loading a 32-bit constant into register
Further reading about MIPS
Important fundamentals
Integral datatypes
Bit
Nibble AKA nybble
Byte
Wide char
Signed integer vs unsigned
Word
Address register
Numbers
Signed number representations
Using IMUL over MUL
Couple of additions about two's complement form
-1
Integer overflow
AND
Checking if a value is on 2n boundary
KOI-8R Cyrillic encoding
AND and OR as subtraction and addition
ZX Spectrum ROM text strings
XOR (exclusive OR)
Logical difference
Everyday speech
Encryption
RAID4
XOR swap algorithm
XOR linked list
Switching value trick
Zobrist hashing / tabulation hashing
By the way
AND/OR/XOR as MOV
Population count
Endianness
Big-endian
Little-endian
Example
Bi-endian
Converting data
Memory
CPU
Branch predictors
Data dependencies
Hash functions
How do one-way functions work?
Slightly more advanced examples
Zero register
Double negation
const correctness
Overlapping const strings
strstr() example
qsort() revisited
Temperature converting
Integer values
Floating-point values
Fibonacci numbers
Example #1
Example #2
Summary
CRC32 calculation example
Network address calculation example
calc_network_address()
form_IP()
print_as_IP()
form_netmask() and set_bit()
Summary
Loops: several iterators
Three iterators
Two iterators
Intel C++ 2011 case
Duff's device
Should one use unrolled loops?
Division using multiplication
x86
How it works
ARM
MIPS
Exercise
String to number conversion (atoi())
Simple example
A slightly advanced example
Exercise
Inline functions
Strings and memory functions
C99 restrict
Branchless abs() function
Optimizing GCC 4.9.1 x64
Optimizing GCC 4.9 ARM64
Variadic functions
Computing arithmetic mean
vprintf() function case
Pin case
Format string exploit
Strings trimming
x64: Optimizing MSVC 2013
x64: Non-optimizing GCC 4.9.1
x64: Optimizing GCC 4.9.1
ARM64: Non-optimizing GCC (Linaro) 4.9
ARM64: Optimizing GCC (Linaro) 4.9
ARM: Optimizing Keil 6/2013 (ARM mode)
ARM: Optimizing Keil 6/2013 (Thumb mode)
MIPS
toupper() function
x64
ARM
Using bit operations
Summary
Obfuscation
Text strings
Executable code
Virtual machine / pseudo-code
Other things to mention
Exercise
C++
Classes
ostream
References
STL
Memory
Negative array indices
Addressing string from the end
Addressing some kind of block from the end
Arrays started at 1
More about pointers
Working with addresses instead of pointers
Passing values as pointers; tagged unions
Pointers abuse in Windows kernel
Null pointers
Array as function argument
Pointer to a function
Pointer to a function: copy protection
Pointer to a function: a common bug (or typo)
Pointer as object identificator
Oracle RDBMS and a simple garbage collector for C/C++
Loop optimizations
Weird loop optimization
Another loop optimization
More about structures
Sometimes a C structure can be used instead of array
Unsized array in C structure
Version of C structure
High-score file in "Block out" game and primitive serialization
memmove() and memcpy()
Anti-debugging trick
setjmp/longjmp
Other weird stack hacks
Accessing arguments/local variables of caller
Returning string
OpenMP
MSVC
GCC
Signed division using shifts
Another heisenbug
The case of forgotten return
Homework: more about function pointers and unions
Windows 16-bit
Example#1
Example #2
Example #3
Example #4
Example #5
Example #6
Java
Java
Introduction
Returning a value
Simple calculating functions
JVM memory model
Simple function calling
Calling beep()
Linear congruential PRNG
Conditional jumps
Passing arguments
Bitfields
Loops
switch()
Arrays
Strings
Exceptions
Classes
Simple patching
Summary
Finding important/interesting stuff in the code
Identification of executable files
Microsoft Visual C++
GCC
Intel Fortran
Watcom, OpenWatcom
Borland
Other known DLLs
Communication with outer world (function level)
Communication with the outer world (win32)
Often used functions in the Windows API
Extending trial period
Removing nag dialog box
tracer: Intercepting all functions in specific module
Strings
Text strings
Finding strings in binary
Error/debug messages
Suspicious magic strings
Calls to assert()
Constants
Magic numbers
Specific constants
Searching for constants
Finding the right instructions
Suspicious code patterns
XOR instructions
Hand-written assembly code
Using magic numbers while tracing
Loops
Some binary file patterns
Memory "snapshots" comparing
ISA detection
Incorrectly disassembled code
Correctly disassembled code
Other things
General idea
Order of functions in binary code
Tiny functions
C++
Crash on purpose
OS-specific
Arguments passing methods (calling conventions)
cdecl
stdcall
fastcall
thiscall
x86-64
Return values of float and double type
Modifying arguments
Taking a pointer to function argument
Python ctypes problem (x86 assembly homework)
Cdecl example: a DLL
Thread Local Storage
Linear congruential generator revisited
System calls (syscall-s)
Linux
Windows
Linux
Position-independent code
LD_PRELOAD hack in Linux
Windows NT
CRT (win32)
Win32 PE
Windows SEH
Windows NT: Critical section
Tools
Binary analysis
Disassemblers
Decompilers
Patch comparison/diffing
Live analysis
Debuggers
Library calls tracing
System calls tracing
Network sniffing
Sysinternals
Valgrind
Emulators
Other tools
SMT solvers
Calculators
Do You Think Something Is Missing Here?
Case studies
Mahjong solitaire prank (Windows 7)
Task manager practical joke (Windows Vista)
Using LEA to load values
Color Lines game practical joke
Minesweeper (Windows XP)
Finding grid automatically
Exercises
Hacking Windows clock
(Windows 7) Solitaire: practical jokes
51 cards
53 cards
FreeCell prank (Windows 7)
Part I
Part II: breaking the Select Game submenu
Dongles
Example #1: MacOS Classic and PowerPC
Example #2: SCO OpenServer
Example #3: MS-DOS
Encrypted database case #1
Base64 and entropy
Is data compressed?
Is data encrypted?
CryptoPP
Cipher Feedback mode
Initializing Vector
Structure of the buffer
Noise at the end
Conclusion
Post Scriptum: brute-forcing IV
Overclocking Cointerra Bitcoin miner
Breaking simple executable code encryptor
Other ideas to consider
SAP
About SAP client network traffic compression
SAP 6.0 password checking functions
Oracle RDBMS
V$VERSION table in the Oracle RDBMS
X$KSMLRU table in Oracle RDBMS
V$TIMER table in Oracle RDBMS
Handwritten assembly code
EICAR test file
Demos
10 PRINT CHR$(205.5+RND(1)); : GOTO 10
Mandelbrot set
A nasty bug in MSVCRT.DLL
Other examples
Examples of reversing proprietary file formats
Primitive XOR-encryption
Simplest ever XOR encryption
Norton Guide: simplest possible 1-byte XOR encryption
Simplest possible 4-byte XOR encryption
Simple encryption using XOR mask
Simple encryption using XOR mask, case II
Homework
Information entropy
Analyzing entropy in Mathematica
Conclusion
Tools
A word about primitive encryption like XORing
More about entropy of executable code
PRNG
More examples
Entropy of various files
Making lower level of entropy
Millenium game save file
fortune program indexing file
Hacking
The files
Oracle RDBMS: .SYM-files
Oracle RDBMS: .MSB-files
Summary
Exercises
Further reading
Dynamic binary instrumentation
Using PIN DBI for XOR interception
Cracking Minesweeper with PIN
Intercepting all rand() calls
Replacing rand() calls with our function
Peeking into placement of mines
Exercise
Building Pin
Why ``instrumentation''?
Other things
Executable files patching
x86 code
Function arguments number statistics
Compiler intrinsic
Compiler's anomalies
Oracle RDBMS 11.2 and Intel C++ 10.1
MSVC 6.0
ftol2() in MSVC 2012
Summary
Itanium
8086 memory model
Basic blocks reordering
Profile-guided optimization
My experience with Hex-Rays 2.2.0
Bugs
Odd peculiarities
Silence
Comma
Data types
Long and messed expressions
De Morgan's laws and decompilation
My plan
Summary
Cyclomatic complexity
Books/blogs worth reading
Books and other materials
Reverse Engineering
Windows
C/C++
x86 / x86-64
ARM
Assembly language
Java
UNIX
Programming in general
Cryptography
Something even easier
Communities
Afterword
Questions?
Appendix
x86
Terminology
General purpose registers
FPU registers
SIMD registers
Debugging registers
Instructions
npad
ARM
Terminology
Versions
32-bit ARM (AArch32)
64-bit ARM (AArch64)
Instructions
MIPS
Registers
Instructions
Some GCC library functions
Some MSVC library functions
Cheatsheets
IDA
OllyDbg
MSVC
GCC
GDB
Acronyms Used
Glossary
Index