R: Data Analysis and Visualization

Cover......Page 1
TOC......Page 10
Navigating the basics......Page 26
Getting help in R......Page 32
Vectors......Page 33
Functions......Page 39
Matrices......Page 42
Loading data into R......Page 45
_GoBack......Page 48
Univariate data......Page 52
Frequency distributions......Page 53
_GoBack......Page 57
Spread......Page 61
Populations, samples, and estimation......Page 64
Probability distributions......Page 66
Visualization methods......Page 71
Multivariate data......Page 80
Relationships between a categorical and a continuous variable......Page 81
Relationships between two categorical variables......Page 86
The relationship between two continuous variables......Page 89
Visualization methods......Page 97
Basic probability......Page 106
A tale of two interpretations......Page 112
Sampling from distributions......Page 113
The normal distribution......Page 117
Estimating means......Page 124
The sampling distribution......Page 127
Interval estimation......Page 130
Smaller samples......Page 134
Null Hypothesis Significance Testing......Page 138
Testing the mean of one sample......Page 148
Testing two means......Page 154
Testing more than two means......Page 159
Testing independence of proportions......Page 162
What if my assumptions are unfounded?......Page 164
Bayesian Methods......Page 170
The big idea behind Bayesian analysis......Page 171
Choosing a prior......Page 177
Who cares about coin flips......Page 180
Enter MCMC – stage left......Page 182
Using JAGS and runjags......Page 185
Fitting distributions the Bayesian way......Page 190
The Bayesian independent samples t-test......Page 194
Predicting Continuous Variables......Page 200
Linear models......Page 201
Simple linear regression......Page 203
Simple linear regression with a binary predictor......Page 210
Multiple regression......Page 215
Regression with a non-binary predictor......Page 219
Kitchen sink regression......Page 221
The bias-variance trade-off......Page 223
Linear regression diagnostics......Page 231
Advanced topics......Page 237
Exercises......Page 239
Summary......Page 240
Predicting Categorical Variables......Page 242
k-Nearest Neighbors......Page 243
Logistic regression......Page 252
Decision trees......Page 258
Random forests......Page 263
Choosing a classifier......Page 265
Sources of Data......Page 274
Relational Databases......Page 275
Using JSON......Page 280
XML......Page 289
Online repositories......Page 297
Dealing with Messy Data......Page 300
Analysis with missing data......Page 301
Analysis with unsanitized data......Page 322
Other messiness......Page 330
Dealing with Large Data......Page 334
Wait to optimize......Page 335
Using a bigger and faster machine......Page 336
Be smart about your code......Page 337
Using optimized packages......Page 340
Using another R implementation......Page 342
Use parallelization......Page 343
Using Rcpp......Page 356
Be smarter about your code......Page 362
Reproducibility and
Best Practices......Page 366
R Scripting......Page 367
R projects......Page 377
Version control......Page 379
Communicating results......Page 381
R Graphics......Page 396
Base graphics using the default package......Page 397
Trellis graphs using lattice......Page 398
Graphs inspired by Grammar of Graphics......Page 403
Introduction......Page 406
Creating basic scatter plots......Page 407
Creating line graphs......Page 412
Creating bar charts......Page 415
Creating histograms and density plots......Page 418
Creating box plots......Page 422
Adjusting x and y axes' limits......Page 424
Creating heat maps......Page 428
Creating pairs plots......Page 431
Creating multiple plot matrix layouts......Page 433
Adding and formatting legends......Page 437
Creating graphs with maps......Page 440
Saving and exporting graphs......Page 444
Beyond the Basics – Adjusting Key Parameters......Page 448
Setting colors of points, lines, and bars......Page 449
Setting plot background colors......Page 454
Setting colors for text elements – axis annotations, labels, plot titles, and legends......Page 456
Choosing color combinations and palettes......Page 459
Setting fonts for annotations and titles......Page 461
Choosing plotting point symbol styles
and sizes......Page 463
Choosing line styles and width......Page 466
Choosing box styles......Page 468
Adjusting axis annotations and tick marks......Page 471
Formatting log axes......Page 473
Setting graph margins and dimensions......Page 474
Introduction......Page 476
Grouping data points within a scatter plot......Page 477
Highlighting grouped data points by size
and symbol type......Page 480
Labeling data points......Page 483
Correlation matrix using pairs plots......Page 485
Adding error bars......Page 487
Using jitter to distinguish closely packed data points......Page 490
Adding linear model lines......Page 493
Adding nonlinear model curves......Page 494
Adding nonparametric model curves
with lowess......Page 495
Creating three-dimensional scatter plots......Page 497
Creating Quantile-Quantile plots......Page 500
Displaying the data density on axes......Page 502
Creating scatter plots with a smoothed density representation......Page 504
Introduction......Page 508
Adding customized legends for
multiple-line graphs......Page 509
Using margin labels instead of legends for multiple-line graphs......Page 512
Adding horizontal and vertical grid lines......Page 515
Adding marker lines at specific x and y values using abline......Page 517
Creating sparklines......Page 519
Plotting functions of a variable in a dataset......Page 521
Formatting time series data for plotting......Page 524
Plotting the date or time variable on the
x axis......Page 526
Annotating axis labels in different human-readable time formats......Page 528
Adding vertical markers to indicate specific time events......Page 529
Plotting data with varying time-averaging periods......Page 531
Creating stock charts......Page 533
Introduction......Page 536
Creating bar charts with more than one factor variable......Page 537
Creating stacked bar charts......Page 539
Adjusting the orientation of bars – horizontal and vertical......Page 542
Adjusting bar widths, spacing, colors,
and borders......Page 544
Displaying values on top of or next to
the bars......Page 547
Placing labels inside bars......Page 550
Creating bar charts with vertical error bars......Page 551
Modifying dot charts by grouping variables......Page 553
Making better, readable pie charts with clockwise-ordered slices......Page 556
Labeling a pie chart with percentage
values for each slice......Page 558
Adding a legend to a pie chart......Page 560
Introduction......Page 564
Visualizing distributions as count frequencies or probability densities......Page 565
Setting the bin size and the number
of breaks......Page 567
Adjusting histogram styles – bar colors, borders, and axes......Page 570
Overlaying a density line over a histogram......Page 572
Multiple histograms along the diagonal of
a pairs plot......Page 573
Histograms in the margins of line and scatter plots......Page 575
Introduction......Page 580
Creating box plots with narrow boxes for a small number of variables......Page 581
Grouping over a variable......Page 584
Varying box widths by the number of observations......Page 586
Creating box plots with notches......Page 588
Including or excluding outliers......Page 589
Creating horizontal box plots......Page 591
Changing the box styling......Page 592
Adjusting the extent of plot whiskers outside the box......Page 594
Showing the number of observations......Page 596
Splitting a variable at arbitrary values
into subsets......Page 599
Introduction......Page 604
Creating heat maps of a single Z variable with a scale......Page 605
Creating correlation heat maps......Page 608
Summarizing multivariate data in a single heat map......Page 611
Creating contour plots......Page 616
Creating filled contour plots......Page 618
Creating three-dimensional surface plots......Page 622
Visualizing time series as calendar
heat maps......Page 624
Introduction......Page 630
Plotting global data by countries on a
world map......Page 631
Creating graphs with regional maps......Page 635
Plotting data on Google maps......Page 640
Creating and reading KML data......Page 644
Working with ESRI shapefiles......Page 645
Introduction......Page 650
Creating bar charts......Page 651
Creating stacked bar charts......Page 654
Creating bar charts to visualize cross-tabulation......Page 656
Creating a conditional histogram......Page 659
Visualizing distributions through a kernel-density plot......Page 662
Creating a normal Q-Q plot......Page 665
Visualizing an empirical Cumulative Distribution Function......Page 667
Creating a boxplot......Page 670
Creating a conditional scatter plot......Page 673
Introduction......Page 678
Creating bar charts......Page 679
Creating multiple bar charts......Page 682
Creating a bar chart with error bars......Page 686
Visualizing the density of a numeric variable......Page 688
Creating a box plot......Page 690
Creating a layered plot with a scatter plot and fitted line......Page 691
Creating a line chart......Page 693
Graph annotation with ggplot......Page 695
Introduction......Page 698
Multivariate continuous data visualization......Page 700
Multivariate categorical data visualization......Page 702
Visualizing mixed data......Page 704
Zooming and filtering......Page 705
Introduction......Page 708
Three-dimensional scatter plots......Page 709
Three-dimensional scatter plots with a regression plane......Page 712
Three-dimensional bar charts......Page 714
Three-dimensional density plots......Page 715
Introduction......Page 718
Exporting graphs in high-resolution image formats – PNG, JPEG, BMP, and TIFF......Page 719
Exporting graphs in vector formats – SVG, PDF, and PS......Page 722
Adding mathematical and scientific notations (typesetting)......Page 724
Adding text descriptions to graphs......Page 729
Using graph templates......Page 732
Choosing font families and styles under Windows, Mac OS X, and Linux......Page 736
Choosing fonts for PostScripts and PDFs......Page 739
Warming Up......Page 746
Big data......Page 747
Data mining......Page 749
Social network mining......Page 753
Text mining......Page 756
Web data mining......Page 757
Statistics......Page 759
Machine learning......Page 761
Data attributes and description......Page 762
Data cleaning......Page 766
Data dimension reduction......Page 768
Data transformation and discretization......Page 770
Visualization of results......Page 772
Mining Frequent Patterns, Associations, and Correlations......Page 776
An overview of associations and patterns......Page 777
Market basket analysis......Page 783
Hybrid association rules mining......Page 803
Mining sequence dataset......Page 804
The R implementation......Page 807
High-performance algorithms......Page 810
Classification......Page 812
Classification......Page 813
Generic decision tree induction......Page 815
High-value credit card customers classification using ID3......Page 821
Web spam detection using C4.5......Page 827
Web key resource page judgment using CART......Page 835
Trojan traffic identification method and Bayes classification......Page 838
Identify spam e-mail and Naïve Bayes classification......Page 843
Rule-based classification of player types in computer games and rule-based classification......Page 847
Ensemble (EM) methods......Page 856
Biological traits and the Bayesian belief network......Page 863
Protein classification and the k-Nearest Neighbors algorithm......Page 865
Document retrieval and Support Vector Machine......Page 866
Classification using frequent patterns......Page 873
Classification using the backpropagation algorithm......Page 876
Cluster Analysis......Page 884
Search engines and the k-means algorithm......Page 887
Automatic abstraction of document texts and the k-medoids algorithm......Page 895
The CLARA algorithm......Page 898
CLARANS......Page 900
Unsupervised image categorization and affinity propagation clustering......Page 901
News categorization and hierarchical clustering......Page 905
Customer categorization analysis of e-commerce and DBSCAN......Page 914
Clustering web pages and OPTICS......Page 917
Visitor analysis in the browser cache and DENCLUE......Page 920
Recommendation system and STING......Page 925
Web sentiment analysis and CLIQUE......Page 926
Opinion mining and WAVE clustering......Page 928
User search intent and the EM algorithm......Page 931
Customer purchase data analysis and clustering high-dimensional data......Page 933
SNS and clustering graph and network data......Page 936
Outlier Detection......Page 940
Credit card fraud detection and statistical methods......Page 941
Activity monitoring – the detection of fraud involving mobile phones and proximity-based methods......Page 944
Intrusion detection and density-based methods......Page 950
Intrusion detection and clustering-based methods......Page 955
Monitoring the performance of the web server and classification-based methods......Page 957
Detecting novelty in text, topic detection, and mining contextual outliers......Page 959
Collective outliers on spatial data......Page 962
Outlier detection in high-dimensional data......Page 964
chapter_7......Page 968
The credit card transaction flow and STREAM algorithm......Page 969
Predicting future prices and time-series analysis......Page 972
Stock market data and time-series clustering and classification......Page 975
Web click streams and mining symbolic sequences......Page 978
Mining sequence patterns in transactional databases......Page 982
Graph mining......Page 986
Mining frequent subgraph patterns......Page 987
Social network mining......Page 991
Mining Text and Web Data......Page 996
Text summarization......Page 997
The question answering system......Page 1002
Genre categorization of web pages......Page 1003
Categorizing newspaper articles and newswires into topics......Page 1004
Web usage mining with web logs......Page 1007
Time Series Analysis......Page 1015
Multivariate time series analysis......Page 1016
Volatility modeling......Page 1031
References and reading list......Page 1044
Arbitrage pricing theory......Page 1046
Modeling in R......Page 1050
References......Page 1064
Motivation......Page 1066
The intensity of trading......Page 1067
The volume forecasting model......Page 1068
Implementation in R......Page 1070
References......Page 1083
Big Data – Advanced Analytics......Page 1084
Getting data from open sources......Page 1085
Introduction to big data analysis in R......Page 1090
K-means clustering on big data......Page 1091
Big data linear regression analysis......Page 1096
_GoBack......Page 1099
Terminology and notations......Page 1100
Currency options......Page 1103
Exchange options......Page 1106
Quanto options......Page 1116
References......Page 1122
Interest Rate Derivatives
and Models......Page 1124
The Black model......Page 1125
OLE_LINK2......Page 1131
The Cox-Ingersoll-Ross model......Page 1137
__DdeLink__1761_1471499930......Page 1141
Using the SMFI5 package......Page 1143
References......Page 1144
A general pricing approach......Page 1146
How R can help a lot......Page 1147
A glance beyond vanillas......Page 1148
Greeks – the link back to the vanilla world......Page 1154
Pricing the Double-no-touch option......Page 1157
Another way to price the
Double-no-touch option......Page 1169
The life of a Double-no-touch option – a simulation......Page 1171
Exotic options embedded in structured products......Page 1178
References......Page 1185
Hedging of derivatives......Page 1188
Hedging in the presence of transaction costs......Page 1201
Further extensions......Page 1209
References......Page 1210
The basics of fundamental analysis......Page 1212
Collecting data......Page 1214
Revealing connections......Page 1218
Including multiple variables......Page 1219
Separating investment targets......Page 1220
Setting classification rules......Page 1226
Backtesting......Page 1228
Industry-specific investment......Page 1232
References......Page 1237
Technical Analysis, Neural Networks, and Logoptimal Portfolios......Page 1238
Technical analysis......Page 1239
Neural networks......Page 1254
Logoptimal portfolios......Page 1260
References......Page 1266
Asset and Liability Management......Page 1268
Data preparation......Page 1269
Interest rate risk measurement......Page 1278
Liquidity risk measurement......Page 1282
Modeling non-maturity deposits......Page 1284
References......Page 1294
Capital Adequacy......Page 1296
Principles of the Basel Accords......Page 1297
Risk measures......Page 1303
Risk categories......Page 1310
References......Page 1324
Systemic risk in a nutshell......Page 1326
The dataset used in our examples......Page 1328
Core-periphery decomposition......Page 1330
The simulation method......Page 1334
Possible interpretations and suggestions......Page 1343
References......Page 1344
Introducing Machine Learning......Page 1350
The origins of machine learning......Page 1351
Uses and abuses of machine learning......Page 1353
How machines learn......Page 1358
Machine learning in practice......Page 1365
Machine learning with R......Page 1372
Managing and
Understanding Data......Page 1376
R data structures......Page 1377
Managing data with R......Page 1388
Exploring and understanding data......Page 1391
Lazy Learning – Classification Using Nearest Neighbors......Page 1414
_GoBack......Page 1415
Example – diagnosing breast cancer with the k-NN algorithm......Page 1425
Probabilistic Learning – Classification Using
Naive Bayes......Page 1438
Understanding Naive Bayes......Page 1439
Example – filtering mobile phone spam with the Naive Bayes algorithm......Page 1452
Divide and Conquer – Classification Using Decision Trees and Rules......Page 1476
Understanding decision trees......Page 1477
Example – identifying risky bank loans using C5.0 decision trees......Page 1487
Understanding classification rules......Page 1500
Example – identifying poisonous mushrooms with rule learners......Page 1511
Forecasting Numeric Data – Regression Methods......Page 1522
Understanding regression......Page 1523
Example – predicting medical expenses using linear regression......Page 1537
Understanding regression trees and model trees......Page 1553
Example – estimating the quality of wines with regression trees and model trees......Page 1556
_GoBack......Page 1572
Understanding neural networks......Page 1573
Example – Modeling the strength of concrete with ANNs......Page 1585
Understanding Support Vector Machines......Page 1592
Example – performing OCR with SVMs......Page 1601
Finding Patterns – Market Basket Analysis Using Association Rules......Page 1612
Understanding association rules......Page 1613
Example – identifying frequently purchased groceries with association rules......Page 1619
Finding Groups of Data – Clustering with k-means......Page 1638
Understanding clustering......Page 1639
Example – finding teen market segments using k-means clustering......Page 1649
_GoBack......Page 1664
Measuring performance for classification......Page 1665
Estimating future performance......Page 1689
_GoBack......Page 1700
Tuning stock models for better performance......Page 1701
Improving model performance with
meta-learning......Page 1712
Specialized Machine Learning Topics......Page 1730
Working with proprietary files and databases......Page 1731
Working with online data and services......Page 1734
Working with domain-specific data......Page 1745
Improving the performance of R......Page 1751
_GoBack......Page 1770
Module 2: R Graphs......Page 1772
Module 4: Mastering R for Quantitative Finance......Page 1773
Module 5: Machine Learning with R......Page 1774
Bibliography......Page 1780
_GoBack......Page 90
_GoBack......Page 166
_GoBack......Page 238
_GoBack......Page 269
_GoBack......Page 285
_GoBack......Page 303
_GoBack......Page 349
_GoBack......Page 385
_GoBack......Page 1112
_GoBack......Page 1177
_GoBack......Page 1199
_GoBack......Page 1264
_GoBack......Page 1317
Acerbi......Page 1325
_GoBack......Page 1327
_GoBack......Page 1373
__DdeLink__2136_2072555086......Page 1448
__DdeLink__2314_2072555086......Page 1460
__DdeLink__2318_2072555086......Page 1461
__DdeLink__2407_2072555086......Page 1465
__DdeLink__2409_2072555086......Page 1468
__DdeLink__2493_2072555086......Page 1469
_GoBack......Page 1635
_GoBack......Page 1662
_GoBack......Page 1764