Big Data Analytics in Earth, Atmospheric and Ocean Sciences

Cover
Title Page
Copyright
Contents
List of Contributors
Preface
Chapter 1 An Introduction to Big Data Analytics
1.1 Overview
1.1.1 What Differentiates Spatial Big Data
1.2 Definitions
1.3 Example Problems
1.3.1 Agriculture
1.3.2 Commerce
1.3.3 Connected Cars
1.3.4 Environment
1.3.5 Financial Services
1.3.6 Government Agencies
1.3.7 Health Care
1.3.8 Marketing
1.3.9 Mining
1.3.10 Petroleum
1.3.11 Retail
1.3.12 Telecommunications
1.3.13 Transportation
1.3.14 Utilities
1.4 Big Data Analysis Concepts
1.4.1 Summarizing Data
1.4.2 Identify Locations
1.4.3 Pattern Analysis
1.4.4 Cluster Analysis
1.4.5 Proximity Analysis
1.4.6 Predictive Modeling
1.5 Technology and Tools
1.5.1 Available Tools
1.6 Challenges
1.7 Summary
References
Part I Big Data Analytics Architecture
Chapter 2 Introduction to Big Data Analytics Architecture
References
Chapter 3 Scaling Big Earth Science Data Systems Via Cloud Computing
3.1 Introduction
3.2 Key Concepts of Science Data Systems (SDSes)
3.3 Increasing Data Processing, Volumes, and Rates
3.3.1 Historical Example
3.3.2 Example of On‐Premises SDS
3.3.3 Exceeding On‐Premise Capacities
3.4 Cloud Concepts for SDSes
3.4.1 IaaS (Infrastructure as a Service)
3.4.2 PaaS (Platform as a Service)
3.4.3 SaaS (Software as a Service)
3.4.4 XaaS (Everything as a Service)
3.5 Architecture Components of Cloud‐Based SDS
3.5.1 Algorithm Development Environment
3.5.2 Processing Algorithm Catalog
3.5.3 Resource Management
3.5.4 Processing Orchestration/Workflow Management
3.5.5 Compute Services
3.5.6 Data Catalog
3.5.7 Data Storage Services
3.5.8 Common Services for Logging, Metrics, Events, and Analytics
3.5.9 Integrating Science Data Processing and Algorithm Development, Software Catalog, and Analysis
3.6 Considerations for Multi‐cloud and Hybrid SDS
3.6.1 Collocation of GDS, SDS, and DAAC
3.6.2 All‐In and Lock‐In
3.7 Cloud Economics
3.8 Large‐Scaling Considerations
3.8.1 Metrics for Anomalies
3.8.2 Thundering Herd
3.8.3 Higher SLA
3.8.4 Watchdogs
3.9 Example of Cloud SDSes
3.9.1 SMAP in the Cloud
3.9.2 NISAR SDS
3.10 Conclusion
3.10 References
Chapter 4 NOAA Open Data Dissemination (Formerly NOAA Big Data Project/Program)
4.1 Obstacles to the Public's Use of NOAA Environmental Data
4.2 Public Access of NOAA Data Creates Challenges for the Agency
4.3 The Vision for NOAA's “Oddball” Approach to Big Data
4.4 A NOAA Cooperative Institute Data Broker provides Research and Operational Agility
4.5 Public‐Private Partnerships Provide the Pipeline
4.6 BDP Exceeds Expectations and Evolves into Enterprise Operations
4.7 Engaging Users in the Cloud
4.7.1 Early Insights From User Engagement
4.7.2 Data Analytics and Metrics Informing User Engagement
4.7.3 NODD Supports Industry Challenges in Sustainability
4.7.4 Advancing User Engagement for NODD
4.8 Challenges and Opportunities
4.8.1 Format Conversions and Cloud‐Based Tools
4.8.2 Attention to Data Quality and Provenance
4.9 Vision for the Future
4.9 Acknowledgments
4.9 References
Chapter 5 A Data Cube Architecture for Cloud‐Based Earth Observation Analytics
5.1 Introduction
5.1.1 The Open Data Cube (ODC) Architecture
5.2 Open Data Cube for the Cloud Design
5.2.1 Storage Model
5.2.2 The ODC S3 Native Storage Driver
5.2.3 S3 Array Structure
5.2.4 Implementation of the S3 Array I/O Module
5.2.5 Execution Model
5.2.6 Execution Engine
5.3 S3 Array I/O Performance
5.3.1 Experiment Setup
5.3.2 Raw S3 Read/Write Performance
5.3.3 Data Cube Ingest Scaling (Write)
5.3.4 Data Cube Load Scaling (Read)
5.4 Discussion and Conclusion
5.4.1 S3AIO Advantages
5.4.2 S3AIO Limitations
5.4.3 Next Steps for This Research
5.4 References
Chapter 6 Open Source Exploratory Analysis of Big Earth Data With NEXUS
6.1 Introduction
6.1.1 Cloud Computing
6.1.2 MapReduce Programming Model
6.2 Architecture
6.3 Deployment Architecture
6.4 Benchmarking and Studies
6.4.1 Hurricane Katrina Case Study
6.5 Analytics Collaborative Framework
6.6 Federated Analytics Collaborative Systems
6.7 Conclusion
6.7 References
Chapter 7 Benchmark Comparison of Cloud Analytics Methods Applied to Earth Observations
7.1 Introduction
7.2 Experimental Setup
7.3 AODS Candidates
7.4 Experimental Results
7.5 Conclusions
7.5 References
Part II Analysis Methods for Big Earth Data
Chapter 8 Introduction to Analysis Methods for Big Earth Data
References
Chapter 9 Spatial Statistics for Big Data Analytics in the Ocean and Atmosphere: Perspectives, Challenges, and Opportunities
9.1 Spatial Data and Spatial Statistics
9.2 What Constitutes Big Spatial Data?
9.3 Statistical Implications of the Four Vs of Big Spatial Data
9.3.1 Volume
9.3.2 Variety
9.3.3 Velocity
9.3.4 Veracity
9.4 Challenges to the Statistical Analysis of Big Spatial Data
9.4.1 Randomness and Sampling
9.4.2 High Dimensionality
9.4.3 Independence of Samples and Spatial Autocorrelation
9.4.4 Effect Size
9.5 Opportunities in Spatial Analysis of Big Data
9.5.1 Ecological Marine Units
9.5.2 Spatiotemporal Analysis of California Maximum Temperature
9.6 Conclusion
9.6 References
Chapter 10 Giving Scientists Back Their Flow: Analyzing Big Geoscience Data Sets in the Cloud
10.1 Introduction
10.2 Where's the Opportunity?
10.2.1 Progression of Typical Approaches Over the Past 5 Years
10.2.2 On Premises Approaches
10.2.3 Off‐Premises Solutions: Cloud Computing
10.3 The Future
10.3 Reference
Chapter 11 The Distributed Oceanographic Match‐Up Service
11.1 Introduction
11.2 DOMS Capabilities
11.3 System Architecture
11.3.1 Data Sets
11.3.2 In Situ Data
11.3.3 Satellite Data
11.3.4 User Interface
11.4 Workflow
11.4.1 Match‐Up Algorithms
11.4.2 DOMS API
11.4.3 DOMS File Output
11.4.4 User Interface
11.5 Future Development
11.5 Acknowledgments
11.5 Availability Statement
11.5 References
Part III Big Earth Data Applications
Chapter 12 Introduction to Big Earth Data Applications
References
Chapter 13 Topological Methods for Pattern Detection in Climate Data
13.1 Introduction
13.2 Topological Methods for Pattern Detection
13.2.1 Step 1: Topological Feature Descriptors of Weather Patterns
13.2.2 Step 2: Machine Learning for Classifying Weather Patterns
13.3 Case Study: Atmospheric Rivers Detection
13.3.1 Atmospheric Rivers
13.3.2 Data
13.3.3 Results
13.4 Conclusions and Recommendations
13.4 Acknowledgments
13.4 References
Chapter 14 Exploring Large Scale Data Analysis and Visualization for Atmospheric Radiation Measurement Data Discovery Using NoSQL Technologies
14.1 Introduction
14.2 Software and Workflow
14.2.1 Software
14.2.2 Workflow
14.3 Hardware Architecture
14.4 Applications
14.4.1 LASSO Bundle Browser
14.4.2 ARMBE Visualizations
14.4.3 Data Analytics
14.5 Conclusions
14.5 Acknowledgments
14.5 References
Chapter 15 Demonstrating Condensed Massive Satellite Data Sets for Rapid Data Exploration: The MODIS Land Surface Temperatures of Antarctica
15.1 Introduction
15.2 Data
15.3 Methods
15.3.1 Data Set Cleaning
15.3.2 Baseline Statistics Generation
15.3.3 Anomaly Determination
15.3.4 Database Storage
15.4 Results
15.4.1 Efficiency and Performance
15.4.2 Cloud Masking Quality
15.4.3 Coldest Temperature
15.4.4 Spurious Warm Observations
15.5 Conclusions
15.5 Acknowledgments
15.5 Availability Statement
15.5 References
Chapter 16 Developing Big Data Infrastructure for Analyzing AIS Vessel Tracking Data on a Global Scale
16.1 Introduction
16.2 Background
16.3 Use Case: Producing Heat Maps of Vessel Traffic using AIS Data
16.3.1 Overview
16.3.2 Data Preparation
16.4 Data Processing Overview
16.4.1 Data Processing Steps
16.4.2 Data Processing: Results
16.4.3 Data Curation and Open Access
16.5 Future Work
16.6 Conclusions
16.6 References
Chapter 17 Future of Big Earth Data Analytics
17.1 Introduction
17.2 How Data Get Bigger
17.3 The Evolution of Analytics Algorithms
17.4 Analytics Architectures
17.5 Conclusions
17.5 References
Index
EULA