This book presents the process and framework you need to transform aspects of our world into data that can be collected, analyzed, and used to make decisions. You will understand the technologies used to gather and process data from many sources, and you will learn how to analyze data with AI and ML models.
Datafication is becoming increasingly prevalent in many areas of our lives, from business to education and healthcare. It has the potential to improve decision-making by providing insights into patterns, trends, and correlation between seemingly unconnected pieces of data. This book explains the evolution, principles, and patterns of datafication used in our day-to-day activities. It covers how to collect data from a variety of sources, using technologies such as edge, streaming techniques, REST, and frameworks, as well as data cleansing and data lineage. A data analysis framework is provided to guide you in designing and developing AI and ML projects, including the details of sentiment and behavioral analytics.
Introduction to Datafication teaches you how to engineer AI and ML projects by using various methodologies, covers the security mechanisms to be applied for datafication, and shows you how to govern the datafication process with a well-defined governance framework.
What You Will Learn:
Understand the principles and patterns to be adopted for datafication
Gain techniques for sourcing and mining data, and for sharing data with a data pipeline
Leverage the AI and ML algorithms most suitable for datafication
Understand the data analysis framework used in every AI and ML project
Master the details of sentiment and behavioral analytics through practical examples
Utilize development methodologies for datafication engineering and the related security and governance framework
Who This Book Is For:
Students, data scientists, data analysts, and AI and ML engineers.
Author(s): Shivakumar R. Goniwada
Publisher: Apress
Year: 2023
Language: English
Pages: 288
Chapter 1: Introduction to Datafication
What Is Datafication?
Why Is Datafication Important?
Data for Datafication
Datafication Steps
Digitization vs. Datafication
Types of Data in Datafication
Elements of Datafication
Data Harvesting
Data Curation
Data Storage
Data Analysis
Cloud Computing
Datafication Across Industries
Summary
Chapter 2: Datafication Principles and Patterns
What Are Architecture Principles?
Datafication Principles
Data Integration Principle
Data Quality Principle
Data Governance Principles
Data Is an Asset
Data Is Shared
Data Trustee
Ethical Principle
Security by Design Principle
Datafication Patterns
Data Partitioning Pattern
Data Replication
Stream Processing
Change Data Capture (CDC)
Data Mesh
Machine Learning Patterns
Summary
Chapter 3: Datafication Analytics
Introduction to Data Analytics
What Is Analytics?
Big Data and Data Science
Datafication Analytical Models
Content-Based Analytics
Data Mining
Text Analytics
Sentiment Analytics
Audio Analytics
Video Analytics
Comparison in Analytics
Datafication Metrics
Datafication Analysis
Data Sources
Data Gathering
Introduction to Algorithms
Supervised Machine Learning
Linear Regression
Support Vector Machines (SVM)
Decision Trees
Neural Networks
Naïve Bayes Algorithm
K-Nearest Neighbor (KNN) Algorithm
Random Forest
Unsupervised Machine Learning
Clustering
Association Rule Learning
Dimensionality Reduction
Reinforcement Machine Learning
Summary
Chapter 4: Datafication Data-Sharing Pipeline
Introduction to Data-Sharing Pipelines
Steps in Data Sharing
Data-Sharing Process
Data-Sharing Decisions
Data-Sharing Styles
Unidirectional, Asynchronous Push Integration Style
Real-Time and Event-based Integration Style
Bidirectional, Synchronous, API-led Integration Style
Mediated Data Exchange with an Event-Driven Approach
Designing a Data-Sharing Pipeline
Types of Data Pipeline
Batch Processing
Extract, Transform, and Load Data Pipeline (ETL)
Extract, Load, and Transform Data Pipeline (ELT)
Streaming and Event Processing
Change Data Capture (CDC)
Lambda Data Pipeline Architecture
Kappa Data Pipeline Architecture
Data as a Service (DaaS)
Data Lineage
Data Quality
Data Integration Governance
Summary
Chapter 5: Data Analysis
Introduction to Data Analysis
Data Analysis Steps
Prepare a Question
Prepare Cleansed Data
Identify a Relevant Algorithm
Build a Statistical Model
Match Result
Create an Analysis Report
Summary
Chapter 6: Sentiment Analysis
Introduction to Sentiment Analysis
Use of Sentiment Analysis
Types of Sentiment Analysis
Document-Level Sentiment Analysis
Aspect-Based Sentiment Analysis
Multilingual Sentiment Analysis
Pros and Cons of Sentiment Analysis
Pre-Processing of Data
Tokenization
Stop Words Removal
Stemming and Lemmatization
Handling Negation and Sarcasm
Rule-Based Sentiment Analysis
Lexicon-Based Approaches
Sentiment Dictionaries
Pros and Cons of Rule-Based Approaches
Machine Learning–Based Sentiment Analysis
Supervised Learning Techniques
Unsupervised Learning Techniques
Pros and Cons of the Machine Learning–Based Approach
Best Practices for Sentiment Analysis
Summary
Chapter 7: Behavioral Analysis
Introduction to Behavioral Analytics
Data Collection
Behavioral Science
Importance of Behavioral Science
How Behavioral Analysis and Analytics Are Processed
Cognitive Theory and Analytics
Biological Theories and Analytics
Integrative Model
Behavioral Analysis Methods
Funnel Analysis
Cohort Analysis
Customer Lifetime Value (CLV)
Churn Analysis
Behavioral Segmentation
Analyzing Behavioral Analysis
Descriptive Analysis with Regression
Causal Analysis with Regression
Causal Analysis with Experimental Design
Challenges and Limitations of Behavioral Analysis
Summary
Chapter 8: Datafication Engineering
Steps of AI and ML Engineering
AI and ML Development
Understanding the Problem to Be Solved
Choosing the Appropriate Model
Preparing and Cleaning Data
Feature Selection and Engineering
Model Training and Optimization
AI and ML Testing
Unit Testing
Integration Testing
Non-Functional Testing
Performance
Security Testing
DataOps
MLOps
Summary
Chapter 9: Datafication Governance
Importance of Datafication Governance
Why Is Datafication Governance Required?
Datafication Governance Framework
Oversight and Accountability
Model Risk, Risk Assessment, and Regulatory Guidance
Roles and Responsibilities
Monitoring and Reporting
Datafication Governance Guidelines and Principles
Ethical and Legal Aspects
Datafication Governance Action Framework
Datafication Governance Challenges
Summary
Chapter 10: Datafication Security
Introduction to Datafication Security
Datafication Security Framework
Regulations
Organization Concerns
Governance and Compliance
Business Access Needs
Datafication Security Measures
Encryption
Data Masking
Penetration Testing
Data Security Restrictions
Summary
Index
