Master Data Science and Data Analysis with Pandas

Table Of content
1Introduction
2Advantages
2.1 Speed
2.2 Short code:
2.3 Saves time:
2.4 Easy:
2.5 Versatile:
2.6 Efficiency:
2.7 Customizable:
2.8 Supports multiple formats for I/O:
2.9 Python support:
3Installation
3.1 Install Pandas
3.1.1Installing with Anaconda
3.1.2Installing with PyPI
3.2Install Jupyter Notebook
4Creating DataFrames
4.1Creating DataFrame using dictionary data
4.2Creating DataFrame using list data
4.2.1Creating single column
4.2.2Creating multiple columns
4.3Adding custom column name
4.4creating DataFrame from list of dictionaries
4.5creating DataFrame from other files
4.6Creating blank DataFrame
5Basics of DataFrames
5.1Read data
5.2Shape of the DataFrame
5.3Top ‘n’ rows
5.4Last ‘n’ rows
5.5Range of entries
5.6Accessing the columns
5.7Accessing ‘n’ columns
5.8Type of column
5.9Basic operations on column
5.9.1maximum
5.9.2minimum
5.9.3mean
5.9.4standard deviation
5.10Describe the DataFrame
5.11Conditional operation on columns
5.12accessing row with loc and iloc
5.13Set index
6Reading and writing files
6.1Reading CSV
6.1.1Reading
6.1.2Removing header
6.1.3Adding custom header
6.1.4Reading specific rows
6.1.5Reading the data from specific row
6.1.6Cleaning NA data
6.1.7Reference
6.2Writing to CSV
6.2.1Avoiding index
6.2.2Writing only specific columns
6.2.3Avoid writing headers
6.2.4reference
6.3Read Excel
6.3.1Reading sheets
6.3.2Passing function to columns
6.3.3Some basic common functions in read_excel and read_csv are:
6.3.4Reference
6.4Writing excels
6.4.1Writing to a custom sheet name
6.4.2Avoid index
6.4.3Avoid headers
6.4.4write at a particular row and column
6.4.5Writing multiple sheets to the same excel file
6.4.6reference
6.5Reading and writing txt file
6.5.1Reading txt
6.5.2Writing to txt
6.6Reference
6.6.1https://Pandas.pydata.org/Pandas-docs/stable/user_guide/io.html
7working with missing data
7.1Managing timestamp
7.2Set index
7.3Check if data is “na” or “notna”
7.4Check if a data has missing datetime
7.5Inserting missing date
7.6Filling the missing data
7.6.1Filling a common value to all missing data
7.6.2Adding missing data to individual columns
7.6.3Forward fill (row)
7.6.4Backward fill (row)
7.6.5Forward fill (column)
7.6.6Limiting the forward/backward fill
7.6.7Filling with Pandas objects
7.6.8Filling for specific range of columns
7.7Interpolate missing value
7.7.1Linear interpolate
7.7.2Time interpolate
7.7.3Other methods of interpolation
7.7.4Limiting the interpolation
7.7.5Interpolation direction
7.7.6Limit area of interpolation
7.8Drop the missing value
7.8.1Drop row with at least 1 missing value
7.8.2Drop row with all missing values
7.8.3Set threshold to drop
7.9Replace the data
7.9.1Replace a column with new column
7.9.2Replace with mapping dictionary
7.9.3replacing value with NaN
7.9.4Replace multiple values with NaN
7.9.5Replacing data as per columns
7.9.6Regex and replace
7.9.7Regex on specific columns
8Groupby
8.1Creating group object
8.2Simple operations with group
8.2.1First
8.2.2Last
8.2.3Max
8.2.4Min
8.2.5Mean
8.3Working of groupby
8.4Iterate through groups
8.4.1Group details
8.4.2Iterate for groups
8.5Get a specific group
8.6Detailed view of the groups data
8.7Group by sorting
8.7.1Sorted data (default)
8.7.2Unsorted data
8.8Various functions associated with groupby object
8.9Length
8.9.1Len of an object
8.9.2Length of each group
8.10Groupby with multi-index
8.10.1Grouping on level numbers
8.10.2grouping on level names
8.11Grouping DataFrame with index level and columns
8.12Aggregation
8.12.1Applying multiple aggregate functions at once
8.12.2Multiple aggregate function to selected columns
8.12.3Renaming the column names for aggregate functions
8.12.4Named aggregation
8.12.5Custom agg function on various columns
8.13Transformation
8.13.1Custom functions in transformation
8.13.2Filling missing data
8.14Window operations
8.14.1Rolling
8.14.2Expanding
8.15Filtration
8.16Instance methods
8.16.1Sum, mean, max, min etc
8.16.2Fillna
8.16.3Fetching nth row
8.17Apply
8.18Plotting
8.18.1Lineplot
8.18.2Boxplot
9Concatenation
9.1Concatenate series
9.2Concatenate DataFrames
9.3Managing duplicate index
9.4Adding keys to DataFrames
9.5Use of keys
9.6Adding DataFrame as a new column
9.6.1Removing unwanted columns in column concatenation
9.6.2Series in columns
9.7Rearranging the order of column
9.8Join DataFrame and series
9.9Concatenating multiple DataFrames /series
10Merge
10.1Merging DataFrames
10.2Merging different values of “ON” (joining) column
10.2.1Merging with Inner join
10.2.2Merging with outer join
10.2.3Merging with left join
10.2.4Merging with right join
10.3Knowing the source DataFrame after merge
10.4Merging DataFrames with same column names
10.5Other ways of joining
10.5.1Join
10.5.2Append
11Pivot
11.1Multilevel columns
11.2Data for selected value (column)
11.3Error from duplicate values
12Pivot table
12.1Aggregate function
12.1.1List of function to aggfunc
12.1.2Custom functions to individual columns
12.2Apply pivot_table() on desired columns
12.3Margins
12.3.1Naming the margin column
12.4Grouper
12.5Filling the missing value in pivot table
13Reshape DataFrame using melt
13.1Use of melt
13.2Melt for only one column
13.3Melt multiple columns
13.4Custom column name
13.4.1Custom variable name
13.4.2Custom value name
14Reshaping using stack and unstack
14.1Stack the DataFrame
14.2Stack custom level of column
14.3Stack on multiple levels of column
14.4Dropping missing values
14.5Unstack the stacked DataFrame
14.5.1Default unstack
14.5.2Converting other index levels to column
14.5.3Unstack multiple indexes
15Frequency distribution of DataFrame column
15.1Apply crosstab
15.2Get total of rows/columns
15.3Multilevel columns
15.4Multilevel indexes
15.5Custom name to rows/columns
15.6Normalize (percentage) of the frequency
15.7Analysis using custom function
16Drop unwanted rows/columns
16.1Delete row
16.1.1Delete rows of custom index level
16.1.2Delete multiple rows
16.2Drop column
16.2.1Delete multiple columns
16.2.2Delete multilevel columns
16.3Delete both rows & columns
17Remove duplicate values
17.1Remove duplicate
17.2Fetch custom occurrence of data
17.2.1First occurrence
17.2.2Last occurrence
17.2.3Remove all duplicates
17.3Ignore index
18Sort the data
18.1Sort columns
18.2Sorting multiple columns
18.3Sorting order
18.4Positioning missing value
19Working with date and time
19.1Creation, working and use of DatetimeIndex
19.1.1Converting date to timestamp and set as index
19.1.2Access data for particular year
19.1.3Access data for particular month
19.1.4Calculating average closing price for any month
19.1.5Access a date range
19.1.6Resampling the data
19.1.7Plotting the resampled data
19.1.8Quarterly frequency
19.2Working with date ranges
19.2.1Adding dates to the data
19.2.2Apply the above date range to our data
19.2.3Generate the missing data with missing dates
19.2.4Date range with periods
19.3Working with custom holidays
19.3.1Adding US holidays
19.3.2Creating custom calendar
19.3.3Observance rule
19.3.4Custom week days
19.3.5Custom holiday
19.4Working with date formats
19.4.1Converting to a common format
19.4.2Time conversion
19.4.3Dayfirst formats
19.4.4Remove custom delimiter in date
19.4.5Remove custom delimiter in time
19.4.6Handling errors in datetime
19.4.7Epoch time
19.5Working with periods
19.5.1Annual period
19.5.2Monthly period
19.5.3Daily period
19.5.4Hourly period
19.5.5Quarterly period
19.5.6Converting one frequency to another
19.5.7Arithmetic between two periods
19.6Period Index
19.6.1Getting given number of periods
19.6.2Period index to DataFrame
19.6.3Extract annual data
19.6.4Extract a range of periods data
19.6.5Convert periods to datetime index
19.6.6Convert DatetimeIndex to PeriodIndex
19.7Working with time zones
19.7.1Make naïve time to time zone aware
19.7.2Available timezones
19.7.3Convert on time zone to other
19.7.4Time zone in a date range
19.7.5Time zone with dateutil
19.8Data shifts in DataFrame
19.8.1Shifting the price down
19.8.2Shifting by multiple rows
19.8.3Reverse shifting
19.8.4Use of shift
19.8.5DatetimeIndex shift
19.8.6Reverse DatetimeIndex shift
20Database
20.1Working with MySQL
20.1.1Installations
20.1.2Create connection
20.1.3Read table data
20.1.4Fetching specific columns from table
20.1.5Execute a query
20.1.6Insert data to table
20.1.7Common function to read table and query
20.2Working with MongoDB
20.2.1Installations
20.2.2Create connection
20.2.3Get records
20.2.4Fetching specific columns
20.2.5Insert records
20.2.6Delete records
21About Author