Practical Computing and Bioinformatics for Conservation and Evolutionary Genomics

List of Tables
List of Figures
Preface
Introduction
Eric's Notes of what he might do
Table of topics
I Part I: Essential Computing Skills
Overview of Essential Computing Skills
Essential Unix/Linux Terminal Knowledge
Getting a bash shell on your system
Navigating the Unix filesystem
Changing the working directory with cd
Updating your command prompt
TAB-completion for paths
Listing the contents of a directory with ls
Globbing
What makes a good file-name?
The anatomy of a Unix command
The command
The options
Arguments
Getting information about Unix commands
Handling, Manipulating, and Viewing files and streams
Creating new directories
Fundamental file-handling commands
``Viewing'' Files
Redirecting standard output: > and >>
stdin, < and |
stderr
Symbolic links
File Permissions
Editing text files at the terminal
Customizing your Environment
Appearances matter
Where are my programs/commands at?!
A Few More Important Keystrokes
A short list of additional useful commands.
Two important computing concepts
Compression
Hashing
Unix: Quick Study Guide
Shell programming
An example script
The Structure of a Bash Script
A bit more on ; and &
Variables
Assigning values to variables
Accessing values from variables
What does the shell do with the value substituted for a variable?
Double and Single Quotation Marks and Variable Substitution
One useful, fancy, variable-substitution method
Integer Arithmetic with Shell Variables
Variable arrays
Evaluate a command and substitute the result on the command line
Grouping/Collecting output from multiple commands: (commands) and { commands; }
Exit Status
Combinations of exit statuses
Loops and repetition
More Conditional Evaluation: if, then, else, and friends
Finally…positional parameters
basename and dirname two useful little utilities
bash functions
reading files line by line
Further reading
Sed, awk, and regular expressions
awk
Line-cycling, tests and actions
Column splitting, fields, -F, $, NF, print, OFS and BEGIN
A brief introduction to regular expressions
A variety of tests
Code in the action blocks
Using awk to assign to shell variables
Passing Variables into awk with -v
Writing awk scripts in files
sed
Working on remote servers
Accessing remote computers
Windows
Hummingbird
Summit
Sedna
Transferring files to remote computers
sftp (via lftp)
git
Globus
Interfacing with ``The Cloud''
Getting files from a sequencing center
tmux: the terminal multiplexer
An analogy for how tmux works
First steps with tmux
Further steps with tmux
tmux for Mac users
Installing Software on an HPCC
Modules
Miniconda
Installing Java Programs
vim: it's time to get serious with text editing
Using neovim and Nvim-R and tmux to use R well on the cluster
High Performance Computing Clusters (HPCC's)
An oversimplified, but useful, view of a computing cluster
Cluster computing and the job scheduler
Learning about the resources on your HPCC
Getting compute resources allocated to your jobs on an HPCC
Interactive sessions
Batch jobs
SLURM Job Arrays
PREPATION INTERLUDE: An in-class exercise to make sure everything is configured correctly
More Boneyard…
The Queue (SLURM/SGE/UGE)
Modules package
Compiling programs without admin privileges
Job arrays
Writing stdout and stderr to files
Breaking stuff down
II Part II: Reproducible Research Strategies
Introduction to Reproducible Research
Rstudio and Project-centered Organization
Organizing big projects
Using RStudio in workflows with remote computers and HPCCs
Keeping an RStudio project ``in sync'' with GitHub
Evaluating scripts line by line on a remote machine from within RStudio
Version control
Why use version control?
How git works
git workflow patterns
using git with Rstudio
git on the command line
A fast, furious overview of the tidyverse
Authoring reproducibly with Rmarkdown
Notebooks
References
Zotero and Rmarkdown
Bookdown
Google Docs
Using python
III Part III: Bioinformatic Analyses
Overview of Bioinformatic Analyses
DNA Sequences and Sequencing
DNA Stuff
DNA Replication with DNA Polymerase
The importance of the 3' hydroxyl…
Sanger sequencing
Illumina Sequencing by Synthesis
Library Prep Protocols
WGS
RAD-Seq methods
Amplicon Sequencing
Capture arrays, RAPTURE, etc.
Bioinformatic file formats
Sequences
FASTQ
Line 1: Illumina identifier lines
Line 4: Base quality scores
A FASTQ `tidyverse' Interlude
Comparing read 1 to read 2
FASTA
Genomic ranges
Extracting genomic ranges from a FASTA file
Downloading reference genomes from NCBI
Alignments
How might I align to thee? Let me count the ways…
Play with simple alignments
SAM Flags
The CIGAR string
The SEQ and QUAL columns
SAM File Headers
The BAM format
Quick self study
Variants
VCF Format – The Body
VCF Format – The Header
Boneyard
Segments
Conversion/Extractions between different formats
Visualization of Genomic Data
Sample Data
Genome Assembly
Alignment of sequence data to a reference genome (and associated steps)
The Journey of each DNA Fragment from Organism to Sequencing Read
Read Groups
Aligning reads with bwa
Indexing the genome for alignment
Mapping reads with bwa mem
Hold it Right There, Buddy! What about the Read Groups?
Processing alignment output with samtools
samtools subcommands
BONEYARD BELOW HERE
Preprocess ?
Quick notes to self on chaining things:
Merging BAM files
Divide and Conquer Strategies
Variant calling
Genotype Likelihoods
Basic Sketch of Genotype Likelihood Calculations
Specifics of different genotype likelihoods
Computing genotype likelihoods with three different softwares
A Directed Acyclic Graph For Genotype Likelihoods
Boneyard
Basic Handling of VCF files
bcftools
Tell me about my VCF file!
Get fragments/parts of my VCF file
Combine VCF files in various ways
Filter out variants for a variety of reasons
Bioinformatics for RAD seq data with and without a reference genome
Processing amplicon sequencing data
Genome Annotation
Whole genome alignment strategies
Mapping of scaffolds to a closely related genome
Obtaining Ancestral States from an Outgroup Genome
Using LASTZ to align coho to the chinook genome
Try on the chinook chromosomes
Explore the other parameters more
IV Part IV: Analysis of Big Variant Data
Bioinformatic analysis on variant data
V Part V: Population Genomics
Topics in pop gen
Coalescent
Measures of genetic diversity and such
Demographic inference with \partial a \partial i and moments
Balls in Boxes
Some landscape genetics
Relationship Inference
Tests for Selection
Multivariate Associations, GEA, etc.
Estimating heritability in the wild
Bibliography