Data Preprocessing
Preprocessing and SNP calling
Natasja S. Ehlers, PhD student
Center for Biological Sequence Analysis
Functional Human Variation Group
Next Generation Sequencing analysis
DTU Bioinformatics
36626 - Next Generation Sequencing Analysis
Generalized NGS analysis
Data size
Application
Assembly: Compare
Raw Pre- specific:
Question Alignment / samples / Answer?
reads processing Variant calling,
de novo methods
count matrix, ...
36626 - Next Generation Sequencing Analysis
Generalized NGS analysis
Data size
Application
Assembly: Compare
Raw Pre- specific:
Question Alignment / samples / Answer?
reads processing Variant calling,
de novo methods
count matrix, ...
36626 - Next Generation Sequencing Analysis
Assembly: Two basic approaches
• Alignment: Use a reference genome and align your
reads to the genome
• de novo assembly: Try to assemble the reads into a
genome without any prior knowledge
36626 - Next Generation Sequencing Analysis
Assembly: Two basic approaches
Monday
• Alignment: Use a reference genome and align your
reads to the genome
Wednesday
• de novo assembly: Try to assemble the reads into a
genome without any prior knowledge
36626 - Next Generation Sequencing Analysis
Assembly: Two basic approaches
Monday
• Alignment: Use a reference genome and align your
reads to the genome
Wednesday
• de novo assembly: Try to assemble the reads into a
genome without any prior knowledge
But first a look at data preprocessing
36626 - Next Generation Sequencing Analysis
Preprocessing
• Reads have qualities - bases are not always correct!
• Different error profiles pr. technology
• What can we do?
• Quality trimming
• Adaptor clipping
• 5’ clipping
• k-mer correction
• ...
36626 - Next Generation Sequencing Analysis
Analyze data using FastQC
• Report basic statistics on your
data
• Identify issues with your data
36626 - Next Generation Sequencing Analysis
Per base sequence quality
Illumina
Trim from 3’ to qual 20
36626 - Next Generation Sequencing Analysis
Average quality
Illumina
Remove reads with
avg. qual < 20
Remove reads with
“N” basecalls
36626 - Next Generation Sequencing Analysis
Trim from 5’
• Sometimes something is fishy in the beginning of the
read
Clip a certain number of bases from 5’
36626 - Next Generation Sequencing Analysis
Adapters
• Sometimes adapters/primers are also part of the read
• Adapter/primers are non-biological sequences
• Short read alignment is global - adapters are no-go
• de novo assembly will be confused ~ artificial repeats
• If you dont know which were used: FastQC will (may) find
them for you!
36626 - Next Generation Sequencing Analysis
Adapters - example
We will use “Cutadapt” and “AdapterRemoval” to cut adapters,
many other options exist
Very important if your DNA fragment is shorter than read length
36626 - Next Generation Sequencing Analysis
454 / ion torrent data
Prinseq output
• Main problem is indels at
homopolymer runs
• (Trim homopolymers), trim trailing
poor quality bases
• Remove very short reads
• For de novo adapters should be
removed (prinseq)
• For alignment we use Smith-
Waterman (local) so less important
36626 - Next Generation Sequencing Analysis
k-mer correction
• What is a k-mer?
• Create a sliding window of size k, move it over all
your reads and count occurrence of k-mers
• We can use this to correct sequencing errors!
DNA: ACGTGTAACGTGACGTTGGA
ACGTG
Eg. k=5 CGTGT
GTGTA
36626 - Next Generation Sequencing Analysis
k-mer correction
Page 9 of 13
mer
Concept: Rare k-mers are seq. errors
0.015
rse
Need >15X coverage
na Error k-mers
the
0.010
uch
True k-mers
ACGTGGTTGCCCTTAAA
ACGTGGTTACCCTTAAA
Density
ACGTGGTTACCCTTAAA
(2)
ACGTGGTTACCCTTAAA
0.005
ACGTGGTTACCCTTAAA
ACGTGGTTACCCTTAAA
ACGTGGTTACCCTTAAA
(3) ACGTGGTTACCCTTAAA
ACGTGGTTACCCTTAAA
0.000
et k
me, 0 20 40 60 80 100
ea- Coverage
s in Figure 3 k-mer coverage. 15-mer coverage model fit to 76×
of coverage of 36 bp reads from E. coli. Note that the expected
36626 - coverage
ich of a k-mer
Next Generation
L −k +1
in the genome
Sequencing Analysis using reads of length L will be Kelley et al., 2010
times the expected coverage of a single nucleotide
Merge paired ends
Insert size: 500nt Insert size: 180nt
Reads: 100nt Reads: 100nt
Middle: 300nt Middle: -20nt
• Merge overlapping pairs: single longer read
• Smart because Illumina reads have bad 3’ quals
• Very useful for de novo assembly
36626 - Next Generation Sequencing Analysis Magocˇ and Salzberg, 2011
Merge paired ends
Overlap
Insert size: 500nt Insert size: 180nt
Reads: 100nt Reads: 100nt
Middle: 300nt Middle: -20nt
• Merge overlapping pairs: single longer read
• Smart because Illumina reads have bad 3’ quals
• Very useful for de novo assembly
36626 - Next Generation Sequencing Analysis Magocˇ and Salzberg, 2011
overage Coverage
• Coverage/depth is how many times that your data covers the genome
(on average)
• Example: L
• N: Number of reads: 5 mill C = N ⇥
G
• L: Read length: 100
• G: Genome size: 5 Mbases
•
G OnC = 5*100/5 = 100X
: genome size
• average there are 100 reads covering each position in the genome
N : number of reads
36626 - Next Generation Sequencing Analysis
Last, but important!
• Lots of data - storage is expensive!
• Keep data compressed whenever
possible (gzip, bzip, bam)
• Remove intermediate files and files
that can easily be re-created
36626 - Next Generation Sequencing Analysis