Upgrading to R 3.2.0 on Windows

If you are using Windows you can easily upgrade to the latest version of R using the installr package. Simply run the following code:

# installing/loading the latest installr package:
install.packages("installr"); library(installr) #load / install+load installr
 
updateR() # updating R.

Running “updateR()” will detect if there is a new R version available, and if so it will download+install it (etc.).

If you are an R blogger yourself you are invited to add your own R content feed to this site (Non-English R bloggers should add themselves- here)

NEW FEATURES

• anyNA() gains a recursive argument.
• When x is missing and names is not false (including the default value), Sys.getenv(x, names) returns an object of class "Dlist" and hence prints tidily.
• (Windows.) shell() no longer consults the environment variable SHELL: too many systems have been encountered where it was set incorrectly (usually to a path where software was compiled, not where it was installed). R_SHELL, the preferred way to select a non-default shell, can be used instead.
• Some unusual arguments to embedFonts() can now be specified as character vectors, and the defaults have been changed accordingly.
• Functions in the Summary group duplicate less. (PR#15798)
• (Unix-alikes.) system(cmd, input = ) now uses ‘shell-execution-environment’ redirection, which will be more natural if cmd is not a single command (but requires a POSIX-compliant shell). (Wish of PR#15508)
• read.fwf() and read.DIF() gain a fileEncoding argument, for convenience.
• Graphics devices can add attributes to their description in .Device and .Devices. Several of those included with R use a "filepath" attribute.
• pmatch() uses hashing in more cases and so is faster at the expense of using more memory. (PR#15697)
• pairs() gains new arguments to select sets of variables to be plotted against each other.
• file.info(, extra_cols = FALSE) allows a minimal set of columns to be computed on Unix-alikes: on some systems without properly-configured caching this can be significantly faster with large file lists.
• New function dir.exists() in package base to test efficiently whether one or more paths exist and are directories.
• dput() and friends gain new controls hexNumeric and digits17 which output double and complex quantities as, respectively, binary fractions (exactly, see sprintf("%a")) and as decimals with up to 17 significant digits.
• save(), saveRDS() and serialize() now support ascii = NA which writes ASCII files using sprintf("%a") for double/complex quantities. This is read-compatible with ascii = TRUE but avoids binary->decimal->binary conversions with potential loss of precision. Unfortunately the Windows C runtime’s lack of C99 compliance means that the format cannot be read correctly there in R before 3.1.2.
• The default for formatC(decimal.mark =) has been changed to be getOption("OutDec"); this makes it more consistent with format() and suitable for use in print methods, e.g. those for classes "density", "ecdf", "stepfun" and "summary.lm".

  getOption("OutDec") is now consulted by the print method for class "kmeans", by cut(), dendrogram(), plot.ts() and quantile() when constructing labels and for the report fromlegend(trace = TRUE).

  (In part, wish of PR#15819.)

• printNum() and hence format() and formatC() give a warning if big.mark and decimal.mark are set to the same value (period and comma are not uncommonly used for each, and this is a check that conventions have not got mixed).
• merge() can create a result which uses long vectors on 64-bit platforms.
• dget() gains a new argument keep.source which defaults to FALSE for speed (dput() and dget() are most often used for data objects where this can make dget() many times faster).
• Packages may now use a file of common macro definitions in their help files, and may import definitions from other packages.
• A number of macros have been added in the new ‘share/Rd’ directory for use in package overview help pages, and promptPackage() now makes use of them.
• tools::parse_Rd() gains a new permissive argument which converts unrecognized macros into text. This is used by utils:::format.bibentry to allow LaTeX markup to be ignored.
• options(OutDec =) can now specify a multi-byte character, e.g., options(OutDec = "u00b7") in a UTF-8 locale.
• is.recursive(x) is no longer true when x is an external pointer, a weak reference or byte code; the first enables all.equal(x, x) when x .
• ls() (aka objects()) and as.list.environment() gain a new argument sorted.
• The "source" attribute (which has not been added to functions by R since before R version 2.14.0) is no longer treated as special.
• Function returnValue() has been added to give on.exit() code access to a function’s return value for debugging purposes.
• crossprod(x, y) allows more matrix coercions when x or y are vectors, now equalling t(x) %*% y in these cases (also reported by Radford Neal). Similarly, tcrossprod(x,y) and %*% work in more cases with vector arguments.
• Utility function dynGet() useful for detecting cycles, aka infinite recursions.
• The byte-code compiler and interpreter include new instructions that allow many scalar subsetting and assignment and scalar arithmetic operations to be handled more efficiently. This can result in significant performance improvements in scalar numerical code.
• apply(m, 2, identity) is now the same as the matrix m when it has named row names.
• A new function debuggingState() has been added, allowing to temporarily turn off debugging.
• example() gets a new optional argument run.donttest and tools::Rd2ex() a corresponding commentDonttest, with a default such that example(..) in help examples will run donttest code only if used interactively (a change in behaviour).
• rbind.data.frame() gains an optional argument make.row.names, for potential speedup.
• New function extSoftVersion() to report on the versions of third-party software in use in this session. Currently reports versions of zlib, bzlib, the liblzma from xz, PCRE, ICU, TRE and the iconv implementation.

  A similar function grSoftVersion() in package grDevices reports on third-party graphics software.

  Function tcltk::tclVersion() reports the Tcl/Tk version.

• Calling callGeneric() without arguments now works with primitive generics to some extent.
• vapply(x, FUN, FUN.VALUE) is more efficient notably for large length(FUN.VALUE); as extension of PR#16061.
• as.table() now allows tables with one or more dimensions of length 0 (such as as.table(integer())).
• names(x) now clears the names of call and ... objects.
• library() will report a warning when an insufficient dependency version is masking a sufficient one later on the library search path.
• A new plot() method for class "raster" has been added.
• New check_packages_in_dir_changes() function in package tools for conveniently analyzing how changing sources impacts the check results of their reverse dependencies.
• Speed-up from Peter Haverty for ls() and methods:::.requirePackage() speeding up package loading. (PR#16133)
• New get0() function, combining exists() and get() in one call, for efficiency.
• match.call() gains an envir argument for specifying the environment from which to retrieve the ... in the call, if any; this environment was wrong (or at least undesirable) when thedefinition argument was a function.
• topenv() has been made .Internal() for speedup, based on Peter Haverty’s proposal in PR#16140.
• getOption() no longer calls options() in the main case.
• Optional use of libcurl (version 7.28.0 from Oct 2012 or later) for Internet access:
  • capabilities("libcurl") reports if this is available.
  • libcurlVersion() reports the version in use, and other details of the "libcurl" build including which URL schemes it supports.
  • curlGetHeaders() retrieves the headers for http://, https://, ftp:// and ftps:// URLs: analysis of these headers can provide insights into the ‘existence’ of a URL (it might for example be permanently redirected) and is so used in R CMD check --as-cran.
  • download.file() has a new optional method "libcurl" which will handle more URL schemes, follow redirections, and allows simultaneous downloads of multiple URLs.
  • url() has a new method "libcurl" which handles more URL schemes and follows redirections. The default method is controlled by a new option url.method, which applies also to the opening of URLs via file() (which happens implicitly in functions such as read.table.)
  • When file() or url() is invoked with a https:// or ftps:// URL which the current method cannot handle, it switches to a suitable method if one is available.
• (Windows.) The DLLs ‘internet.dll’ and ‘internet2.dll’ have been merged. In this version it is safe to switch (repeatedly) between the internal and Windows internet functions within an Rsession.

  The Windows internet functions are still selected by flag –internet2 or setInternet2(). This can be overridden for an url() connection via its new method argument.

  download.file() has new method "wininet", selected as the default by –internet2 or setInternet2().

• parent.env<- can no longer modify the parent of a locked namespace or namespace imports environment. Contributed by Karl Millar.
• New function isLoadedNamespace() for readability and speed.
• names(env) now returns all the object names of an environment env, equivalently to ls(env, all.names = TRUE, sorted = FALSE) and also to the names of the corresponding list,names(as.list(env, all.names = TRUE)). Note that although names() returns a character vector, the names have no particular ordering.
• The memory manager now grows the heap more aggressively. This reduces the number of garbage collections, in particular while data or code are loaded, at the expense of slightly increasing the memory footprint.
• New function trimws() for removing leading/trailing whitespace.
• cbind() and rbind() now consider S4 inheritance during S3 dispatch and also obey deparse.level.
• cbind() and rbind() will delegate recursively to methods::cbind2 (methods::rbind2) when at least one argument is an S4 object and S3 dispatch fails (due to ambiguity).
• (Windows.) download.file(quiet = FALSE) now uses text rather than Windows progress bars in non-interactive use.
• New function hsearch_db() in package utils for building and retrieving the help search database used by help.search(), along with functions for inspecting the concepts and keywords in the help search database.
• New function .getNamespaceInfo(), a no-check version of getNamespaceInfo() mostly for internal speedups.
• The help search system now takes keyword entries in Rd files which are not standard keywords (as given in ‘KEYWORDS’ in the R documentation directory) as concepts. For standard keyword entries the corresponding descriptions are additionally taken as concepts.
• New lengths() function for getting the lengths of all elements in a list.
• New function toTitleCase() in package tools, tailored to package titles.
• The matrix methods of cbind() and rbind() allow matrices as inputs which have 2^31 or more elements. (For cbind(), wish of PR#16198.)
• The default method of image() has an explicit check for a numeric or logical matrix (which was always required).
• URLencode() will not by default encode further URLs which appear to be already encoded.
• BIC(mod) and BIC(mod, mod2) now give non-NA numbers for arima() fitted models, as nobs(mod) now gives the number of “used” observations for such models. This fixes PR#16198, quite differently than proposed there.
• The print() methods for "htest", "pairwise.htest" and "power.htest" objects now have a digits argument defaulting to (a function of) getOption("digits"), and influencing all printed numbers coherently. Unavoidably, this changes the display of such test results in some cases.
• Code completion for namespaces now recognizes all loaded namespaces, rather than only the ones that are also attached.
• The code completion mechanism can now be replaced by a user-specified completer function, for (temporary) situations where the usual code completion is inappropriate.
• unzip() will now warn if it is able to detect truncation when unpacking a file of 4GB or more (related to PR#16243).
• methods() reports S4 in addition to S3 methods; output is simplified when the class argument is used. .S3methods() and methods::.S4methods() report S3 and S4 methods separately.
• Higher order functions such as the apply functions and Reduce() now force arguments to the functions they apply in order to eliminate undesirable interactions between lazy evaluation and variable capture in closures. This resolves PR#16093.

More at http://cran.rstudio.com/

Reference: http://www.r-bloggers.com/r-3-2-0-is-released-using-the-installr-package-to-upgrade-in-windows-os/

1. Setting up your computing environment
2. Retrieving and storing sequencing files (your own data or from public sources)
3. Checking sequence quality, trimming, general sequence manipulation
4. Mapping reads to a reference genome
5. Manipulating SAM/BAM alignment files
6. Visualizing data in a genome browser

RNA-Seq

1. De novo transcript discovery and differential analysis with Cufflinks
2. Differential expression analysis with R/Bioconductor
3. Clustering of large expression datasets (microarray or RNA-Seq)

Microarray

1. Basic analysis of Affymetrix Gene Expression Arrays using R/Bioconductor

General Tips for Data Analysis

1. Excel workarounds, adding gene annotation, X-Y plots tips, etc.

Address of the bookmark: http://homer.salk.edu/homer/basicTutorial/

Address of the bookmark: http://engr.case.edu/li_jing/papers/00798gpattern.pdf

Address of the bookmark: https://github.com/mattharrison/Tiny-Python-3.6-Notebook/blob/master/python.rst

install and load required packages

  require(RColorBrewer)
  require(ComplexHeatmap)
  require(circlize)
  require(digest)
  require(cluster)

If all load successfully, proceed to Part 3. Otherwise, go through the following code chunks in order to ensure that each package is installed and loaded properly.

BiocManager (Morgan (2019))

Address of the bookmark: https://github.com/kevinblighe/E-MTAB-6141

Why use AWK instead of Perl? Readability. AWK is easier to read than Perl. For simple text-processing scripts, particularly ones that read files line by line and split on delimiters, AWK is probably the right tool for the job.

#!/usr/bin/awk -f

# Comments are like this


# AWK programs consist of a collection of patterns and actions.
pattern1 { action; } # just like lex
pattern2 { action; }

# There is an implied loop and AWK automatically reads and parses each
# record of each file supplied. Each record is split by the FS delimiter,
# which defaults to white-space (multiple spaces,tabs count as one)
# You can assign FS either on the command line (-F C) or in your BEGIN
# pattern

# One of the special patterns is BEGIN. The BEGIN pattern is true
# BEFORE any of the files are read. The END pattern is true after
# an End-of-file from the last file (or standard-in if no files specified)
# There is also an output field separator (OFS) that you can assign, which
# defaults to a single space

BEGIN {

    # BEGIN will run at the beginning of the program. It's where you put all
    # the preliminary set-up code, before you process any text files. If you
    # have no text files, then think of BEGIN as the main entry point.

    # Variables are global. Just set them or use them, no need to declare..
    count = 0;

    # Operators just like in C and friends
    a = count + 1;
    b = count - 1;
    c = count * 1;
    d = count / 1; # integer division
    e = count % 1; # modulus
    f = count ^ 1; # exponentiation

    a += 1;
    b -= 1;
    c *= 1;
    d /= 1;
    e %= 1;
    f ^= 1;

    # Incrementing and decrementing by one
    a++;
    b--;

    # As a prefix operator, it returns the incremented value
    ++a;
    --b;

    # Notice, also, no punctuation such as semicolons to terminate statements

    # Control statements
    if (count == 0)
        print "Starting with count of 0";
    else
        print "Huh?";

    # Or you could use the ternary operator
    print (count == 0) ? "Starting with count of 0" : "Huh?";

    # Blocks consisting of multiple lines use braces
    while (a < 10) {
        print "String concatenation is done" " with a series" " of"
            " space-separated strings";
        print a;

        a++;
    }

    for (i = 0; i < 10; i++)
        print "Good ol' for loop";

    # As for comparisons, they're the standards:
    # a < b   # Less than
    # a <= b  # Less than or equal
    # a != b  # Not equal
    # a == b  # Equal
    # a > b   # Greater than
    # a >= b  # Greater than or equal

    # Logical operators as well
    # a && b  # AND
    # a || b  # OR

    # In addition, there's the super useful regular expression match
    if ("foo" ~ "^fo+$")
        print "Fooey!";
    if ("boo" !~ "^fo+$")
        print "Boo!";

    # Arrays
    arr[0] = "foo";
    arr[1] = "bar";

    # You can also initialize an array with the built-in function split()

    n = split("foo:bar:baz", arr, ":");

    # You also have associative arrays (actually, they're all associative arrays)
    assoc["foo"] = "bar";
    assoc["bar"] = "baz";

    # And multi-dimensional arrays, with some limitations I won't mention here
    multidim[0,0] = "foo";
    multidim[0,1] = "bar";
    multidim[1,0] = "baz";
    multidim[1,1] = "boo";

    # You can test for array membership
    if ("foo" in assoc)
        print "Fooey!";

    # You can also use the 'in' operator to traverse the keys of an array
    for (key in assoc)
        print assoc[key];

    # The command line is in a special array called ARGV
    for (argnum in ARGV)
        print ARGV[argnum];

    # You can remove elements of an array
    # This is particularly useful to prevent AWK from assuming the arguments
    # are files for it to process
    delete ARGV[1];

    # The number of command line arguments is in a variable called ARGC
    print ARGC;

    # AWK has several built-in functions. They fall into three categories. I'll
    # demonstrate each of them in their own functions, defined later.

    return_value = arithmetic_functions(a, b, c);
    string_functions();
    io_functions();
}

# Here's how you define a function
function arithmetic_functions(a, b, c,     d) {

    # Probably the most annoying part of AWK is that there are no local
    # variables. Everything is global. For short scripts, this is fine, even
    # useful, but for longer scripts, this can be a problem.

    # There is a work-around (ahem, hack). Function arguments are local to the
    # function, and AWK allows you to define more function arguments than it
    # needs. So just stick local variable in the function declaration, like I
    # did above. As a convention, stick in some extra whitespace to distinguish
    # between actual function parameters and local variables. In this example,
    # a, b, and c are actual parameters, while d is merely a local variable.

    # Now, to demonstrate the arithmetic functions

    # Most AWK implementations have some standard trig functions
    localvar = sin(a);
    localvar = cos(a);
    localvar = atan2(b, a); # arc tangent of b / a

    # And logarithmic stuff
    localvar = exp(a);
    localvar = log(a);

    # Square root
    localvar = sqrt(a);

    # Truncate floating point to integer
    localvar = int(5.34); # localvar => 5

    # Random numbers
    srand(); # Supply a seed as an argument. By default, it uses the time of day
    localvar = rand(); # Random number between 0 and 1.

    # Here's how to return a value
    return localvar;
}

function string_functions(    localvar, arr) {

    # AWK, being a string-processing language, has several string-related
    # functions, many of which rely heavily on regular expressions.

    # Search and replace, first instance (sub) or all instances (gsub)
    # Both return number of matches replaced
    localvar = "fooooobar";
    sub("fo+", "Meet me at the ", localvar); # localvar => "Meet me at the bar"
    gsub("e+", ".", localvar); # localvar => "m..t m. at th. bar"

    # Search for a string that matches a regular expression
    # index() does the same thing, but doesn't allow a regular expression
    match(localvar, "t"); # => 4, since the 't' is the fourth character

    # Split on a delimiter
    n = split("foo-bar-baz", arr, "-"); # a[1] = "foo"; a[2] = "bar"; a[3] = "baz"; n = 3

    # Other useful stuff
    sprintf("%s %d %d %d", "Testing", 1, 2, 3); # => "Testing 1 2 3"
    substr("foobar", 2, 3); # => "oob"
    substr("foobar", 4); # => "bar"
    length("foo"); # => 3
    tolower("FOO"); # => "foo"
    toupper("foo"); # => "FOO"
}

function io_functions(    localvar) {

    # You've already seen print
    print "Hello world";

    # There's also printf
    printf("%s %d %d %d\n", "Testing", 1, 2, 3);

    # AWK doesn't have file handles, per se. It will automatically open a file
    # handle for you when you use something that needs one. The string you used
    # for this can be treated as a file handle, for purposes of I/O. This makes
    # it feel sort of like shell scripting, but to get the same output, the string
    # must match exactly, so use a variable:

    outfile = "/tmp/foobar.txt";

    print "foobar" > outfile;

    # Now the string outfile is a file handle. You can close it:
    close(outfile);

    # Here's how you run something in the shell
    system("echo foobar"); # => prints foobar

    # Reads a line from standard input and stores in localvar
    getline localvar;

    # Reads a line from a pipe (again, use a string so you close it properly)
    cmd = "echo foobar";
    cmd | getline localvar; # localvar => "foobar"
    close(cmd);

    # Reads a line from a file and stores in localvar
    infile = "/tmp/foobar.txt";
    getline localvar < infile; 
    close(infile);
}

# As I said at the beginning, AWK programs consist of a collection of patterns
# and actions. You've already seen the BEGIN pattern. Other
# patterns are used only if you're processing lines from files or standard
# input.
#
# When you pass arguments to AWK, they are treated as file names to process.
# It will process them all, in order. Think of it like an implicit for loop,
# iterating over the lines in these files. these patterns and actions are like
# switch statements inside the loop. 

/^fo+bar$/ {

    # This action will execute for every line that matches the regular
    # expression, /^fo+bar$/, and will be skipped for any line that fails to
    # match it. Let's just print the line:

    print;

    # Whoa, no argument! That's because print has a default argument: $0.
    # $0 is the name of the current line being processed. It is created
    # automatically for you.

    # You can probably guess there are other $ variables. Every line is
    # implicitly split before every action is called, much like the shell
    # does. And, like the shell, each field can be access with a dollar sign

    # This will print the second and fourth fields in the line
    print $2, $4;

    # AWK automatically defines many other variables to help you inspect and
    # process each line. The most important one is NF

    # Prints the number of fields on this line
    print NF;

    # Print the last field on this line
    print $NF;
}

# Every pattern is actually a true/false test. The regular expression in the
# last pattern is also a true/false test, but part of it was hidden. If you
# don't give it a string to test, it will assume $0, the line that it's
# currently processing. Thus, the complete version of it is this:

$0 ~ /^fo+bar$/ {
    print "Equivalent to the last pattern";
}

a > 0 {
    # This will execute once for each line, as long as a is positive
}

# You get the idea. Processing text files, reading in a line at a time, and
# doing something with it, particularly splitting on a delimiter, is so common
# in UNIX that AWK is a scripting language that does all of it for you, without
# you needing to ask. All you have to do is write the patterns and actions
# based on what you expect of the input, and what you want to do with it.

# Here's a quick example of a simple script, the sort of thing AWK is perfect
# for. It will read a name from standard input and then will print the average
# age of everyone with that first name. Let's say you supply as an argument the
# name of a this data file:
#
# Bob Jones 32
# Jane Doe 22
# Steve Stevens 83
# Bob Smith 29
# Bob Barker 72
#
# Here's the script:

BEGIN {

    # First, ask the user for the name
    print "What name would you like the average age for?";

    # Get a line from standard input, not from files on the command line
    getline name < "/dev/stdin";
}

# Now, match every line whose first field is the given name
$1 == name {

    # Inside here, we have access to a number of useful variables, already
    # pre-loaded for us:
    # $0 is the entire line
    # $3 is the third field, the age, which is what we're interested in here
    # NF is the number of fields, which should be 3
    # NR is the number of records (lines) seen so far
    # FILENAME is the name of the file being processed
    # FS is the field separator being used, which is " " here
    # ...etc. There are plenty more, documented in the man page.

    # Keep track of a running total and how many lines matched
    sum += $3;
    nlines++;
}

# Another special pattern is called END. It will run after processing all the
# text files. Unlike BEGIN, it will only run if you've given it input to
# process. It will run after all the files have been read and processed
# according to the rules and actions you've provided. The purpose of it is
# usually to output some kind of final report, or do something with the
# aggregate of the data you've accumulated over the course of the script.

END {
    if (nlines)
        print "The average age for " name " is " sum / nlines;
}

Prerequisites

This is an intermediate lesson and assumes learners have already done some bioinformatics:

• Familiarity with the BASH command shell, including concepts like pipes, variables and loops.
• Knowledge of bioinformatics fundamentals like the FASTQ file format and transcriptome sequencing, in order to understand the example workflow.

No previous knowledge of Snakemake or workflow systems is required.

https://carpentries-incubator.github.io/snakemake-novice-bioinformatics/index.html

Address of the bookmark: https://carpentries-incubator.github.io/snakemake-novice-bioinformatics/aio/index.html

Title: Comparison of Short Read De Novo Alignment Algorithms 

Author: Nikhil Gopal

Address of the bookmark: http://biochem218.stanford.edu/Projects%202011/Gopal%202011.pdf

Unless otherwise noted, all terminology below is consistent with this paper, and all references to figures and tables in this readme refer to this paper. Specifically, some of the terminology used below is outlined in Figure S2. The assembly procedure is described in detail in the Supporting Online Materials, specifically in the section labelled “Pipeline description”.

In addition, the pipeline uses tools and methods from Juicer (Durand & Shamim et al., Cell Systems, 2016) and Juicebox (Durand & Robinson et al., Cell Systems, 2016), as well as additional dependencies noted below.

Feel free to post your questions and comments at: http://www.aidenlab.org/forum.html

http://aidenlab.org/documentation.html

Address of the bookmark: https://github.com/theaidenlab/3d-dna

Detail paper at https://bmcgenomics.biomedcentral.com/articles/10.1186/s12864-017-3927-8

Address of the bookmark: https://github.com/VicugnaPacos/ALPACA