tags: `R` `error` `bash` `n` `glm()` `hoslem.test()` `complete.cases` `cat` `merge` `dplyr::filter` `dplyr::group_by_` `dplyr::summarize` `read.table(fill=TRUE)` `tryCatch()` `.libPaths()` `R.home()` `Sys.getenv()` `ifelse()` `unloadNamespace()` `is.na()` `dplyr::mutate()` `dplyr::case_when()` `dplyr::summarise_at()` `dplyr` `rlang` `attributes()` `foreach()` `%dopar%` `%:%` `%do` `devtools::install_github()` `install.packages()`

Errors, warnings, issues in R programming

Issue Graph color not matched legend color. activity.type='Strength & Stability Workout' is shown in gray, not pink. Using a different color palette doesn't make a difference.

Screen Shot 12-02-23 at 11.17 AM

tail(activities.2023.11, n=3)
#   start.date.local moving.time.hour                activity.type
#23       2023-11-15         2.182778                         Ride
#24       2023-11-15         1.505278 Strength & Stability Workout
#25       2023-11-16         1.158056                          Run

Issue R files shown as 0 KB. R files reopened as empty in RStudio.
Solution I had a similar issue with older R files that opened as empty. It turned out that RStudio didn't use the correct encoding as default and therefore wasn't able to read the file (presented the file as empty). You can make sure that you are using the correct encoding by:
0. Copying R code to a text file

Opening the file in RStudio as you normally would (the file will be empty)
Navigate to File -> Reopen with Encoding…
Select UTF-8 and click OK. This will clear the content in the R file
Copy the R code back to the R file
UTF-8 will most likely be the encoding you need. You can also choose to set this as the default for all source files.
.Rmd files open as completely empty

Error Warning in install.packages : 'lib = "C:/Program Files/R/R-4.0.3/library"' is not writable Error in install.packages : unable to install packages
Solution Close RStudio/R. Run it as administrator
"not writable" error when installing new packages on 3.2.0

Error devtools::install_github("DillonHammill/CytoExploreRData")
Downloading GitHub repo DillonHammill/CytoExploreRData@HEAD
Error in utils::download.file(url, path, method = method, quiet = quiet, :
cannot open URL 'https://api.github.com/repos/DillonHammill/CytoExploreRData/tarball/HEAD'

Solution In RStudio, Tools > Global options> Package > check the Use secure download method for HTTP box
having trouble getting devtools::install_github() to work in R on Win 7 64bit machine
Secure Package Downloads for R

Secure Package Downloads for R

# Before checking the Use secure download method for HTTP box in RStudio
devtools::install_github("DillonHammill/CytoExploreRData")
# Downloading GitHub repo DillonHammill/CytoExploreRData@HEAD
# Error in utils::download.file(url, path, method = method, quiet = quiet,  : 
#  cannot open URL 'https://api.github.com/repos/DillonHammill/CytoExploreRData/tarball/HEAD'

# After checking the Use secure download method for HTTP box in RStudio
devtools::install_github("DillonHammill/CytoExploreRData", lib=dir.R.packages)
# Downloading GitHub repo DillonHammill/CytoExploreRData@HEAD
# √  checking for file 'C:\Users\lunC\AppData\Local\Temp\Rtmp4erTUM\remotes2d6c53fb506c\DillonHammill-CytoExploreRData-488edf0/DESCRIPTION'
# -  preparing 'CytoExploreRData': (1s)
# √  checking DESCRIPTION meta-information ...
# -  checking for LF line-endings in source and make files and shell scripts
# -  checking for empty or unneeded directories
#      NB: this package now depends on R (>= 3.5.0)
#      WARNING: Added dependency on R >= 3.5.0 because serialized objects in  serialize/load version 3 cannot be read in older versions of R.  File(s) containing such objects:  'CytoExploreRData/data/Activation.rda'  WARNING: Added dependency on R >= 3.5.0 because serialized objects in  serialize/load version 3 cannot be read in older versions of R.  File(s) containing such objects:  'CytoExploreRData/data/Activation_gatingTemplate.rda'  WARNING: Added dependency on R >= 3.5.0 because serialized objects in  serialize/load version 3 cannot be read in older versions of R.  File(s) containing such objects:  'CytoExploreRData/data/Compensation.rda'  WARNING: Added dependency on R >= 3.5.0 because serialized objects in  serialize/load version 3 cannot be read in older versions of R.  File(s) containing such objects:  'CytoExploreRData/data/Compensation_gatingTemplate.rda'
# -  building 'CytoExploreRData_1.0.3.tar.gz'
   
# * installing *source* package 'CytoExploreRData' ...
# ** using staged installation
# ** R
# ** data
# *** moving datasets to lazyload DB
# ** inst
# ** byte-compile and prepare package for lazy loading
# ** help
# *** installing help indices
#   converting help for package 'CytoExploreRData'
#     finding HTML links ... done
#     Activation                              html  
#     Activation_gatingTemplate               html  
#     Compensation                            html  
#     Compensation_gatingTemplate             html  
#     CytoExploreRData                        html 
# ** building package indices
# ** installing vignettes
# ** testing if installed package can be loaded from temporary location
# *** arch - i386
# *** arch - x64
# ** testing if installed package can be loaded from final location
# *** arch - i386
# *** arch - x64
# ** testing if installed package keeps a record of temporary installation path
# * DONE (CytoExploreRData)

Error The procedure entry point EXTPTR_PTR Rcould not be located in the dynamic link library

Solution Use a R version no later than the package to load. For instance, to load package ‘magick’ that was built under R version 4.0.2, use R version 4.0.2. Using a version older than R-4.0.2 could result in the error.

I experienced that the "Entry Point" error message (concerning "rlang.dll", "Rcpp.dll" or other .dll) occures when using/choosing a lower R version in RStudio on packages of a higher built , i.e. using R 3.4 on packages of built 3.5, e.g. after using update.packages(…, checkBuilt = TRUE) on user library of lower R Version 3.4…after getting rid of the packages of the higher built or re-installing them on the lower built in the user library, no "Entry Point" error messages occure anymore when using the lower R Version R 3.4… Entry point Not Found #416

# Not working
dir.R.packages <- "C:/Program Files/R/R-4.0.0/library" #"C:/Program Files/R/R-3.6.2/library"

install.packages("magick",lib=dir.R.packages)
library(magick)

# Error: package or namespace load failed for ‘magick’ in inDL(x, as.logical(local), as.logical(now), ...):
# unable to load shared object 'C:/Program Files/R/R-4.0.0/library/Rcpp/libs/x64/Rcpp.dll':
#  LoadLibrary failure:  The specified procedure could not be found.
# In addition: Warning message:
# package ‘magick’ was built under R version 4.0.2 

# Working
# RStudio > Tools > Global Options > General > R version > [64 bit] C:\Program Files\R\R-4.0.2
dir.R.packages <- "C:/Program Files/R/R-4.0.2/library" #"C:/Program Files/R/R-3.6.2/library"

install.packages("magick",lib=dir.R.packages)
library(magick)
# Linking to ImageMagick 6.9.9.14
# Enabled features: cairo, freetype, fftw, ghostscript, lcms, pango, rsvg, webp
# Disabled features: fontconfig, x11

WARNING: Rtools is required to build R packages but is not currently installed. Please download and install the appropriate version of Rtools before proceeding: https://cran.rstudio.com/bin/windows/Rtools/
Warning in install.packages :
package ‘edgeR’ is not available (for R version 3.6.2)
Solution

Install Rtools35.exe resolved the warning Building R for Windows
Installing bioconductor packages using BiocManager::install(pkgs=)

# Not working
install.packages("edgeR", lib="C:/Program Files/R/R-3.6.2/library")
# WARNING: Rtools is required to build R packages but is not currently installed. Please download and install the appropriate version of Rtools before proceeding: https://cran.rstudio.com/bin/windows/Rtools/
# Warning in install.packages :
#  package ‘edgeR’ is not available (for R version 3.6.2)

# Working
dir.R.packages <- "C:/Program Files/R/R-3.6.2/library"

if (!requireNamespace("BiocManager", quietly = TRUE))
  install.packages("BiocManager")

BiocManager::install(pkgs = "Glimma", lib=dir.R.packages)
BiocManager::install(pkgs = "Mus.musculus", lib=dir.R.packages)

Warning message: funs() is deprecated as of dplyr 0.8.0. Please use a list of either functions or lambdas:
Error: .vars must be a character/numeric vector or a vars() object, not a formula object.

Solution Specify the functions to summarise or rename multiple columns using
vars("column1","column2","column3"), ~functionName()
vars("column1","column2","column3"), list(~function1(), ~function2())
Warning message for dplyr package when trying to utilize the summarise_each function

# Deprecated as of dplyr 0.8.0
dplyr::summarise_at( .vars= c("TMA.core.ID","specimenID","shortID")
                    ,.funs= funs(paste(na.omit(.), collapse = ","))) 
# Working with 1 function to pass to 3 columns
dplyr::summarise_at( vars(c("TMA.core.ID","specimenID","shortID"))
                    , ~ paste(na.omit(.), collapse = ",")) 
# Working code with 2 functions to pass to 1 column
library(tidyverse)
#> Warning: package 'forcats' was built under R version 3.6.3
library(nycflights13)
#> Warning: package 'nycflights13' was built under R version 3.6.3

flights %>%
  group_by(carrier) %>%
  summarise_at(vars(matches("delay")), list(~min(., na.rm = TRUE), ~max(., na.rm = TRUE)))
#> # A tibble: 1

Error: "Error in unserialize(socklist[[n]]) : error reading from connection"
In addition: Warning messages:
1: In .Internal(gc(verbose, reset, full))
closing unused connection 9 (<-QIMR18447.adqimr.ad.lan:11372)
2: In .Internal(gc(verbose, reset, full))
closing unused connection 8 (<-QIMR18447.adqimr.ad.lan:11372)
3: In .Internal(gc(verbose, reset, full))
closing unused connection 7 (<-QIMR18447.adqimr.ad.lan:11372)
4: In .Internal(gc(verbose, reset, full))
closing unused connection 6 (<-QIMR18447.adqimr.ad.lan:11372)
5: In .Internal(gc(verbose, reset, full))
closing unused connection 5 (<-QIMR18447.adqimr.ad.lan:11372)
6: In .Internal(gc(verbose, reset, full))
closing unused connection 4 (<-QIMR18447.adqimr.ad.lan:11372)
7: In .Internal(gc(verbose, reset, full))

Error: Error in foreach(i = 1:length(Exp08.categorised.predictors.list), .combine = "rbind", :
"%:%" was passed an illegal right operand

Solution: Nest foreach loops correctly as shown in the code below. Outer loop variable in nested R foreach loop

X <- c("A", "B")
Y <- 1:3

## (1) EITHER merge two 'foreach' objects using '%:%' ...
foreach (j = X, .combine = c) %:% foreach(i = Y, .combine = c) %do% {
    paste(j, i, sep = "")
}
# [1] "A1" "A2" "A3" "B1" "B2" "B3"


## (2) ... OR Nest two 'foreach' objects using a pair of '%do%' operators ...
foreach(j = X, .combine = c) %do% {
    foreach(i = Y, .combine = c) %do% {
        paste(j, i, sep = "")
    }
}
# [1] "A1" "A2" "A3" "B1" "B2" "B3"

## (3) ... BUT DON'T use a hybrid of the approaches.
foreach(j = X, .combine = c) %:% {
    foreach(i = Y, .combine = c) %do% {
        paste(j, i, sep = "")
    }
}
# Error in foreach(j = X, .combine = c) %:% { : 
#   "%:%" was passed an illegal right operand

Error: x must be a vector, not a data.frame/surv_categorize object.

Solution Multiple classes in the data attributes cause the error. Changing the class to data.frame only gets rid of the error while running base functions (e.g., nrow(), names()) within dply package.

# This generates the error `x` must be a vector, not a `data.frame/surv_categorize` object.

Exp26.predictors.cat.RFS.tumor <- dplyr::left_join(   x= .Exp26.predictors.cat.RFS.tumor
                                                     ,y= Exp26.survival.markers.tumor[,c(ID.columns, "RFS_years","RFS_censor", Exp26.covariates.RFS)]
                                                     ,by=c(  "row.numb"="row.numb"
                                                            ,"RFS_years"="RFS_years"
                                                            ,"RFS_censor"="RFS_censor")) %>%
  # Reorder columns
  dplyr::select_(.dots = c(ID.columns
                           ,"RFS_years","RFS_censor"
                           , sort(grep(pattern = "^number", x= names(.), value = TRUE))
                           , sort(grep(pattern = "^CD", x= names(.), value = TRUE))
  ))

Error: `x` must be a vector, not a `data.frame/surv_categorize` object.

# This is working
# Change the class of data attribute to data.frame rahter than two types ("data.frame",  "surv_categorize")
attributes(.Exp26.predictors.cat.RFS.tumor)$class <- c("data.frame")

# Merge categorised predictors back with predictor value columns
## table x : dim(.Exp26.predictors.cat.RFS.tumor) 29 7
## table y : dim(Exp26.survival.markers.tumor[,c(ID.columns, "RFS_years","RFS_censor", Exp26.covariates.RFS)]) 29 10
Exp26.predictors.cat.RFS.tumor <- dplyr::left_join(   x= .Exp26.predictors.cat.RFS.tumor
                                                     ,y= Exp26.survival.markers.tumor[,c(ID.columns, "RFS_years","RFS_censor", Exp26.covariates.RFS)]
                                                     ,by=c(  "row.numb"="row.numb"
                                                            ,"RFS_years"="RFS_years"
                                                            ,"RFS_censor"="RFS_censor")) %>%
  # Reorder columns
  dplyr::select_(.dots = c(ID.columns
                           ,"RFS_years","RFS_censor"
                           , sort(grep(pattern = "^number", x= names(.), value = TRUE))
                           , sort(grep(pattern = "^CD", x= names(.), value = TRUE))
  )) # dim(Exp26.predictors.cat.RFS.tumor) 29 14

Error: package or namespace load failed for ‘dplyr’: .onLoad failed in loadNamespace() for 'pillar', details: call: loadNamespace(i, c(lib.loc, .libPaths()), versionCheck = vI[[i]]) error: namespace ‘rlang’ 0.4.1 is already loaded, but >= 0.4.2 is required

Solution The error occurred because the rlang package that was automatically loaded when starting R is in a version older than the rlang version required by another package dplyr. To rid of this error, try

Solution 1: unload the loaded package rlang and then reload the rlang, or
Solution 2: Remove the rlang package and reinstall it rror: package or namespace load failed for ‘dplyr’

# Error while loading dplyr package
R_3.6.2_packages_dir <-  "/software/R/R-3.6.2/lib64/R/library"
library(dplyr, lib.loc= R_3.6.2_packages_dir)
#Error: package or namespace load failed for ‘dplyr’:
# .onLoad failed in loadNamespace() for 'pillar', details:
#  call: loadNamespace(i, c(lib.loc, .libPaths()), versionCheck = vI[[i]])
#  error: namespace ‘rlang’ 0.4.1 is already loaded, but >= 0.4.2 is required

sessionInfo()
# R version 3.6.2 (2019-12-12)
# Platform: x86_64-pc-linux-gnu (64-bit)
# Running under: CentOS Linux 7 (Core)
# 
# Matrix products: default
# BLAS/LAPACK: /software/OpenBLAS/OpenBLAS-0.3.3/lib/libopenblasp-r0.3.3.so
# 
# locale:
#   [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C
# [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8
# [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8
# [7] LC_PAPER=en_US.UTF-8       LC_NAME=C
# [9] LC_ADDRESS=C               LC_TELEPHONE=C
# [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
# 
# attached base packages:
#   [1] stats     graphics  grDevices utils     datasets  methods   base
# 
# other attached packages:
#   [1] RColorBrewer_1.1-2 magick_2.0
# 
# loaded via a namespace (and not attached):
# [1] compiler_3.6.2   assertthat_0.2.1 R6_2.4.0         magrittr_1.5
# [5] pillar_1.4.3     glue_1.3.1       tibble_2.1.3     crayon_1.3.4
# [9] Rcpp_1.0.2       pkgconfig_2.0.2  rlang_0.4.1

# This is ther version of rlang that has been loaded
packageVersion("rlang")
#[1] ‘0.4.1’

# This is ther version of rlang that you will reload 
packageVersion("rlang",lib.loc=R_3.6.2_packages_dir)
#[1] ‘0.4.5’

# Unload the loaded package rlang and load it again using latest version. This won't work
detach("package:rlang",unload=TRUE,character.only=TRUE)
# Error in detach("package:rlang", unload = TRUE, character.only = TRUE) :
#   invalid 'name' argument

# Unload the loaded package rlang and load it again using latest version. This works
unloadNamespace("rlang")
# Error in unloadNamespace("rlang") :
#   namespace ‘rlang’ is imported by ‘tibble’, ‘pillar’ so cannot be unloaded

#-----------------------------------
# Solution 1: unload the loaded rlang package and reload it
### Note that this may not work if packages that are dependent on rlang have been loaded
#-----------------------------------
unloadNamespace("tibble")
unloadNamespace("pillar")
unloadNamespace("rlang")

# Reload rlang package
library(rlang, lib.loc= R_3.6.2_packages_dir)

# Load dplyr package
library(dplyr, lib.loc= R_3.6.2_packages_dir)

#----------------------------------------------
# Solution 2: remove rlang and reinstall rlang in a newer version
#----------------------------------------------

# Deal with errors associated with rlang
## Error: package or namespace load failed for ‘Seurat’ in loadNamespace(j <- i[[1L]], c(lib.loc, .libPaths()), versionCheck = vI[[j]]): namespace ‘rlang’ 0.4.7 is already loaded, but >= 0.4.9 is required

# Clear the work environment. Restart R
remove.packages("rlang", lib = dir.R.packages)
install.packages("rlang", lib = dir.R.packages)
# Reload rlang package
library(rlang, lib.loc= dir.R.packages)

`Error in get_oauth_sig() : OAuth has not been registered for this session`

ust be a double vector, not a primitive function Call `rlang::last_error()` to see a backtrace.` Error arises because column c is not found in the data.frame

a = c(3,7,NA, 9)
b = c(2,NA,9,3)
f = c(5,2,5,6)
d = c(NA,3,4,NA)
mydf = data.frame(a=a,b=b,f=f,d=d)

# This code generates the error above because column c is not found in the dataframe
library(dplyr)
mydf <- mydf %>% dplyr::mutate(e= dplyr::case_when( is.na(a) ~ b
                                                    ,is.na(b) ~ d
                                                    , TRUE ~ c))
# This code is working
mydf[,5] <- ifelse(is.na(mydf[,1]) & !is.na(mydf[,2]),
               mydf[,2],
               ifelse(is.na(mydf[,2]) & !is.na(mydf[,4]),
               mydf[,4], mydf[,3]))

# This code is working
mydf <- mydf %>% dplyr::mutate(e= dplyr::case_when( is.na(a) & !is.na(b) ~ b
                                                    ,is.na(b) & !is.na(d) ~ d
                                                    , TRUE ~ f))
# This code is working
a = c(3,7,NA, 9)
b = c(2,NA,9,3)
c = c(5,2,5,6)
d = c(NA,3,4,NA)
e <- c 
e[which (is.na(mydf$a))] <- b [which (is.na(mydf$a))]
e[which (is.na(mydf$b))] <- d [which (is.na(mydf$b))]

mydf = data.frame(a=a,b=b,c=c,d=d, e=e)

`Error: package dplyr was installed by an R version with different internals; it needs to be reinstalled for use with this R version In addition: Warning message: package ‘dplyr’ was built under R version 3.5.1` It depends on how you use your custom installed libraries. If you use them "stand-alone" i.e. without /software/R/version in your libPaths, then deleting that folder will break anything installed to your ~/R directory that depends on Rcpp (which could be a lot of things), so I wouldn't necessarily recommend that. The real root of the issue here is that there is no way to guarantee that software you've installed yourself in one location is compatible with software compiled and installed to a different location, but you can try one of the following:

As you suggested, just set .libPaths("/software/R/R-3.4.1/lib64/R") at the start of your session so you're only using the installed system libraries. This is the safest option.
Reverse the order of paths in libPaths, e.g. .libPaths(system path, custom path) so that the system libraries are found first. Then at least the system libraries will work. If (and there's no guarantee) your custom libraries are compatible with the system libraries, then those will also work.
Xikun: if you mould load R/3.5.1 then in the R script simply library(package). Don't add lib path as this will result in a conflict

`> detach("package:plyr", unload=TRUE) Error in detach("package:plyr", unload = TRUE) : invalid 'name' argument`

How to unload a package without restarting R

# detach() did not work
detach("package:plyr", unload=TRUE)
Error in detach("package:plyr", unload = TRUE) : invalid 'name' argument

# unloadNamespace() worked
unloadNamespace("plyr")

`Error in ifelse(m$cohorts == "UKB" & m$traits.abb == "CI", "OA", ifelse(m$cohorts == : unused arguments ("OA, GA", "GA")` This error occurred when a ifelse() doesn't contain 3 arguments. A simple check is to see if there are 2 commas in each ifelse, and the number of ifelse() functions is similar to the number of ending brackets.

Unused argument error in nested ifelse statements

# Six nested ifelse() should have six ending brackets
rem <- 21%%7
day <- ifelse(rem==0, "Thursday", 
         ifelse (rem==1, "Friday", 
           ifelse (rem==2, "Saturday", 
             ifelse (rem==3, "Sunday", 
               ifelse (rem==4, "Monday", 
                 ifelse(rem==5, "Tuesday", "Wednesday")
               )
             )
           )
         )
       )

`Error in filter_impl(.data, quo) : Evaluation error:` as_dictionary()`is defunct as of rlang 0.3.0. Please use`as_data_pronoun() `instead.` The error occurred when running `dplyr::filter()` The error is gone after HPC Admin updated the package dplyr

`Error in loadNamespace(i, c(lib.loc, .libPaths()), versionCheck = vI[[i]]) : namespace ‘rlang’ 0.2.1 is already loaded, but >= 0.3.1 is required` occurred while running `library(dplyr)` There are two problems (1)`rlang` in the dafault path is in an old version, (2) the default path is in a user created folder that is not centrally controlled.

# Loading the package dplyr resulted in an error
library(dplyr)
Error in loadNamespace(i, c(lib.loc, .libPaths()), versionCheck = vI[[i]]) : 
  namespace ‘rlang’ 0.2.1 is already loaded, but >= 0.3.1 is required
  
# In your RStudio session, unload the package that the error is based on (rlang here). Every library you've ever loaded is likely still in your session and conflicts will arise
detach("package:rlang", unload=TRUE)

# Configure R to *not* use you locally installed packages in /mnt/backedup/home/lunC/R/x86_64-pc-linux-gnu-library/3.4. If you need packages installed, let us know. We're happy to put the in centrally so you don't have conflicts between your personally installed and our centrally installed packagesUse
## .libPaths() inside R to check current library paths
.libPaths()
[1] "/mnt/backedup/home/lunC/R/x86_64-pc-linux-gnu-library/3.4" # This is a user-created folder
[2] "/software/R/R-3.4.1/lib64/R/library" # This is a folder under HPC Admin's central control
## Identify which paths to keep. In my case, it kept R's original library but removed link to my documents. Find R-Home path using R.home() or Sys.getenv("R_HOME"). 
R.home()
[1] "/software/R/R-3.4.1/lib64/R" # This is read every time R kernel starts. Therefore, any modification will be persistent to every run of R
## Keep the second lib path, by removing all lib paths but the second path
.libPaths(.libPaths()[2])
.libPaths()
[1] "/software/R/R-3.4.1/lib64/R/library"

# Delete the rlang package folder under the user-created path
unlink("/mnt/backedup/home/lunC/R/x86_64-pc-linux-gnu-library/3.4/rlang", recursive = TRUE) # recursive=TRUE deletes a directory

# Try loading dplyr package again
library(dplyr)

Attaching package: ‘dplyr’

The following objects are masked from ‘package:stats’:

    filter, lag

The following objects are masked from ‘package:base’:

    intersect, setdiff, setequal, union

Handle errors with `tryCatch()`

# Step 1: copy this 
tryCatch({},error=function(e){cat("ERROR :",conditionMessage(e), "\n")})
# Step 2: add code that is likely to result in error to the {}

`Error in [[.default(group, 1) : subscript out of bounds` It means you're trying to get something, say the column you are subsetting has a varying number of delimiter _ but your search pattern is built without considering this.

Subscript out of bounds - general definition and solution?

`mutate_each()` is deprecated. Use `mutate_all()`, `mutate_at()` or `mutate_if()` instead. To map `funs` over a selection of variables, use `mutate_at()`

# A dataset to work on
PRS_everDrug1to10_CUD <- read.table(paste0(locPheno,"pheno2drugUseCUD-IDremapped_standardised-IDremapped-PRS-GSCAN.txt")
                                 ,header = T
                                 , sep=" "
                                 , stringsAsFactors = F
                                 , na.strings=c('.','NA'))
# The data structure:                                 
> str(PRS_everDrug1to10_CUD)
'data.frame':	2463 obs. of  76 variables:
 $ FAMID       : chr  "80020" "80020" "80021" "80030" ...
 $ ID          : int  8002001 8002002 8002102 8003001 8003002 8003101 8003102 8003201 8003202 8003301 ...
 $ FATHERID    : chr  "8002003" "8002003" "8002103" "8003003" ...
 $ MOTHERID    : chr  "8002004" "8002004" "8002104" "8003004" ...
 $ GENDER      : int  1 2 2 1 1 1 1 2 2 1 ...
 $ famID       : int  80020 80020 80021 80030 80030 80031 80031 80032 80032 80033 ...
 $ wave        : chr  "NU2" "NU3" "NU3" "NU2" ...
 $ ZYGOSITY    : int  6 6 1 4 4 2 2 1 1 6 ...
 $ everDrug1   : int  0 0 NA 0 0 1 0 0 0 0 ...
 $ everDrug2   : int  0 0 NA 0 0 1 0 0 0 0 ...
 $ everDrug3   : int  0 0 NA 1 0 0 0 0 0 0 ...
 $ everDrug4   : int  0 0 NA 0 1 0 0 0 0 0 ...
 $ everDrug5   : int  0 0 NA 0 0 1 0 1 0 0 ...
 $ everDrug6   : int  0 0 NA 0 0 0 0 0 0 0 ...
 $ everDrug7   : int  0 0 NA 0 0 1 0 1 0 0 ...
 $ everDrug8   : int  0 0 NA 0 0 1 0 0 0 0 ...
 $ everDrug9   : int  0 0 NA 0 0 0 0 0 0 0 ...
 $ everDrug10  : int  0 0 NA 0 0 0 0 0 0 0 ...
 $ CUD         : int  NA NA NA 3 NA 10 9 NA NA NA ...
 $ age         : num  30.3 33.8 33.1 30.6 30.7 ...
 $ nSEX        : int  1 2 2 1 1 1 1 2 2 1 ...
 $ ageSq       : num  920 1141 1095 935 941 ...
 $ sexAge      : num  30.3 67.5 66.2 30.6 30.7 ...
 $ sexAgeSq    : num  920 2281 2190 935 941 ...
 $ DOB         : chr  "15JUL1980" "15JUL1980" "20OCT1979" "08MAR1980" ...
 $ GSCAN.ai.S1 : num  -0.515 -1.141 -1.313 -1.185 -1.134 ...
 $ GSCAN.ai.S2 : num  0.325 -2.008 -0.627 0.366 -0.731 ...
 $ GSCAN.ai.S3 : num  0.99 -0.252 -2.104 -1.147 -1.202 ...
 $ GSCAN.ai.S4 : num  1.505 3.152 -0.597 -1.691 -0.385 ...
 $ GSCAN.ai.S5 : num  0.668 2.307 -2.531 -1.427 -0.707 ...
 $ GSCAN.ai.S6 : num  0.922 1.447 -1.837 -1.418 -1.357 ...
 $ GSCAN.ai.S7 : num  -0.249 0.978 -3.346 -2.074 -2.022 ...
 $ GSCAN.ai.S8 : num  -0.302 1.045 -3.165 -2.128 -1.888 ...
 $ GSCAN.cpd.S1: num  -0.9651 0.799 -0.1756 0.1013 -0.0447 ...
 $ GSCAN.cpd.S2: num  -0.6752 0.8222 0.5144 -0.0802 -0.379 ...
 $ GSCAN.cpd.S3: num  -0.676 0.218 -0.765 -0.141 -0.543 ...
 $ GSCAN.cpd.S4: num  -0.9386 3.2686 -0.577 0.0432 -0.2242 ...
 $ GSCAN.cpd.S5: num  -0.095 4.892 -0.495 0.124 -0.11 ...
 $ GSCAN.cpd.S6: num  0.276 5.415 -0.456 0.039 -0.173 ...
 $ GSCAN.cpd.S7: num  0.2396 4.6077 -0.336 0.3825 -0.0603 ...
 $ GSCAN.cpd.S8: num  0.138 4.398 -0.294 0.308 -0.247 ...
 $ GSCAN.dpw.S1: num  0.118 -0.474 -0.551 -2.615 0.184 ...
 $ GSCAN.dpw.S2: num  0.201 0.875 0.913 -1.487 0.314 ...
 $ GSCAN.dpw.S3: num  -1.583 -0.446 1.746 -0.259 0.44 ...
 $ GSCAN.dpw.S4: num  -0.162 0.621 1.23 -1.076 0.286 ...
 $ GSCAN.dpw.S5: num  1.334 1.184 -0.477 -1.833 0.188 ...
 $ GSCAN.dpw.S6: num  1.535 0.335 -1.644 -1.583 0.319 ...
 $ GSCAN.dpw.S7: num  1.69 0.174 -1.82 -1.467 0.221 ...
 $ GSCAN.dpw.S8: num  1.7488 -0.0555 -2 -1.3446 0.1773 ...
 $ GSCAN.sc.S1 : num  -0.7893 -1.171 -2.5624 -0.077 -0.0439 ...
 $ GSCAN.sc.S2 : num  -0.1717 -1.4473 -2.1254 0.304 -0.0508 ...
 $ GSCAN.sc.S3 : num  -0.858 -1.781 3.678 0.811 0.543 ...
 $ GSCAN.sc.S4 : num  -0.953 0.537 1.975 1.286 0.857 ...
 $ GSCAN.sc.S5 : num  -0.642 1.786 0.866 1.275 1.037 ...
 $ GSCAN.sc.S6 : num  -0.243 2.205 1.777 0.956 0.71 ...
 $ GSCAN.sc.S7 : num  0.166 2.894 1.907 0.728 0.263 ...
 $ GSCAN.sc.S8 : num  0.126 2.791 1.814 0.843 0.334 ...
 $ GSCAN.si.S1 : num  -1.273 -1.833 0.667 0.166 -0.717 ...
 $ GSCAN.si.S2 : num  -1.541 -2.335 0.444 0.276 -0.193 ...
 $ GSCAN.si.S3 : num  -2.082 -2.688 -1.063 -0.764 -0.981 ...
 $ GSCAN.si.S4 : num  -2.4386 -2.2505 -1.3685 -1.0822 -0.0356 ...
 $ GSCAN.si.S5 : num  -1.7702 -2.1189 -3.2785 -1.4399 0.0188 ...
 $ GSCAN.si.S6 : num  -1.622 -3.822 -2.089 -1.53 -0.833 ...
 $ GSCAN.si.S7 : num  -1.09 -3.89 -1.89 -2.13 -1.27 ...
 $ GSCAN.si.S8 : num  -1.11 -4.07 -1.71 -2.12 -1.24 ...
 $ PC1         : num  -0.0125 -0.0124 -0.0128 -0.0129 -0.0132 -0.0122 -0.0122 -0.0132 -0.0132 -0.0126 ...
 $ PC2         : num  0.0088 0.0092 0.0091 0.0087 0.0088 0.0082 0.0082 0.0087 0.0087 0.0079 ...
 $ PC3         : num  0.0013 0.0027 0.0005 0.0006 0.0036 -0.0004 -0.0004 0.0052 0.0052 0.0021 ...
 $ PC4         : num  -0.0003 -0.0031 0.0005 -0.0021 -0.0015 -0.0079 -0.0079 -0.0012 -0.0012 -0.0005 ...
 $ PC5         : num  -0.0031 -0.0055 -0.004 -0.0051 -0.0011 -0.0011 -0.0011 -0.003 -0.003 -0.0016 ...
 $ PC6         : num  -0.0071 -0.0061 -0.0025 -0.0066 -0.0073 -0.012 -0.012 -0.0088 -0.0088 -0.0063 ...
 $ PC7         : num  0.0009 0.0009 -0.0004 0.0024 -0.0004 0.0029 0.0029 -0.0021 -0.0021 -0.0009 ...
 $ PC8         : num  -0.0035 -0.0018 -0.0067 -0.007 -0.0032 -0.0004 -0.0004 0.0018 0.0018 -0.0004 ...
 $ PC9         : num  -0.0005 -0.004 0.0012 -0.0055 -0.004 0.0009 0.0009 -0.0054 -0.0054 0.0009 ...
 $ PC10        : num  -0.0018 -0.0015 0.0019 0.0004 -0.0011 0.0024 0.0024 0.0018 0.0018 0.0038 ...
 $ impCov      : int  0 0 0 0 0 0 0 0 0 0 ...

# Multiply every column that starts with GSCAN with nSEX, creating 40 new columns for the multiplication
library(dplyr)

# Multiply every column that starts with GSCAN (PRS columns) with nSEX and suffix newly created columns with nSEX
# To change the suffix "_nSEX" to prefix "sex_", use rename_at()
PRS_everDrug1to10_CUD2 <- PRS_everDrug1to10_CUD %>% mutate_at(vars(starts_with("GSCAN")),.funs= funs(nSEX= .*nSEX)) %>%
                            rename_at(vars(contains("_nSEX")),funs(paste("sex", gsub("_nSEX", "", .), sep = "_")))

# Compare an old column with a new column
> head(PRS_everDrug1to10_CUD2$GSCAN.ai.S1)
[1] -0.5150501 -1.1410706 -1.3126117 -1.1851313 -1.1344985 -0.6225980
> head(PRS_everDrug1to10_CUD2$sex_GSCAN.ai.S1)
[1] -0.5150501 -2.2821412 -2.6252234 -1.1851313 -1.1344985 -0.6225980
>

`Error in scan(file = file, what = what, sep = sep, quote = quote, dec = dec, : line 194 did not have 14 elements` `read.table` wants to return a data.frame, which must have an element in each column. Therefore R expects each row to have the same number of elements and it doesn't fill in empty spaces by default. Try read.table("/PathTo/file.csv" , fill = TRUE ) to fill in the blanks.

Confusing error in R: Error in scan(file, what, nmax, sep, dec, quote, skip, nlines, na.strings, : line 1 did not have 42 elements) [duplicate]

`Error in full_join_impl(x, y, by$x, by$y, suffix$x, suffix$y, check_na_matches(na_matches)) : std::bad_alloc` means you are running out of RAM

Using swap in R

`Error: This function should not be called directly` or `Error in n() : This function should not be called directly` Both dplyr and plyr have the functions summarise/summarize. The error is due to the conflict of the same function of different packages

dplyr: “Error in n(): function should not be called directly”

# Bad
data %>% 
    filter(pvalue2sided < signi_threshod) %>%
        group_by_(.dots=c("name_fixEffect_trait","phenotype")) %>%
            summarise(count=n())
# Good
data %>% 
    filter(pvalue2sided < signi_threshod) %>%
        group_by_(.dots=c("name_fixEffect_trait","phenotype")) %>%
            dplyr::summarise(count=n())

`Error in fix.by(by.x, x) : 'by' must specify a uniquely valid column`

Merge r brings error “'by' must specify uniquely valid columns”

# Wrong
merge(baseGscan_joined,binary.var.names.labels,by.x =variable, by.y =var.name, all.x = TRUE)

# Correct
merge(baseGscan_joined,binary.var.names.labels,by.x ="variable", by.y ="var.name", all.x = TRUE)

`Warning messages: In cor(variable1, variable2, method = correlation_method, use = "complete.obs") : the standard deviation is zero`

Remove rows with all or some NAs (missing values) in data.frame

# Subset non-missing values from both variable1 and variable2 before calculating correlation between them
# Calculate phenotypic correlation or correlation between any 2 variables
  for (i in 1:length(input.data)){
    for (j in 1:length(input.data)){      
      # Select observations where both variable 1 and variable 2 are non-missing
      input.data.complete.cases <- input.data[complete.cases(input.data[,c(i,j)]),]      
      # Get variable 1
      variable1 <- input.data.complete.cases[,i]      
      # Get variable 2
      variable2 <- input.data.complete.cases[,j]      
      # Compute correlation
      correlation <- cor(variable1,variable2,method=correlation.method,use = "complete.obs")      
      # Print the result on R console
      cat(correlation.method, "correlation between ", colnames[i], " and ", colnames[j], " is", correlation, "\n")      
      phenotypic.correlation.matrix[i,j] <- correlation
    }
  }

`ERROR converting summary statistics` in LD score regression is possibly caused by one of the columns are empty

*Error converting summary statistics

`Error in model.frame.default(formula = cbind(y0 = 1 - y, y1 = y) ~ cutyhat) : variable lengths differ (found for 'cutyhat')`

hoslem.test(na.omit(joined_rm2_ID2rm[,"X20453_0_0_recoded"]), fitted(mod_cann_pyos),g=10)

`Error in` contrasts<-`(`tmp`, value = contr.funs[1 + isOF[nn]]) : contrasts can be applied only to factors with 2 or more levels` occurs when there is not enough variation in the dependent variable with only one value. So, you need to drop that variable, irrespective of whether that is numeric or character or factor variable.

*Error in contrasts when defining a linear model in R

# Logistic regression: Effect of exposure trait on outcome trait

## Set up covariates
formula_covariates="age + TDI + overall_health_rating + factor(inferred.sex) + factor(qualification_edu_6138)+factor(X20116_recodeFinal)"

## Set up dependent and independent variables
y=grep("X20453_0_0_recoded",names(joined_rm2_ID2rm),value = T)
x1=grep("all_coffee_cpd",names(joined_rm2_ID2rm),value = T)
x2=grep("complete_alcohol_unitsweekly",names(joined_rm2_ID2rm),value = T)
x3=grep("X3436_recodeMean",names(joined_rm2_ID2rm),value = T)

## Set up formula
formual_cann_coff <- paste0(y," ~ ", x1," +", formula_covariates)
formual_cann_sdpw <- paste0(y," ~ ", x2," +", formula_covariates)
formual_cann_aass <- paste0(y," ~ ", x3," +", formula_covariates)

# Run logistic regression (models that are working)
mod_cann_coff <- glm(formula= formual_cann_coff,data=joined_rm2_ID2rm,family=binomial(link = "logit"))
mod_cann_spdw <- glm(formula= formual_cann_sdpw,data=joined_rm2_ID2rm,family=binomial(link = "logit"))

summary(mod_cann_coff)
# Coefficients:
#                                   Estimate Std. Error z value Pr(>|z|)    
# (Intercept)                       6.330231   0.103336  61.259  < 2e-16 ***
#   all_coffee_cpd                  -0.029542   0.005432  -5.438 5.39e-08 ***
#   age                             -0.106128   0.001625 -65.329  < 2e-16 ***
#   TDI                              0.117505   0.003944  29.796  < 2e-16 ***
#   overall_health_rating           -0.022334   0.016544  -1.350    0.177    
# factor(inferred.sex)1           -0.357984   0.023277 -15.379  < 2e-16 ***
#   factor(qualification_edu_6138)2 -0.555283   0.032778 -16.941  < 2e-16 ***
#   factor(qualification_edu_6138)3 -1.114870   0.030114 -37.022  < 2e-16 ***
#   factor(qualification_edu_6138)4 -1.616490   0.062381 -25.913  < 2e-16 ***
#   factor(qualification_edu_6138)5 -1.162332   0.051091 -22.750  < 2e-16 ***
#   factor(qualification_edu_6138)6 -0.943528   0.056040 -16.837  < 2e-16 ***
#   factor(X20116_recodeFinal)2      0.328791   0.030763  10.688  < 2e-16 ***
# ---
# Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

summary(mod_cann_spdw)
# Coefficients:
#                                     Estimate  Std. Error z value Pr(>|z|)    
# (Intercept)                        6.0543672  0.1002092  60.417   <2e-16 ***
#   complete_alcohol_unitsweekly     0.0089900  0.0006606  13.609   <2e-16 ***
#   age                             -0.1065484  0.0015752 -67.640   <2e-16 ***
#   TDI                              0.1183401  0.0038145  31.023   <2e-16 ***
#   overall_health_rating           -0.0124434  0.0160451  -0.776    0.438    
# factor(inferred.sex)1           -0.2521848  0.0235726 -10.698   <2e-16 ***
#   factor(qualification_edu_6138)2 -0.5448068  0.0316619 -17.207   <2e-16 ***
#   factor(qualification_edu_6138)3 -1.1059661  0.0291064 -37.997   <2e-16 ***
#   factor(qualification_edu_6138)4 -1.6050344  0.0607355 -26.427   <2e-16 ***
#   factor(qualification_edu_6138)5 -1.1467857  0.0496522 -23.096   <2e-16 ***
#   factor(qualification_edu_6138)6 -0.9088833  0.0540306 -16.822   <2e-16 ***
#   factor(X20116_recodeFinal)2      0.2888456  0.0296775   9.733   <2e-16 ***
#   ---
#   Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

# Run logistic regression (models that are NOT working)
mod_cann_aass <- glm(formula= formual_cann_aass,data=joined_rm2_ID2rm,family=binomial(link = "logit"))
# Error in `contrasts<-`(`*tmp*`, value = contr.funs[1 + isOF[nn]]) : 
#   contrasts can be applied only to factors with 2 or more levels
summary(joined_rm2_ID2rm)
# X20453_0_0_recoded X3436_recodeMean
# Min.   :0.00       Min.   : 5.0    
# 1st Qu.:0.00       1st Qu.:15.0    
# Median :0.00       Median :16.0    
# Mean   :0.22       Mean   :17.7    
# 3rd Qu.:0.00       3rd Qu.:19.0    
# Max.   :1.00       Max.   :69.0    
# NA's   :243314     NA's   :333374

t <- subset(joined_rm2_ID2rm,(!is.na(joined_rm2_ID2rm[,x3])),select=c(y,x3))
table(t$X20453_0_0_recoded)
#    0    1 
# 3127 2566

`dplyr: Error in n(): function should not be called directly` Possibly you have dplyr and plyr loaded in the same session. Both dplyr and plyr have the functions summarise/summarize. You may have a conflict between plyr and dplyr. You can unload the plyr package and load dplyr package.

Both dplyr and plyr have the functions summarise/summarize.

dplyr: “Error in n(): function should not be called directly”

# 
detach("package:reshape2", unload=TRUE) 
detach("package:plyr", unload=TRUE) 
library(dplyr)

`cat: write error: Broken pipe` occurred while using CompileProfileFiles.sh

# Here one of the multiple jobs had walltime shorter than the runtime. But this is not the cause to the error
[lunC@hpcpbs01 Scripts]$ cat /mnt/lustre/working/lab_nickm/lunC/PRS_UKB_201711/allelicScoresCompiled/output/uniqSNPs_from_metaDataQCed-Release8-HRCr1.1_AND_SNP-rsNum_from_all-QCed_GWAS-GSCAN/innerJoinedSNPsByCHRBP_metaDataQCed-Release8-HRCr1.1_AND_si-noQIMR-noBLTS.ambiguSNPRemoved.subset/dosageFam_Release8_HRCr1.1/pbs_output/sumPRS_Release8_HRCr1.1_innerJoinedSNPsByCHRBP_metaDataQCed-Release8-HRCr1.1_AND_si-noQIMR-noBLTS.ambiguSNPRemoved.subset_S5.pbs.err
=>> PBS: job killed: walltime 3627 exceeded limit 3600
-bash: line 1:  9919 Terminated              /var/spool/PBS/mom_priv/jobs/5674535.hpcpbs02.SC

`Error in plot.new() : figure margins too large` first appears. Then `Error in ... plot.new has not been called yet` appears from every plotting function

# The error occurred when exported plot size is too small
plotDimensionRow=1
plotDimensionCol=1
png(file=outputFigFilePath
    ,width = plotDimensionRow*10
    ,height =plotDimensionCol*10 )

# The error is gone when exported plot size enlarges
plotDimensionRow=1
plotDimensionCol=1
png(file=outputFigFilePath
    ,width = plotDimensionRow*450
    ,height =plotDimensionCol*450 )

`foreach error “could not find function ”%do%“”` Each of the parallel workers operates in a clean R session, so you have to load the foreach package in each worker. Try adding .packages="foreach" to your first line

foreach error “could not find function ”%do%“

# Code working
x <- foreach(i=1:length(filePath_clumpedSNP_GSCAN_test), .combine='rbind', .packages="foreach") %dopar% {
        foreach(j=1:length(pThresholds), .combine='rbind') %do% {
    }
}

# Code not working
x <- foreach(i=1:length(filePath_clumpedSNP_GSCAN_test), .combine='rbind' ) %dopar% {
        foreach(j=1:length(pThresholds), .combine='rbind') %do% {
    }
}

`Error in plot.window(xlim, ylim, log = log, ...) : need finite 'ylim' values`

*R: need finite 'ylim' values in function

# The error occurred when the range() read NA. Add na.rm=T as the following
R2_min= range(subplotDataAll$R2,na.rm = TRUE)[1]*100  
R2_max= range(subplotDataAll$R2,na.rm = TRUE)[2]*100

An example of incorrectly specifying for loops

# Incorrect way to write iteration
for (i in 1:length(phenoNames)*length(PRSpheno)){
actions
} 

# Correctly write iteration. Put the multiplication in the ()
for (i in 1:(length(phenoNames)*length(PRSpheno))){
actions
}

Error in \[.data.frame`(dataPhenoPRS, , fixEffect) : undefined columns selected This usually results from the variable name is not exactly matched between a data set and another data set that the first data set looks for

# The case of "ImpCov" doesn't match
if (i >= 401 & i <=480){
    categoricalCovars=c("wave","nSEX","ImpCov")
    } else {
    categoricalCovars=c("wave","wave_dup","nSEX","ImpCov") 
    }
# the case of "impCov" in the data that the code looks for
for (fixEffect in allFixedEffectVar){
   dfPart3$R2[which(dfPart3$rowNames== fixEffect)] <- (dfPart3$fix_eff[which(dfPart3$rowNames== fixEffect)]/sd(x=dataPhenoPRS[,depVarName],na.rm= T)*sd(dataPhenoPRS[,fixEffect],na.rm= T))**2
  }

Errors, warnings, issues in R programming

Error in get_oauth_sig() : OAuth has not been registered for this session

ust be a double vector, not a primitive function Call rlang::last_error() to see a backtrace.` Error arises because column c is not found in the data.frame

> detach("package:plyr", unload=TRUE) Error in detach("package:plyr", unload = TRUE) : invalid 'name' argument

Error in filter_impl(.data, quo) : Evaluation error: as_dictionary()is defunct as of rlang 0.3.0. Please useas_data_pronoun() instead. The error occurred when running dplyr::filter() The error is gone after HPC Admin updated the package dplyr

Handle errors with tryCatch()

Error in [[.default(group, 1) : subscript out of bounds It means you're trying to get something, say the column you are subsetting has a varying number of delimiter _ but your search pattern is built without considering this.

mutate_each() is deprecated. Use mutate_all(), mutate_at() or mutate_if() instead. To map funs over a selection of variables, use mutate_at()

Error in full_join_impl(x, y, by$x, by$y, suffix$x, suffix$y, check_na_matches(na_matches)) : std::bad_alloc means you are running out of RAM

Error: This function should not be called directly or Error in n() : This function should not be called directly Both dplyr and plyr have the functions summarise/summarize. The error is due to the conflict of the same function of different packages

Error in fix.by(by.x, x) : 'by' must specify a uniquely valid column

Warning messages: In cor(variable1, variable2, method = correlation_method, use = "complete.obs") : the standard deviation is zero

ERROR converting summary statistics in LD score regression is possibly caused by one of the columns are empty

Error in model.frame.default(formula = cbind(y0 = 1 - y, y1 = y) ~ cutyhat) : variable lengths differ (found for 'cutyhat')

dplyr: Error in n(): function should not be called directly Possibly you have dplyr and plyr loaded in the same session. Both dplyr and plyr have the functions summarise/summarize. You may have a conflict between plyr and dplyr. You can unload the plyr package and load dplyr package.

cat: write error: Broken pipe occurred while using CompileProfileFiles.sh

Error in plot.new() : figure margins too large first appears. Then Error in ... plot.new has not been called yet appears from every plotting function

foreach error “could not find function ”%do%“” Each of the parallel workers operates in a clean R session, so you have to load the foreach package in each worker. Try adding .packages="foreach" to your first line

Error in plot.window(xlim, ylim, log = log, ...) : need finite 'ylim' values