Chapter 12 Coding and Development
12.1 Installation and Packages
12.1.1 R Installation and Set-Up
Install R new, or update an existing R installation.
Go back to fan’s REconTools research support package, R4Econ examples page, PkgTestR packaging guide, or Stat4Econ course page.
12.1.1.1 Uninstall r
Uninstall R, RStudio and RTools from Windows “Programs and Features” menu. After uninstaller finishes, check in the libPath folders to see if there are still stuff there, delete all, delete the folders.
If R was installed in a virtual environment, delete the environment. Otherwise, use system’s uninstaller. To check on this, from terminal/command-prompt for each virtual environment, type R, and .libPaths() to find paths. Do so inside conda main, outside of conda main, inside different environments, find all paths.
Inside R:
## [1] "C:/Users/fan/AppData/Local/R/win-library/4.3"
## [2] "C:/Program Files/R/R-4.3.1/library"Check which conda env is installed, if there is an env installed for R.
For Linux and for unsintalling inside conda:
# Exit Conda
conda deactivate
# where is R installed outside of Conda
which R
# /usr/bin/R
# To remove all
sudo apt-get remove r-base
sudo apt-get remove r-base-core
# Inside Conda base
conda activate
# Conda r_env
conda activate r_envr
# Where is it installed?
which R
# /home/wangfanbsg75/anaconda3/bin/R
conda uninstall r-base12.1.1.2 Install R
12.1.1.2.1 Install R for the First Time
- download R
- for debian: Johannes Ranke. For Linux/Debian installation, crucial to update the source.list to include sources that have more recent versions of R. If not, will get very old R versions that is not compatible with many packages.
- add R to path for Windows. In Windows Path, add for example: C:/Program Files/R/R-3.6.2/bin/x64/ and C:/Rtools/bin
- Install Rtools for building R packages.
- download R-studio
- Open R-studio and auto-detect R
- Install additional packages
12.1.1.2.2 Linux R Install
For linux/Debian, to Install latest R:
# Go to get latesdebian latest r sources.list
cat /etc/apt/sources.list
# Install this First (should already be installed)
sudo apt install dirmngr
# Debian R is maintained by Johannes Ranke, copied from https://cran.r-project.org/bin/linux/debian/:
apt-key adv --keyserver keys.gnupg.net --recv-key 'E19F5F87128899B192B1A2C2AD5F960A256A04AF'
# Add to source.list, for debian stretch (9)
# sudo su added for security issue as super-user
sudo su -c "sudo echo 'deb http://cloud.r-project.org/bin/linux/debian stretch-cran35/' >> /etc/apt/sources.list"
# if added wrong lines, delete 3rd line
sudo sed '3d' /etc/apt/sources.list
# Update and Install R, should say updated from cloud.r
sudo apt-get update
sudo apt-get install r-base r-base-dev
# Also install these, otherwise r-packages do not install
# libxml2 seems need for tidymodels
sudo apt-get install libcurl4-openssl-dev
sudo apt-get install libssl-dev
sudo apt-get install libxml2-dev12.1.1.2.3 Update R on Windows
First, use the updateR() function from the installr package.
- On windows, install the installr package, and use updateR()
- At the end, will ask if want to move all old packages to new R directory
New R will have new package directory, could keep all in old, should copy all to new, and not keep old. Can choose to copy all old packages to new folder, but still keep old packages in prior folder as they were
# https://www.r-project.org/nosvn/pandoc/installr.html
install.packages('installr')
# update R from inside R (not Rstudio)
require(installr)
# this will open dialog boxes to take you through the steps.
updateR()
# Set Rstudio to the Latest RSecond, after updating, might go into “Apps and Features” on Windows to unstaill the previous R version.
Third, update RTools. Uninstall RTools. Installation can be very large.
Fourth, update RStudio. Upon opening RStudio, if prior installations of R have been uninstalled, RStudio will auto-detect as ask if the latest version of R should be used. Choose yes. Upon entering RStudio, if there are updates, might prompt to RStudio website to download and update.
Fifth, follow the package installation directions below to update that.
12.1.1.2.4 R Add to Path
To be able to use R via command line, make sure Windows knows where the path to R.exe is.
First, find where the installed R.exe path is. Open up the R installation, and then check path as below.
ls_spn_paths <- .libPaths()
print(ls_spn_paths)
# "C:/Users/fan/AppData/Local/R/win-library/4.2"
# "C:/Program Files/R/R-4.2.1/library"Second, given the path found, the R.exe is at “C:/Program Files/R/R-4.2.1/bin”. So now, in windows, System Properties -> Advanced -> Environment Variables -> System Variables -> Path -> Edit -> New -> Paste “C:/Program Files/R/R-4.2.1/bin”
Third, now open up command-prompt/terminal/git-bash, enter R, this will take us into the R console via command-line.
12.1.1.3 R Package Installations
12.1.1.3.1 Update R Install Directory
After installing R, change the path sequence so that packages install for all users.
ls_spn_paths <- .libPaths()
print(ls_spn_paths)
# [1] "C:/Users/fan/AppData/Local/R/win-library/4.2" "C:/Program Files/R/R-4.2.1/library"
ls_spn_paths <- c(ls_spn_paths[2], ls_spn_paths[1])
.libPaths(ls_spn_paths)
ls_spn_paths <- .libPaths()
print(ls_spn_paths)
# [1] "C:/Program Files/R/R-4.2.1/library" "C:/Users/fan/AppData/Local/R/win-library/4.2"12.1.1.3.2 Install vearious directory
After updating R, sometimes, old packages are not copied over to new directory, so need to reinstall all packages.
Having set the directories earlier so that packages do not install in user’s personal folder, but the library folder where the R version is installed, we can find all installed packages in the C:/Program Files/R/R-4.2.1/library folder.
# Install RTools First!
# https://cran.r-project.org/bin/windows/Rtools/
# Install system tools
install.packages(c("backports"))
# Install tidyverse
install.packages(c("tidyverse", "tidymodels", "vroom"))
# Install Packaging tools
install.packages(c("devtools", "pkgdown", "roxygen2", "bookdown", "knitr", "kableExtra", "formatR", "revealjs"))
# Install Statistics models
install.packages(c("AER", "minpack.lm"))
install.packages(c("quantreg"))
# Install Tools to Work with Other Packages
# matconv: converts matlab programs to R
install.packages(c("reticulate", "JuliaCall", "matconv"))
install.packages(c("matconv"))
# for reticulate errors, install directly from: devtools::install_github("rstudio/reticulate")
# Install Paralell Tools
install.packages(c("parallel", "doParallel", "foreach"))
# Install personal Packages
devtools::install_github("fanwangecon/REconTools")
devtools::install_github("fanwangecon/PrjOptiAlloc")
# Stata in Rmd
# devtools::install_github("Hemken/Statamarkdown")
# VScode integration and also sublime r-ide
install.packages("languageserver")Temp Installs:
12.1.1.4 R Tests
Test the following file to see if we can execute a R file. Do it inside r_env and inside a r session.
# # A simple file with summary statistics using tidyverse
# source('C:/Users/fan/R4Econ/summarize/dist/htmlpdfr/fst_hist_onevar.R')
# source('G:/repos/R4Econ/summarize/dist/htmlpdfr/fst_hist_onevar.R')
# # Another simple file with summary statistics using tidyverse
# source('C:/Users/fan/R4Econ/support/tibble/htmlpdfr/fs_tib_basics.R')
# source('G:/repos/R4Econ/support/tibble/htmlpdfr/fs_tib_basics.R')
# # A file involving estimation
# source('C:/Users/fan/R4Econ/optimization/cesloglin/htmlpdfr/fst_ces_plan_linlog.R')
#
# C:/Users/fan/R4Econ/summarize/dist/fst_hist_onevar.Rmd
# C:/Users/fan/R4Econ/support/tibble/fs_tib_basics.Rmd
# C:/Users/fan/R4Econ/optimization/cesloglin/fst_ces_plan_linlog.Rmd12.1.1.6 Running R Inside VSCode
Rstudio seems laggy sometimes, nice to be able to run R from VSCode, use VSCode as an alternative editor.
To Run .R scripts from inside VSCode. Here is the offical guide: R in Visual Studio Code.
- Install R following prior steps.
- Make sure that the package languageserver is installed, check “require(languageserver)”.
- Install R Extension for Visual Studio Code
- change setting for “r.rterm.windows” to “C:/Program Files/R/R-4.2.1/bin/R.exe”
- Click File -> Open Folder -> Select the folder where the “.R” file to run is located at, this way, the Terminal will be directred to the folder that is currently open, rather than home default for example. By default, the terminal will open at the folder that is opened in the Explorer
- Run individual files in the folder just opened.
12.1.1.6.1 RMD in VSCode
Steps for RMD:
- VSCode already has default markdown editor
- Install Markdown Preview ehance, which generates a table of content bar on the side, and has math preview correclty
- In file, F1, and type markdown preview and open up preview
12.1.1.6.2 RMD and MARKDOWN File Associations
Change File Extension Association
- Working with RMD file, sometimes want to preview as MD file to view equations, sometimes want to view as RMD file to edit the R code. See How to make VS Code to treat other file extensions as certain language?
- change extension association between md and Rmd for Rmd files for example: “Ctrl + shift + p” and “change language mode” from one file association to another.
- Additionally, if there is a standard association we want for RMD, for it to be markdown for example, can add to JSON settings for file.associations.
"files.associations": {
"*.Rmd": "markdown"
}12.2 Files In and Out
12.2.1 File Path
Go back to fan’s REconTools research support package, R4Econ examples page, PkgTestR packaging guide, or Stat4Econ course page.
12.2.1.1 Check if File or Directory Exists on Computer
Take different string-based actions if some path exists on a computer.
# Check if either of the path exists on the computer
spt_root_computer_a <- 'G:/repos/R4Econ/'
spt_root_computer_b <- 'C:/Users/fan/R4Econ/'
# Checking if directory exists
if (dir.exists(spt_root_computer_a)) {
print(paste(spt_root_computer_a, 'exists'))
} else if (dir.exists(spt_root_computer_b)) {
print(paste(spt_root_computer_b, 'exists', spt_root_computer_a, 'does not exist'))
} else {
print(paste(spt_root_computer_b, spt_root_computer_a, 'both do does not exist'))
}## [1] "C:/Users/fan/R4Econ/ exists G:/repos/R4Econ/ does not exist"Now, we check if a particular file exists.
# Check if either of the files exists on the computer
spn_root_computer_a <- 'G:/repos/R4Econ/index.Rmd'
spn_root_computer_b <- 'C:/Users/fan/R4Econ/index.Rmd'
# Checking if file exists
if (file.exists(spn_root_computer_a)) {
print(paste(spn_root_computer_a, 'exists'))
} else if (file.exists(spn_root_computer_b)) {
print(paste(spn_root_computer_b, 'exists', spn_root_computer_a, 'does not exist'))
} else {
print(paste(spn_root_computer_b, spn_root_computer_a, 'both do does not exist'))
}## [1] "C:/Users/fan/R4Econ/index.Rmd exists G:/repos/R4Econ/index.Rmd does not exist"12.2.1.2 Compose a Path
File Path might contain information related to the file, decompose the file path, keep the final N folder names, to be possibled stored as a variable in the datafile stored inside.
## [1] ".Platform$file.sep=/"spn_file_path = file.path("C:", "Users", "fan", "R4Econ", "amto", "tibble",
"fs_tib_basics.Rmd",
fsep = .Platform$file.sep)
# print
print(spn_file_path)## [1] "C:/Users/fan/R4Econ/amto/tibble/fs_tib_basics.Rmd"12.2.1.3 Substring and File Name
From path, get file name without suffix.
- r string split
- r list last element
- r get file name from path
- r get file path no name
st_example <- 'C:/Users/fan/R4Econ/amto/tibble/fs_tib_basics.Rmd'
st_file_wth_suffix_s <- tail(strsplit(st_example, "/")[[1]],n=1)
st_file_wno_suffix_s <- tools::file_path_sans_ext(basename(st_example))
st_fullpath_nosufx_s <- sub('\\.Rmd$', '', st_example)
st_fullpath_noname_s <- dirname(st_example)
print(paste0('st_file_wth_suffix_s:', st_file_wth_suffix_s))## [1] "st_file_wth_suffix_s:fs_tib_basics.Rmd"## [1] "st_file_wno_suffix_s:fs_tib_basics"## [1] "st_fullpath_nosufx_s:C:/Users/fan/R4Econ/amto/tibble/fs_tib_basics"## [1] "st_fullpath_noname_s:C:/Users/fan/R4Econ/amto/tibble"12.2.1.4 Get Subset of Path Folder Names
File Path might contain information related to the file, decompose the file path, keep the final N folder names, to be possibled stored as a variable in the datafile stored inside.
# Compose together a path
spn_example <- 'C:/Users/fan/R4Econ/amto/tibble/fs_tib_basics.Rmd'
# Replace default system slash, assume spn was generated by system.
ls_srt_folders_file <- strsplit(st_example, .Platform$file.sep)[[1]]
# Keep the last N layers
it_folders_names_to_keep = 2
snm_file_name <- tail(strsplit(st_example, "/")[[1]],n=1)
ls_srt_folders_keep <- head(tail(ls_srt_folders_file, it_folders_names_to_keep+1),
it_folders_names_to_keep)
# Show folder names
print(paste0('snm_file_name:', snm_file_name))## [1] "snm_file_name:fs_tib_basics.Rmd"## [1] "last 2 folders:"## [1] "amto" "tibble"Shorter lines, to make copying easier.
# inputs
it_folders_names_to_keep = 2
spn_example <- 'C:/Users/fan/R4Econ/amto/tibble/fs_tib_basics.Rmd'
# copy these
ls_srt_folders_name_keep <- tail(strsplit(st_example, "/")[[1]], n=it_folders_names_to_keep+1)
snm_file_name <- tail(ls_srt_folders_name_keep, 1)
ls_srt_folders_keep <- head(ls_srt_folders_name_keep, it_folders_names_to_keep)
# print
print(paste0('snm_file_name:', snm_file_name))## [1] "snm_file_name:fs_tib_basics.Rmd"## [1] "last 2 folders:"## [1] "amto" "tibble"12.2.2 Text to File
Go back to fan’s REconTools research support package, R4Econ examples page, PkgTestR packaging guide, or Stat4Econ course page.
12.2.2.1 Latex Table to File
Tabular outputs, text outputs, etc are saved as variables, which could be printed in console. They can also be saved to file for future re-used. For example, latex outputs need to be saved to file.
12.2.2.2 Create a Text File from Strings
st_file <- "\\documentclass[12pt,english]{article}
\\usepackage[bottom]{footmisc}
\\usepackage[urlcolor=blue]{hyperref}
\\begin{document}
\\title{A Latex Testing File}
\\author{\\href{http://fanwangecon.github.io/}{Fan Wang} \\thanks{See information \\href{https://fanwangecon.github.io/Tex4Econ/}{Tex4Econ} for more.}}
\\maketitle
Ipsum information dolor sit amet, consectetur adipiscing elit. Integer Latex placerat nunc orci.
\\paragraph{\\href{https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3140132}{Data}}
Village closure information is taken from a village head survey.\\footnote{Generally students went to schools.}
output:
pdf_document:
pandoc_args: '..//..//_output_kniti_pdf.yaml'
includes:
in_header: '..//..//preamble.tex'
html_document:
pandoc_args: '..//..//_output_kniti_html.yaml'
includes:
in_header: '..//..//hdga.html'
\\end{document}"
print(st_file)## [1] "\\documentclass[12pt,english]{article}\n\\usepackage[bottom]{footmisc}\n\\usepackage[urlcolor=blue]{hyperref}\n\\begin{document}\n\\title{A Latex Testing File}\n\\author{\\href{http://fanwangecon.github.io/}{Fan Wang} \\thanks{See information \\href{https://fanwangecon.github.io/Tex4Econ/}{Tex4Econ} for more.}}\n\\maketitle\nIpsum information dolor sit amet, consectetur adipiscing elit. Integer Latex placerat nunc orci.\n\\paragraph{\\href{https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3140132}{Data}}\nVillage closure information is taken from a village head survey.\\footnote{Generally students went to schools.}\noutput:\n pdf_document:\n pandoc_args: '..//..//_output_kniti_pdf.yaml'\n includes:\n in_header: '..//..//preamble.tex'\n html_document:\n pandoc_args: '..//..//_output_kniti_html.yaml'\n includes:\n in_header: '..//..//hdga.html'\n\\end{document}"12.2.2.3 Open A File and Read Lines
Open and Replace Text in File:
## [1] "\\documentclass[12pt,english]{article}"
## [2] "\\usepackage[bottom]{footmisc}"
## [3] "\\usepackage[urlcolor=blue]{hyperref}"
## [4] "\\begin{document}"
## [5] "\\title{A Latex Testing File}"
## [6] "\\author{\\href{http://fanwangecon.github.io/}{Fan Wang} \\thanks{See information \\href{https://fanwangecon.github.io/Tex4Econ/}{Tex4Econ} for more.}}"
## [7] "\\maketitle"
## [8] "Ipsum information dolor sit amet, consectetur adipiscing elit. Integer Latex placerat nunc orci."
## [9] "\\paragraph{\\href{https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3140132}{Data}}"
## [10] "Village closure information is taken from a village head survey.\\footnote{Generally students went to schools.}"
## [11] "output:"
## [12] " pdf_document:"
## [13] " pandoc_args: '..//..//_output_kniti_pdf.yaml'"
## [14] " includes:"
## [15] " in_header: '..//..//preamble.tex'"
## [16] " html_document:"
## [17] " pandoc_args: '..//..//_output_kniti_html.yaml'"
## [18] " includes:"
## [19] " in_header: '..//..//hdga.html'"
## [20] "\\end{document}"12.2.2.4 Open a File and Replace Some Lines
Append additional strings into the file after html_document with proper spacings:
# Read in Lines from existing file
fileConn <- file(fl_test_tex, "r")
st_file_read <- readLines(fileConn)
close(fileConn)
# Search and Replace String
st_search <- "html_document:"
st_replace <- paste0("html_document:\r\n",
" toc: true\r\n",
" number_sections: true\r\n",
" toc_float:\r\n",
" collapsed: false\r\n",
" smooth_scroll: false\r\n",
" toc_depth: 3")
# Search and Replace
st_file_updated <- gsub(x = st_file_read,
pattern = st_search,
replacement = st_replace)
# Print
print(st_file_updated)## [1] "\\documentclass[12pt,english]{article}"
## [2] "\\usepackage[bottom]{footmisc}"
## [3] "\\usepackage[urlcolor=blue]{hyperref}"
## [4] "\\begin{document}"
## [5] "\\title{A Latex Testing File}"
## [6] "\\author{\\href{http://fanwangecon.github.io/}{Fan Wang} \\thanks{See information \\href{https://fanwangecon.github.io/Tex4Econ/}{Tex4Econ} for more.}}"
## [7] "\\maketitle"
## [8] "Ipsum information dolor sit amet, consectetur adipiscing elit. Integer Latex placerat nunc orci."
## [9] "\\paragraph{\\href{https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3140132}{Data}}"
## [10] "Village closure information is taken from a village head survey.\\footnote{Generally students went to schools.}"
## [11] "output:"
## [12] " pdf_document:"
## [13] " pandoc_args: '..//..//_output_kniti_pdf.yaml'"
## [14] " includes:"
## [15] " in_header: '..//..//preamble.tex'"
## [16] " html_document:\r\n toc: true\r\n number_sections: true\r\n toc_float:\r\n collapsed: false\r\n smooth_scroll: false\r\n toc_depth: 3"
## [17] " pandoc_args: '..//..//_output_kniti_html.yaml'"
## [18] " includes:"
## [19] " in_header: '..//..//hdga.html'"
## [20] "\\end{document}"12.2.3 Rmd to HTML
Go back to fan’s REconTools research support package, R4Econ examples page, PkgTestR packaging guide, or Stat4Econ course page.
12.2.3.1 Search and Find all Files in Repository
Search inside directories, for all files in a repository that have a particular suffix and that don’t contain skip pattern list string items.
# Serch Folder and skip list
spt_roots <- c('C:/Users/fan/R4Econ/amto', 'C:/Users/fan/R4Econ/summarize')
spn_skip <- c('summarize', 'panel', 'support')
# Search and get all Path
ls_sfls <- list.files(path=spt_roots, recursive=T, pattern=".Rmd", full.names=T)
# Skip path if contains words in skip list
if(!missing(spn_skip)) {
ls_sfls <- ls_sfls[!grepl(paste(spn_skip, collapse = "|"), ls_sfls)]
}
# Loop and print
for (spt_file in ls_sfls) {
st_fullpath_nosufx <- tail(strsplit(spt_file, "/")[[1]],n=1)
print(paste0(spt_file, '---', st_fullpath_nosufx))
}## [1] "C:/Users/fan/R4Econ/amto/array/fs_ary_basics.Rmd---fs_ary_basics.Rmd"
## [1] "C:/Users/fan/R4Econ/amto/array/fs_ary_generate.Rmd---fs_ary_generate.Rmd"
## [1] "C:/Users/fan/R4Econ/amto/array/fs_ary_mesh.Rmd---fs_ary_mesh.Rmd"
## [1] "C:/Users/fan/R4Econ/amto/array/fs_ary_string.Rmd---fs_ary_string.Rmd"
## [1] "C:/Users/fan/R4Econ/amto/array/main.Rmd---main.Rmd"
## [1] "C:/Users/fan/R4Econ/amto/list/fs_lst_basics.Rmd---fs_lst_basics.Rmd"
## [1] "C:/Users/fan/R4Econ/amto/list/main.Rmd---main.Rmd"
## [1] "C:/Users/fan/R4Econ/amto/main.Rmd---main.Rmd"
## [1] "C:/Users/fan/R4Econ/amto/matrix/fs_mat_demo_trans.Rmd---fs_mat_demo_trans.Rmd"
## [1] "C:/Users/fan/R4Econ/amto/matrix/fs_mat_generate.Rmd---fs_mat_generate.Rmd"
## [1] "C:/Users/fan/R4Econ/amto/matrix/fs_mat_linear_algebra.Rmd---fs_mat_linear_algebra.Rmd"
## [1] "C:/Users/fan/R4Econ/amto/matrix/main.Rmd---main.Rmd"
## [1] "C:/Users/fan/R4Econ/amto/misc/fs_parse_regex.Rmd---fs_parse_regex.Rmd"
## [1] "C:/Users/fan/R4Econ/amto/misc/main.Rmd---main.Rmd"
## [1] "C:/Users/fan/R4Econ/amto/tibble/fs_tib_basics.Rmd---fs_tib_basics.Rmd"
## [1] "C:/Users/fan/R4Econ/amto/tibble/fs_tib_factors.Rmd---fs_tib_factors.Rmd"
## [1] "C:/Users/fan/R4Econ/amto/tibble/fs_tib_na.Rmd---fs_tib_na.Rmd"
## [1] "C:/Users/fan/R4Econ/amto/tibble/fs_tib_random_draws.Rmd---fs_tib_random_draws.Rmd"
## [1] "C:/Users/fan/R4Econ/amto/tibble/fs_tib_string.Rmd---fs_tib_string.Rmd"
## [1] "C:/Users/fan/R4Econ/amto/tibble/main.Rmd---main.Rmd"12.2.3.2 Search and Find all Git Modified or New Rmd
Search inside directories, for all files in a git repo folder that are new or have been modified. Ignore possible subset of file based on string search.
# Serch Folder and skip list
spt_roots <- c('C:/Users/fan/R4Econ/amto', 'C:/Users/fan/R4Econ/development')
spn_skip <- c('summarize', 'panel', 'support')
ls_sfls <- list.files(path=spt_roots, recursive=T, pattern=".Rmd", full.names=T)
if(!missing(spn_skip)) {
ls_sfls <- ls_sfls[!grepl(paste(spn_skip, collapse = "|"), ls_sfls)]
}
# Loop and print
for (spt_file in ls_sfls) {
spg_check_git_status <- paste0('git status -s ', spt_file)
st_git_status <- toString(system(spg_check_git_status, intern=TRUE))
bl_modified <- grepl(' M ', st_git_status, fixed=TRUE)
bl_anewfile <- grepl('?? ', st_git_status, fixed=TRUE)
bl_nochange <- (st_git_status == "")
if (bl_modified == 1) {
print(paste0('MODIFIED: ', spt_file))
} else if (bl_anewfile == 1) {
print(paste0('A NEW FL: ', spt_file))
} else {
print(paste0('NO CHNGE: ', spt_file))
}
}## [1] "NO CHNGE: C:/Users/fan/R4Econ/amto/array/fs_ary_basics.Rmd"
## [1] "MODIFIED: C:/Users/fan/R4Econ/amto/array/fs_ary_generate.Rmd"
## [1] "NO CHNGE: C:/Users/fan/R4Econ/amto/array/fs_ary_mesh.Rmd"
## [1] "NO CHNGE: C:/Users/fan/R4Econ/amto/array/fs_ary_string.Rmd"
## [1] "NO CHNGE: C:/Users/fan/R4Econ/amto/array/main.Rmd"
## [1] "NO CHNGE: C:/Users/fan/R4Econ/amto/list/fs_lst_basics.Rmd"
## [1] "NO CHNGE: C:/Users/fan/R4Econ/amto/list/main.Rmd"
## [1] "NO CHNGE: C:/Users/fan/R4Econ/amto/main.Rmd"
## [1] "NO CHNGE: C:/Users/fan/R4Econ/amto/matrix/fs_mat_demo_trans.Rmd"
## [1] "NO CHNGE: C:/Users/fan/R4Econ/amto/matrix/fs_mat_generate.Rmd"
## [1] "NO CHNGE: C:/Users/fan/R4Econ/amto/matrix/fs_mat_linear_algebra.Rmd"
## [1] "NO CHNGE: C:/Users/fan/R4Econ/amto/matrix/main.Rmd"
## [1] "MODIFIED: C:/Users/fan/R4Econ/amto/misc/fs_parse_regex.Rmd"
## [1] "NO CHNGE: C:/Users/fan/R4Econ/amto/misc/main.Rmd"
## [1] "NO CHNGE: C:/Users/fan/R4Econ/amto/tibble/fs_tib_basics.Rmd"
## [1] "NO CHNGE: C:/Users/fan/R4Econ/amto/tibble/fs_tib_factors.Rmd"
## [1] "NO CHNGE: C:/Users/fan/R4Econ/amto/tibble/fs_tib_na.Rmd"
## [1] "NO CHNGE: C:/Users/fan/R4Econ/amto/tibble/fs_tib_random_draws.Rmd"
## [1] "NO CHNGE: C:/Users/fan/R4Econ/amto/tibble/fs_tib_string.Rmd"
## [1] "NO CHNGE: C:/Users/fan/R4Econ/amto/tibble/main.Rmd"
## [1] "NO CHNGE: C:/Users/fan/R4Econ/development/inout/_file/rmd/fs_rmd_pdf_html_mod.Rmd"
## [1] "NO CHNGE: C:/Users/fan/R4Econ/development/inout/_file/rmd/fs_rmd_pdf_html_mod_mod.Rmd"
## [1] "NO CHNGE: C:/Users/fan/R4Econ/development/inout/_file/rmd/fs_text_save_mod.Rmd"
## [1] "NO CHNGE: C:/Users/fan/R4Econ/development/inout/_file/rmd/main_mod.Rmd"
## [1] "NO CHNGE: C:/Users/fan/R4Econ/development/inout/fs_path.Rmd"
## [1] "NO CHNGE: C:/Users/fan/R4Econ/development/inout/fs_rmd_pdf_html.Rmd"
## [1] "NO CHNGE: C:/Users/fan/R4Econ/development/inout/fs_text_save.Rmd"
## [1] "NO CHNGE: C:/Users/fan/R4Econ/development/inout/main.Rmd"
## [1] "NO CHNGE: C:/Users/fan/R4Econ/development/install/fs_install_R.Rmd"
## [1] "A NEW FL: C:/Users/fan/R4Econ/development/install/fs_packages_R.Rmd"
## [1] "MODIFIED: C:/Users/fan/R4Econ/development/install/main.Rmd"
## [1] "NO CHNGE: C:/Users/fan/R4Econ/development/main.Rmd"
## [1] "MODIFIED: C:/Users/fan/R4Econ/development/parallel/fs_parallel.Rmd"
## [1] "NO CHNGE: C:/Users/fan/R4Econ/development/parallel/main.Rmd"
## [1] "NO CHNGE: C:/Users/fan/R4Econ/development/python/fs_python_reticulate.Rmd"
## [1] "NO CHNGE: C:/Users/fan/R4Econ/development/python/main.Rmd"
## [1] "NO CHNGE: C:/Users/fan/R4Econ/development/system/fs_system_shell.Rmd"
## [1] "NO CHNGE: C:/Users/fan/R4Econ/development/system/main.Rmd"12.2.3.3 Resave an Existing File with Different Name Different Folder
Given an existing Rmd File, Resave it with a different name (add to name suffix), and then save in a different folder:
- old file: /R4Econ/development/fs_rmd_pdf_html.Rmd
- new file: *R4Econ/development/inout/_file/rmd/fs_rmd_pdf_html_mod.Rmd*
# Serch Folder and skip list
spt_roots <- c('C:/Users/fan/R4Econ/development/inout/')
spn_skip <- c('_main', '_file')
ls_sfls <- list.files(path=spt_roots, recursive=T, pattern=".Rmd", full.names=T)
if(!missing(spn_skip)) {
ls_sfls <- ls_sfls[!grepl(paste(spn_skip, collapse = "|"), ls_sfls)]
}
# Loop and print
for (spt_file in ls_sfls) {
spt_new <- paste0('_file/rmd/')
spn_new <- paste0(spt_new, sub('\\.Rmd$', '', basename(spt_file)), '_mod.Rmd')
print(spt_new)
print(spn_new)
fileConn_rd <- file(spt_file, "r")
st_file_read <- readLines(fileConn_rd)
fileConn_sr <- file(spn_new)
writeLines(st_file_read, fileConn_sr)
close(fileConn_rd)
close(fileConn_sr)
}12.2.3.3.1 Replacment Function Change Markdown Hierarchy and Add to YAML
Given an existing Rmd File, Resave it with a different name, and replace (add in) additional yaml contents.
spn_file = '_file/rmd/fs_rmd_pdf_html_mod.Rmd'
fileConn_sr <- file(spn_file)
st_file <- readLines(fileConn_sr)
# print(st_file)
st_search <- "html_document:
toc: true
number_sections: true
toc_float:
collapsed: false
smooth_scroll: false
toc_depth: 3"
st_replace <- paste0("html_document:
toc: true
number_sections: true
toc_float:
collapsed: false
smooth_scroll: false
toc_depth: 3\n",
" toc: true\n",
" number_sections: true\n",
" toc_float:\n",
" collapsed: false\n",
" smooth_scroll: false\n",
" toc_depth: 3")
st_file_updated <- gsub(x = st_file,
pattern = st_search,
replacement = st_replace)
st_search <- "../../"
st_replace <- paste0("../../../../")
st_file_updated <- gsub(x = st_file_updated,
pattern = st_search,
replacement = st_replace)
st_file_updated <- gsub(x = st_file_updated, pattern = '# ', replacement = '# ')
st_file_updated <- gsub(x = st_file_updated, pattern = '## ', replacement = '## ')
st_file_updated <- gsub(x = st_file_updated, pattern = '### ', replacement = '# ')
spn_file = '_file/rmd/fs_rmd_pdf_html_mod.Rmd'
fileConn_sr <- file(spn_file)
st_file <- writeLines(st_file_updated, fileConn_sr)12.2.3.4 Search and Render Rmd File and Save HTML, PDF or R
- Search files satisfying conditions in a folder
- knit files to HTML (and re-run the contents of the file)
- Save output to a different folder
# Specify Parameters
ar_spt_root = c('C:/Users/fan/R4Econ/amto/array/', 'C:/Users/fan/R4Econ/math/integration')
bl_recursive = TRUE
st_rmd_suffix_pattern = "*.Rmd"
ar_spn_skip <- c('basics', 'integrate', 'main', 'mesh')
ls_bool_convert <- list(bl_pdf=TRUE, bl_html=TRUE, bl_R=TRUE)
spt_out_directory <- 'C:/Users/fan/Downloads/_data'
bl_verbose <- TRUE
# Get Path
ls_sfls <- list.files(path=ar_spt_root,
recursive=bl_recursive,
pattern=st_rmd_suffix_pattern,
full.names=T)
# Exclude Some Files given ar_spn_skip
if(!missing(ar_spn_skip)) {
ls_sfls <- ls_sfls[!grepl(paste(ar_spn_skip, collapse = "|"), ls_sfls)]
}
# Loop over files
for (spn_file in ls_sfls) {
# Parse File Name
spt_file <- dirname(spn_file)
sna_file <- tools::file_path_sans_ext(basename(spn_file))
# Output FIles
spn_file_pdf <- paste0(spt_file, sna_file, '.pdf')
spn_file_html <- paste0(spt_file, sna_file, '.html')
spn_file_R <- paste0(spt_file, sna_file, '.R')
# render to PDF
if (ls_bool_convert$bl_pdf) {
if (bl_verbose) message(paste0('spn_file_pdf:',spn_file_pdf, ', PDF started'))
rmarkdown::render(spn_file, output_format='pdf_document',
output_dir = spt_out_directory, output_file = sna_file)
if (bl_verbose) message(paste0('spn_file_pdf:',spn_file_pdf, ', PDF finished'))
spn_pdf_generated <- paste0(spt_out_directory, '/', spn_file_pdf)
}
# render to HTML
if (ls_bool_convert$bl_html) {
if (bl_verbose) message(paste0('spth_html:',spn_file_html, ', HTML started.'))
rmarkdown::render(spn_file, output_format='html_document',
output_dir = spt_out_directory, output_file = sna_file)
if (bl_verbose) message(paste0('spth_html:',spn_file_html, ', HTML finished.'))
spn_html_generated <- paste0(spt_out_directory, '/', spn_file_html)
}
# purl to R
if (ls_bool_convert$bl_R) {
if (bl_verbose) message(paste0('purl_to:', paste0(spn_file_R, ".R")))
knitr::purl(spn_file, output=paste0(spt_out_directory, '/', sna_file, '.R'), documentation = 1)
spn_R_generated <- paste0(spt_out_directory, '/', sna_file, '.R')
}
# return(list(ls_spt_pdf_generated=ls_spt_pdf_generated,
# ls_spt_html_generated=ls_spt_html_generated,
# ls_spt_R_generated=ls_spt_R_generated))
}12.3 Python with R
12.3.1 Reticulate Basics
Go back to fan’s REconTools research support package, R4Econ examples page, PkgTestR packaging guide, or Stat4Econ course page.
12.3.1.1 Basic Python Tests with RMD
Could specify: python, engine.path = “C:/ProgramData/Anaconda3/envs/wk_pyfan/python.exe”, this is already set inside Rprofile: knitr::opts_chunk$set(engine.path = “C:/ProgramData/Anaconda3/envs/wk_pyfan/python.exe”)
## 212.3.1.2 Install and Python Path
Install reticulate from github directly to get latest version: devtools::install_github(“rstudio/reticulate”)
Check python version on computer:
## python
## "C:\\PROGRA~3\\ANACON~1\\envs\\wk_pyfan\\python.exe"After installing reticulate, load in the library: library(reticulate). With “py_config()” to see python config. First time, might generate “No non-system installation of Python could be found.” and ask if want to install Miniconda. Answer NO.
Correct outputs upon checking py_config():
python: C:/ProgramData/Anaconda3/python.exe
libpython: C:/ProgramData/Anaconda3/python37.dll
pythonhome: C:/ProgramData/Anaconda3
version: 3.7.9 (default, Aug 31 2020, 17:10:11) [MSC v.1916 64 bit (AMD64)]
Architecture: 64bit
numpy: C:/ProgramData/Anaconda3/Lib/site-packages/numpy
numpy_version: 1.19.1
python versions found:
C:/ProgramData/Anaconda3/python.exe
C:/ProgramData/Anaconda3/envs/wk_cgefi/python.exe
C:/ProgramData/Anaconda3/envs/wk_jinja/python.exe
C:/ProgramData/Anaconda3/envs/wk_pyfan/python.exeSet which python to use:
# Sys.setenv(RETICULATE_PYTHON = "C:/programdata/Anaconda3/python.exe")
# Sys.setenv(RETICULATE_PYTHON = "C:/ProgramData/Anaconda3/envs/wk_pyfan/python.exe")
library(reticulate)
# What is the python config
py_config()## python: C:/ProgramData/anaconda3/envs/wk_pyfan/python.exe
## libpython: C:/ProgramData/anaconda3/envs/wk_pyfan/python311.dll
## pythonhome: C:/ProgramData/anaconda3/envs/wk_pyfan
## version: 3.11.7 | packaged by conda-forge | (main, Dec 23 2023, 14:27:59) [MSC v.1937 64 bit (AMD64)]
## Architecture: 64bit
## numpy: C:/ProgramData/anaconda3/envs/wk_pyfan/Lib/site-packages/numpy
## numpy_version: 1.26.2
##
## NOTE: Python version was forced by RETICULATE_PYTHON# set python
# use_python("C:/programdata/Anaconda3/python.exe")
# use_python("C:/ProgramData/Anaconda3/envs/wk_pyfan/python.exe")
use_condaenv('wk_pyfan')
# Sys.which('python')
py_run_string('print(1+1)')## 212.3.1.3 Error
12.3.1.3.1 py_call_impl error
The error appeared when calling any python operations, including “1+1”, resolved after installing reticulate from github: devtools::install_github(“rstudio/reticulate”)
Error in py_call_impl(callable, dots$args, dots$keywords) :
TypeError: use() got an unexpected keyword argument 'warn'12.4 Command Line
12.4.1 Shell and System Commands
Go back to fan’s REconTools research support package, R4Econ examples page, PkgTestR packaging guide, or Stat4Econ course page.
12.4.1.1 Basic Shell Commands
Run basic shell commands in windows:
## [1] "C:\\Users\\fan\\R4Econ"## [1] " Volume in drive C is OS, Volume Serial Number is 123A-85A2, , Directory of C:\\Users\\fan\\R4Econ, , 01/01/2024 07:59 PM <DIR> ., 01/01/2024 12:02 AM <DIR> .., 10/16/2023 03:53 AM 203 .gitattributes, 10/16/2023 03:53 AM 1,460 .gitignore, 10/16/2023 03:53 AM 30 .Rbuildignore, 01/01/2024 07:54 PM 16,430 .Rhistory, 01/01/2024 04:30 PM 7,808 .Rprofile, 12/31/2023 10:37 PM <DIR> amto, 10/16/2023 03:53 AM 278 book.bib, 01/01/2024 07:52 PM <DIR> bookdown, 12/31/2023 10:37 PM <DIR> development, 10/16/2023 03:53 AM <DIR> dynamic, 10/16/2023 03:53 AM <DIR> factor, 10/16/2023 03:53 AM <DIR> function, 10/16/2023 03:53 AM <DIR> generate, 10/16/2023 03:53 AM <DIR> getdata, 10/16/2023 03:53 AM 2,951 gitsetup.md, 10/16/2023 03:53 AM 321 hdga.html, 10/16/2023 03:53 AM <DIR> htmlpdfr, 01/01/2024 07:54 PM 3,062 index.Rmd, 12/31/2023 10:58 PM 1,091 LICENSE, 10/16/2023 03:53 AM <DIR> linreg, 10/16/2023 03:53 AM <DIR> math, 10/16/2023 03:53 AM <DIR> optimization, 10/16/2023 03:53 AM 11,018 packages.bib, 12/31/2023 10:39 PM <DIR> panel, 01/01/2024 07:59 PM 667,911 Panel Data and Optimization with R.Rmd, 01/01/2024 07:52 PM 85,457 Panel-Data-and-Optimization-with-R.log, 01/01/2024 07:59 PM 667,911 Panel-Data-and-Optimization-with-R.Rmd, 01/01/2024 07:28 PM <DIR> Panel-Data-and-Optimization-with-R_files, 10/16/2023 03:53 AM 480 preamble.tex, 10/16/2023 03:53 AM 583 preamble_book.tex, 01/01/2024 07:54 PM 416 R4Econ.Rproj, 01/01/2024 07:59 PM 70,714 README.md, 01/01/2024 07:59 PM 68,252 README_appendix.md, 10/16/2023 03:53 AM 762 README_end.md, 10/16/2023 03:53 AM 30,249 README_OLD.md, 01/01/2024 07:53 PM 3,256 README_pre.md, 01/01/2024 07:59 PM 66,690 README_toc.md, 10/16/2023 03:53 AM <DIR> regnonlin, 10/16/2023 03:53 AM <DIR> statistics, 10/16/2023 03:53 AM 595 style.css, 10/16/2023 03:53 AM <DIR> summarize, 10/16/2023 03:53 AM <DIR> support, 10/16/2023 03:53 AM <DIR> tabgraph, 10/16/2023 03:53 AM 136 title.Rmd, 10/16/2023 03:53 AM 2,570 toc.css, 01/01/2024 07:58 PM 5,229 _bookdown.yml, 01/01/2024 07:59 PM <DIR> _bookdown_files, 10/16/2023 03:53 AM 151 _config.yml, 10/16/2023 03:53 AM <DIR> _data, 10/16/2023 03:53 AM <DIR> _file, 10/16/2023 03:53 AM <DIR> _img, 10/16/2023 03:53 AM <DIR> _log, 10/16/2023 03:53 AM 1,438 _output.yml, 10/16/2023 03:53 AM 97 _output_kniti_html.yaml, 10/16/2023 03:53 AM 166 _output_kniti_pdf.yaml, 31 File(s) 1,717,715 bytes, 26 Dir(s) 193,147,244,544 bytes free"12.4.1.2 Run Python Inside a Conda Environment
Use shell rather than system to activate a conda environment, check python version:
# activate conda env
print(toString(shell(paste0("activate base & python --version"), intern=TRUE)))## [1] ", C:\\Users\\fan\\R4Econ>conda.bat activate base , Python 3.11.5"Activate conda env and run a line:
spg_runpython <- paste0("activate base &",
"python --version &",
"python -c ",
"\"st_var='this is string var';",
"print(f'{st_var}');",
"\"")
print(toString(shell(spg_runpython, intern=TRUE)))## [1] ", C:\\Users\\fan\\R4Econ>conda.bat activate base , Python 3.11.5, this is string var"12.5 Run Code in Parallel in R
12.5.1 Parallel Loop in R
Go back to fan’s REconTools research support package, R4Econ examples page, PkgTestR packaging guide, or Stat4Econ course page.
12.5.1.1 Setting Up and First Run
First, install several packages.
Second, we load the libraries, and check on the parallel processing capacities on the local machine.
# Load libraries
library(dplyr)
library(readr)
library(tibble)
library(iterators)
library(parallel)
library(foreach)
library(doParallel)
# Check number of cores
it_n_cores_computer <- parallel::detectCores()
glue::glue("Number of cores on computers:{it_n_cores_computer}")
# OUTPUT
## Number of cores on computers:20Third, we might want to use less than the total number of cores available. Specifying the number of cores to be used, we can initiate a local cluster. “PSOCK” below copies everything to each worker.
# Start cluster
ob_cluster <- parallel::makeCluster(
it_n_cores_computer - 2,
type = "PSOCK"
)
# Register cluster
doParallel::registerDoParallel(cl = ob_cluster)Fourth, run first parallel task, concurrent base-10 exponentiation.
# c(a,b,c,d) outputs together with combine
ar_test_parallel <- foreach(
it_power = seq(1, 10), .combine = 'c'
) %dopar% {
return(10^(it_power))
}
glue::glue("dopar outputs: {ar_test_parallel}")
# Output
## dopar outputs: 10
## dopar outputs: 100
## dopar outputs: 1000
## dopar outputs: 10000
## dopar outputs: 1e+05
## dopar outputs: 1e+06
## dopar outputs: 1e+07
## dopar outputs: 1e+08
## dopar outputs: 1e+09
## dopar outputs: 1e+10Fifth, close cluster. When work is done, close the cluster.
12.5.1.2 Parallel Function Run with Different Parameters, Aggregate Output Files
In this example, we create a function, we run the function with different parameters, each time generating a data output file to be stored, and then review results after.
First, we create a function. In this function, we generate a random matrix, the it_nrow parameter controls the number of rows in this random matrix. We store this matrix as csv.
Note, for each function used, such as as_tibble below, we should write it as tibble::as_tibble, to declare package and function jointly.
ffi_rand2csv <- function(
spt_path_out,
it_nrow = 3,
st_file_prefix = "prefix") {
# Generate a matrix and tibble
mt_rnorm_a <- matrix(
rnorm(it_nrow*3, mean=0, sd=1),
nrow=it_nrow, ncol=3)
tb_test <- tibble::as_tibble(mt_rnorm_a)
# File output path
spn_output_file <- file.path(
spt_path_out,
paste0(st_file_prefix, '_nrow', it_nrow, '.csv'),
fsep = .Platform$file.sep)
# Write file out
readr::write_csv(tb_test, spn_output_file)
print(glue::glue(
"File saved successfully: ", spn_output_file))
}Second, we initialize the cluster.
# Get the number of cores
it_n_cores_computer <- parallel::detectCores()
glue::glue("Number of cores on computers:{it_n_cores_computer}")
# Start cluster
ob_cluster <- parallel::makeCluster(
it_n_cores_computer - 2,
type = "PSOCK"
)
# Register cluster
doParallel::registerDoParallel(cl = ob_cluster)
# OUTPUT
## Number of cores on computers:20Third, we run the function in parallel.
# Define shared Path
spt_root <- "C:/Users/fan/"
spt_rmd <- "R4Econ/development/parallel/_file/"
spt_path_out <- file.path(spt_root, spt_rmd, fsep = .Platform$file.sep)
# Parallel Run
foreach(
it_nrow = seq(2, 4)
) %dopar% {
# Run function
ffi_rand2csv(
spt_path_out,
it_nrow = it_nrow,
st_file_prefix = "ffi_para_test")
}
# Output
## [[1]]
## File saved successfully: C:/Users/fan//R4Econ/development/parallel/_file/ffi_para_test_nrow2.csv
##
## [[2]]
## File saved successfully: C:/Users/fan//R4Econ/development/parallel/_file/ffi_para_test_nrow3.csv
##
## [[3]]
## File saved successfully: C:/Users/fan//R4Econ/development/parallel/_file/ffi_para_test_nrow4.csvFourth, adapting the parallel loop to other functions. Note that:
- In the
forachloop ablow, we iterate overseq(2,4), assigning in parallel2,3, and4to the parameterit_nrow. it_nrowis a parameter for theffi_rand2csvfunction, so we will generate different outputs associated withit_nrow=2,it_nrow=3, andit_nrow=4.- The code above can be adapted to other functions that one wants to run in parallel by changing only one parameter of a function. For example, suppose we want to run
ffp_demo_loc_env_inequality(spt_path_data, fl_temp_bound=fl_temp_bound), wherespt_path_datais common across parallel calls, but we want to updatefl_temp_boundfor each parallel call, then we need to iterate overfl_temp_bound. See example below: