Chapter 9 System and Support

9.1 Command Line

9.1.1 Python Command Line

Go back to fan’s Python Code Examples Repository (bookdown site) or the pyfan Package (API).

9.1.1.1 Run Command Line from Inside Python

Use subprocess, where is python:

import subprocess
cmd_popen = subprocess.Popen(["where", "python"],
                             stdin=subprocess.PIPE,
                             stdout=subprocess.PIPE,
                             stderr=subprocess.PIPE)
output, err = cmd_popen.communicate()
print(output.decode('utf-8'))

## G:\ProgramData\Anaconda3\envs\wk_pyfan\python.exe
## G:\ProgramData\Anaconda3\python.exe
## C:\Users\fan\AppData\Local\Microsoft\WindowsApps\python.exe

Command line file redirection symbol,

# The > command line sends current console output to file.txt
# cd "C:\users\fan"
# ls > ls_files.txt
# rm ls_files.txt
import os
import subprocess

# ls in current location
cmd_ls = subprocess.Popen(["ls"], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True);
stf_out_cmd_ls, err = cmd_ls.communicate()
stf_out_cmd_ls = stf_out_cmd_ls.decode('utf-8')
print(stf_out_cmd_ls)

srt_file_tex = "_file/"
sna_file_tex = "test_ls_pyfanvig_stdout"
srn_file_tex = srt_file_tex + sna_file_tex + ".txt"
fl_tex_contents = open(srn_file_tex, 'w')
fl_tex_contents.write(stf_out_cmd_ls)

## 0

fl_tex_contents.close()

9.1.1.2 Execute Command Line Python Functions

run python from command line
run python function with parameters from command line

Here run python from command line inside python itself.

# Run:
from py.fan.util.rmd.mattexmd import fp_mlxtex2md
fp_mlxtex2md(spt_root='C:/Users/fan/Math4Econ/matrix_application/', ls_srt_subfolders=None, st_rglob_tex='twogoods.tex', verbose=True)

# Run:
python -c "from pyfan.util.rmd.mattexmd import fp_mlxtex2md; fp_mlxtex2md(spt_root='C:/Users/fan/Math4Econ/matrix_application/', ls_srt_subfolders=None, st_rglob_tex='twogoods.tex', verbose=True)"

9.1.2 Run Matlab Functions

Go back to fan’s Python Code Examples Repository (bookdown site) or the pyfan Package (API).

9.1.2.1 Generate A template Matlab Script

Generate an example matlab script file with parameter x.

# Example Matlab Function
stf_m_contents = """\
a = x + 1
b = 10*x\
"""
# Print
print(stf_m_contents)
# Open new file

## a = x + 1
## b = 10*x

fl_m_contents = open("_m/fs_test.m", 'w')
# Write to File
fl_m_contents.write(stf_m_contents)
# print

## 18

fl_m_contents.close()

9.1.2.2 Run the Matlab Function from Commandline

First, check where matlab is installed:

import subprocess
cmd_popen = subprocess.Popen(["where", "matlab"],
                             stdin=subprocess.PIPE,
                             stdout=subprocess.PIPE,
                             stderr=subprocess.PIPE)
output, err = cmd_popen.communicate()
print(output.decode('utf-8'))

## G:\ProgramData\MATLAB\R2020b\bin\matlab.exe

Second, run the matlab file, first definet he parameter x:

import os
# print and set directory
print(os.getcwd())

## G:\repos\Py4Econ

os.chdir('_m')
print(os.getcwd())
# run matlab script saved prior
# running command line: matlab -batch "fs_test; exit"

## G:\repos\Py4Econ\_m

cmd_popen = subprocess.Popen(["matlab", "-batch", "\"x=1; fs_test; exit\""],
                             stdin=subprocess.PIPE,
                             stdout=subprocess.PIPE,
                             stderr=subprocess.PIPE)
output, err = cmd_popen.communicate()
print(output.decode('utf-8'))

## 
## a =
## 
##      2
## 
## 
## b =
## 
##     10
##

Third, run the function again, but with x=3:

os.chdir('_m')
print(os.getcwd())

## G:\repos\Py4Econ\_m

print(subprocess.Popen(["matlab", "-batch", "\"x=5; fs_test; exit\""],
                       stdin=subprocess.PIPE,
                       stdout=subprocess.PIPE,
                       stderr=subprocess.PIPE).communicate()[0].decode('utf-8'))

## 
## a =
## 
##      6
## 
## 
## b =
## 
##     50
##

9.2 File In and Out

9.2.1 Check, Read, Write and Convert Files

Go back to fan’s Python Code Examples Repository (bookdown site) or the pyfan Package (API).

9.2.1.1 Check if where a Program is.

Suppose we want to generate a commandline call from within python and want to start it in bash. Calling something like “C:/Program Files/Git/git-bash.exe -c COMMANDS.” However, depending on the computer that is been used, the git-bash command might be in different spots. How do we find the right path to the git-bash file. Accomplish this by using shutil.which, which can find the path to different commands, including the git command. Given the path that we found, the git-bash.exe file is stored in the Git folder, two levels up. So we will use pathlib to get to the correct path location. To be safe, go up one level, and then two levels to search for git-bash.exe.

First, find the path to the git exe command:

# Imports
import os  
import shutil  
# cmd  
st_cmd = 'git'
# Using shutil.which() method to find local path to the *git* command
spn_path_to_git = shutil.which(st_cmd) 
# Print result 
print(f'{spn_path_to_git=}')

## spn_path_to_git='G:\\ProgramData\\Git\\cmd\\git.EXE'

Second, find the parent and grandparent folders:

from pathlib import Path
# Get the parent folder 2 levels up
srt_path_git_parent_folder = Path(spn_path_to_git).parents[0]
srt_path_git_grandparent_folder = Path(spn_path_to_git).parents[1]
# Print
print(f'{srt_path_git_parent_folder=}')

## srt_path_git_parent_folder=WindowsPath('G:/ProgramData/Git/cmd')

print(f'{srt_path_git_grandparent_folder=}')
# Search for for the git-bash.exe file in parent and then in the grandparent folder.

## srt_path_git_grandparent_folder=WindowsPath('G:/ProgramData/Git')

Third, search inside parent folder first, and then grand until find the path to git-bash.exe. Will put all three steps code together:

# required packages
import shutil
from pathlib import Path
# find path to git
st_cmd = 'git'
spn_path_to_git = shutil.which(st_cmd) 
# find path to git-bash.exe
spn_path_to_gitbash = ''
for it_up_iter in [0,1]:
    # up-tier folder
    srt_path_git_up_folder = Path(spn_path_to_git).parents[it_up_iter]
    # search
    # get file names in folders (not recursively)
    ls_spn_found_git_bash = [spn_file for spt_srh in [srt_path_git_up_folder]
                             for spn_file in Path(spt_srh).glob('git-bash.exe')]
    # if found, length of ls of founds files must be 1
    if len(ls_spn_found_git_bash) == 1:
        spn_path_to_gitbash = ls_spn_found_git_bash[0]
        break
        
if spn_path_to_gitbash == '':
    raise NameError(f'failed to find git-bash, {spn_path_to_git=}')
else:
    print(f'Found git-bash: {spn_path_to_gitbash} by searching around {spn_path_to_git=}')

## Found git-bash: G:\ProgramData\Git\git-bash.exe by searching around spn_path_to_git='G:\\ProgramData\\Git\\cmd\\git.EXE'

9.2.1.2 Generate a tex file

Will a bare-bone tex file with some texts inside, save inside the *_file* subfolder.

First, create the text text string, note the the linebreaks utomatically generate linebreaks, note that slash need double slash:

# Create the Tex Text
# Note that trible quotes begin first and end last lines
stf_tex_contents = """\\documentclass[12pt,english]{article}
\\usepackage[bottom]{footmisc}
\\usepackage[urlcolor=blue]{hyperref}
\\begin{document}
\\title{A Latex Testing File}
\\author{\\href{http://fanwangecon.github.io/}{Fan Wang} \\thanks{See information \\href{https://fanwangecon.github.io/Tex4Econ/}{Tex4Econ} for more.}}
\\maketitle
Ipsum information dolor sit amet, consectetur adipiscing elit. Integer Latex placerat nunc orci.
\\paragraph{\\href{https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3140132}{Data}}
Village closure information is taken from a village head survey.\\footnote{Generally students went to schools.}
\\end{document}"""
# Print
print(stf_tex_contents)

## \documentclass[12pt,english]{article}
## \usepackage[bottom]{footmisc}
## \usepackage[urlcolor=blue]{hyperref}
## \begin{document}
## \title{A Latex Testing File}
## \author{\href{http://fanwangecon.github.io/}{Fan Wang} \thanks{See information \href{https://fanwangecon.github.io/Tex4Econ/}{Tex4Econ} for more.}}
## \maketitle
## Ipsum information dolor sit amet, consectetur adipiscing elit. Integer Latex placerat nunc orci.
## \paragraph{\href{https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3140132}{Data}}
## Village closure information is taken from a village head survey.\footnote{Generally students went to schools.}
## \end{document}

Second, write the contents of the file to a new tex file stored inside the *_file* subfolder of the directory:

# Relative file name
srt_file_tex = "_file/"
sna_file_tex = "test_fan"
srn_file_tex = srt_file_tex + sna_file_tex + ".tex"
# Open new file
fl_tex_contents = open(srn_file_tex, 'w')
# Write to File
fl_tex_contents.write(stf_tex_contents)
# print

## 617

fl_tex_contents.close()

9.2.1.3 Replace Strings in a tex file

Replace a set of strings in the file just generated by a set of alternative strings.

# Open file Get text
fl_tex_contents = open(srn_file_tex)
stf_tex_contents = fl_tex_contents.read()
print(srn_file_tex)

# define new and old

## _file/test_fan.tex

ls_st_old = ['information', 'Latex']
ls_st_new = ['INFOREPLACE', 'LATEX']

# zip and loop and replace
for old, new in zip(ls_st_old, ls_st_new):
  stf_tex_contents = stf_tex_contents.replace(old, new)
print(stf_tex_contents)

# write to file with replacements

## \documentclass[12pt,english]{article}
## \usepackage[bottom]{footmisc}
## \usepackage[urlcolor=blue]{hyperref}
## \begin{document}
## \title{A LATEX Testing File}
## \author{\href{http://fanwangecon.github.io/}{Fan Wang} \thanks{See INFOREPLACE \href{https://fanwangecon.github.io/Tex4Econ/}{Tex4Econ} for more.}}
## \maketitle
## Ipsum INFOREPLACE dolor sit amet, consectetur adipiscing elit. Integer LATEX placerat nunc orci.
## \paragraph{\href{https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3140132}{Data}}
## Village closure INFOREPLACE is taken from a village head survey.\footnote{Generally students went to schools.}
## \end{document}

sna_file_edited_tex = "test_fan_edited"
srn_file_edited_tex = srt_file_tex + sna_file_edited_tex + ".tex"
fl_tex_ed_contents = open(srn_file_edited_tex, 'w')
fl_tex_ed_contents.write(stf_tex_contents)

## 617

fl_tex_ed_contents.close()

9.2.1.4 Convert Tex File to Pandoc and Compile Latex

Compile tex file to pdf and clean up the extraneous pdf outputs. See ff_pdf_gen_clean.

import subprocess
import os

# Change to local directory so path in tex respected.
os.chdir("C:/Users/fan/py4econ/support/inout")

# Convert tex to pdf
subprocess.call(['C:/texlive/2020/bin/win32/xelatex.exe', '-output-directory',
                 srt_file_tex, srn_file_edited_tex], shell=False)
# Clean pdf extraneous output

## 0

ls_st_remove_suffix = ['aux','log','out','bbl','blg']
for st_suffix in ls_st_remove_suffix:
    srn_cur_file = srt_file_tex + sna_file_edited_tex + "." + st_suffix
    if (os.path.isfile(srn_cur_file)):
        os.remove(srt_file_tex + sna_file_edited_tex + "." + st_suffix)

Use pandoc to convert tex file

import subprocess

# md file name
srn_file_md = srt_file_tex + "test_fan_edited.md"
# Convert tex to md
subprocess.call(['pandoc', '-s', srn_file_tex, '-o', srn_file_md])
# Open md file

## 0

fl_md_contents = open(srn_file_md)
print(fl_md_contents.read())

## ---
## author:
## - "[Fan Wang](http://fanwangecon.github.io/) [^1]"
## title: A Latex Testing File
## ---
## 
## Ipsum information dolor sit amet, consectetur adipiscing elit. Integer
## Latex placerat nunc orci.
## 
## #### [Data](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3140132)
## 
## Village closure information is taken from a village head survey.[^2]
## 
## [^1]: See information
##     [Tex4Econ](https://fanwangecon.github.io/Tex4Econ/) for more.
## 
## [^2]: Generally students went to schools.

9.2.1.5 Search for Files with Suffix in Several Folders

python search all files in folders with suffix

Search for files in several directories that have a particular suffix. Then decompose directory into sub-components.

Search file inside several folders (not recursively in subfolders):

from pathlib import Path

# directories to search in
ls_spt_srh = ["C:/Users/fan/R4Econ/amto/",
              "C:/Users/fan/R4Econ/function/"]

# get file names in folders (not recursively)
ls_spn_found = [spn_file for spt_srh in ls_spt_srh
                         for spn_file in Path(spt_srh).glob('*.Rmd')]
for spn_found in ls_spn_found:
  print(spn_found)

## C:\Users\fan\R4Econ\amto\main.Rmd
## C:\Users\fan\R4Econ\function\main.Rmd

Search file recursivesly in all subfolders of folders:

from pathlib import Path

# directories to search in
ls_spt_srh = ["C:/Users/fan/R4Econ/amto/array/",
              "C:/Users/fan/R4Econ/amto/list"]

# get file names recursively in all subfolders
ls_spn_found = [spn_file for spt_srh in ls_spt_srh
                         for spn_file in Path(spt_srh).rglob('*.R')]
for spn_found in ls_spn_found:
  drive, path_and_file = os.path.splitdrive(spn_found)
  path_no_suffix = os.path.splitext(spn_found)[0]
  path_no_file, file = os.path.split(spn_found)
  file_no_suffix = Path(spn_found).stem
  print('file:', file, '\ndrive:', drive,
        '\nfile no suffix:', file_no_suffix,
        '\nfull path:', spn_found,
        '\npt no fle:', path_no_file,
        '\npt no suf:', path_no_suffix, '\n')

## file: fs_ary_basics.R 
## drive: C: 
## file no suffix: fs_ary_basics 
## full path: C:\Users\fan\R4Econ\amto\array\htmlpdfr\fs_ary_basics.R 
## pt no fle: C:\Users\fan\R4Econ\amto\array\htmlpdfr 
## pt no suf: C:\Users\fan\R4Econ\amto\array\htmlpdfr\fs_ary_basics 
## 
## file: fs_ary_generate.R 
## drive: C: 
## file no suffix: fs_ary_generate 
## full path: C:\Users\fan\R4Econ\amto\array\htmlpdfr\fs_ary_generate.R 
## pt no fle: C:\Users\fan\R4Econ\amto\array\htmlpdfr 
## pt no suf: C:\Users\fan\R4Econ\amto\array\htmlpdfr\fs_ary_generate 
## 
## file: fs_ary_mesh.R 
## drive: C: 
## file no suffix: fs_ary_mesh 
## full path: C:\Users\fan\R4Econ\amto\array\htmlpdfr\fs_ary_mesh.R 
## pt no fle: C:\Users\fan\R4Econ\amto\array\htmlpdfr 
## pt no suf: C:\Users\fan\R4Econ\amto\array\htmlpdfr\fs_ary_mesh 
## 
## file: fs_ary_string.R 
## drive: C: 
## file no suffix: fs_ary_string 
## full path: C:\Users\fan\R4Econ\amto\array\htmlpdfr\fs_ary_string.R 
## pt no fle: C:\Users\fan\R4Econ\amto\array\htmlpdfr 
## pt no suf: C:\Users\fan\R4Econ\amto\array\htmlpdfr\fs_ary_string 
## 
## file: fs_listr.R 
## drive: C: 
## file no suffix: fs_listr 
## full path: C:\Users\fan\R4Econ\amto\list\htmlpdfr\fs_listr.R 
## pt no fle: C:\Users\fan\R4Econ\amto\list\htmlpdfr 
## pt no suf: C:\Users\fan\R4Econ\amto\list\htmlpdfr\fs_listr 
## 
## file: fs_lst_basics.R 
## drive: C: 
## file no suffix: fs_lst_basics 
## full path: C:\Users\fan\R4Econ\amto\list\htmlpdfr\fs_lst_basics.R 
## pt no fle: C:\Users\fan\R4Econ\amto\list\htmlpdfr 
## pt no suf: C:\Users\fan\R4Econ\amto\list\htmlpdfr\fs_lst_basics

9.2.2 Folder Operations

Go back to fan’s Python Code Examples Repository (bookdown site) or the pyfan Package (API).

9.2.2.1 Create an Absolute Folder with Path join

Create a platform free full absolute path to a particular folder

import os 
import pathlib

# suffix
st_suffix = "_mlt_region_ne"
srt_folder = "testfolder" + st_suffix + '_other_stuff'
# path join with os.sep
srt_path = os.path.join(os.sep, "users", "fan", "pyfan", "vig", "support", "inout", "_folder", "testfolder" + st_suffix, 'subfolder')
# Path Name
spn_path = os.path.abspath(srt_path)
# Create the folder
pathlib.Path(spn_path).mkdir(parents=True, exist_ok=True)
# Print
print(f'{srt_folder=}')

## srt_folder='testfolder_mlt_region_ne_other_stuff'

print(f'{srt_path=}')

## srt_path='\\users\\fan\\pyfan\\vig\\support\\inout\\_folder\\testfolder_mlt_region_ne\\subfolder'

print(f'{spn_path=}')

## spn_path='G:\\users\\fan\\pyfan\\vig\\support\\inout\\_folder\\testfolder_mlt_region_ne\\subfolder'

Slash converion, convert the system slash to forward-slash all:

srt_folder_slashconverted = spn_path.replace(os.sep, '/')
print(f'{srt_folder_slashconverted=}')

## srt_folder_slashconverted='G:/users/fan/pyfan/vig/support/inout/_folder/testfolder_mlt_region_ne/subfolder'

see: constructing absolute path with os.path.join().

9.2.2.2 Get the Last Directory in a Path without Some Suffix

Suppose there is a directory with ‘abc_suffix_other/subfolder’ as the name, generate a new folder that has ‘abc’ as the folder name without ’_suffix’. Generate this folder in the same root folder that the abc_suffix folder resides in.

# Absolute path just created:
print(f'{spn_path=}')
# the suffix used

## spn_path='G:\\users\\fan\\pyfan\\vig\\support\\inout\\_folder\\testfolder_mlt_region_ne\\subfolder'

print(f'{st_suffix=}')
# get path without what comes after suffix

## st_suffix='_mlt_region_ne'

spn_path_no_suffix = spn_path[:spn_path.index(st_suffix)]
# Create the folder
pathlib.Path(spn_path_no_suffix).mkdir(parents=True, exist_ok=True)
# Get the new folder name create
spt_root_main, srt_new_subfolder = os.path.split(spn_path_no_suffix)
# Add Slash to new subfolder
spn_path_no_suffix = spn_path_no_suffix + os.sep
# Print
print(f'{spn_path_no_suffix=}')

## spn_path_no_suffix='G:\\users\\fan\\pyfan\\vig\\support\\inout\\_folder\\testfolder\\'

print(f'{spt_root_main=}')

## spt_root_main='G:\\users\\fan\\pyfan\\vig\\support\\inout\\_folder'

print(f'{srt_new_subfolder=}')

## srt_new_subfolder='testfolder'

9.2.2.3 New Folder and Files

create a folder and subfolder
create two files in the new folder

import pathlib

# folder root
srt_folder = "_folder/"

# new folder
srt_subfolder = srt_folder + "fa/"
# new subfolder
srt_subfolder = srt_subfolder + "faa/"
# generate folders recursively
pathlib.Path(srt_subfolder).mkdir(parents=True, exist_ok=True)

# Open new file
fl_tex_contents_aa = open(srt_subfolder + "file_a.txt", 'w')
# Write to File
fl_tex_contents_aa.write('contents of file a')

## 18

fl_tex_contents_aa.close()

# Open another new file and save
fl_tex_contents_ab = open(srt_subfolder + "file_b.txt", 'w')
# Write to File
fl_tex_contents_ab.write('contents of file b')

## 18

fl_tex_contents_ab.close()

Generate more folders without files:

# generate folders recursively
pathlib.Path("_folder/fb/fba/").mkdir(parents=True, exist_ok=True)
# generate folders recursively
pathlib.Path("_folder/fc/").mkdir(parents=True, exist_ok=True)
# generate folders recursively
pathlib.Path("_folder/fd/").mkdir(parents=True, exist_ok=True)

9.2.2.4 Copy a File from One Folder to Another

Move the two files from *_folder/fa/faa* to *_folder/faa* as well as to *_folder/fb/faa. Use shutil.copy2* so that more metadata is copied over. But copyfile is faster.

How do I copy a file in Python?

Moving one file:

import shutil
# Faster method
shutil.copyfile('_folder/fa/faa/file_a.txt', '_folder/fb/file_a.txt')
# More metadat copied, and don't need to specify name

## '_folder/fb/file_a.txt'

shutil.copy2('_folder/fa/faa/file_a.txt', '_folder/fb/fba')

## '_folder/fb/fba\\file_a.txt'

9.2.2.5 Copy Folder to Multiple Destimations

Move Entire Folder, How do I copy an entire directory of files into an existing directory using Python?:

from distutils.dir_util import copy_tree

# Move contents from fa/faa/ to fc/faa
srt_curroot = '_folder/fa/'
srt_folder = 'faa/'
srt_newroot = '_folder/fc/'

# Full source and destination
srt_sourc = srt_curroot + srt_folder
srt_desct = srt_newroot + srt_folder

# Check/Create new Directory 
pathlib.Path(srt_desct).mkdir(parents=True, exist_ok=True)

# Move
copy_tree(srt_sourc, srt_desct)

## ['_folder/fc/faa/file_a.txt', '_folder/fc/faa/file_b.txt']

Move contents to multiple destinations:

from distutils.dir_util import copy_tree
# Check/Create new Directory 
pathlib.Path('_folder/fd/faa/fa_images').mkdir(parents=True, exist_ok=True)
pathlib.Path('_folder/fd/faa/fb_images').mkdir(parents=True, exist_ok=True)
pathlib.Path('_folder/fd/faa/fc_images').mkdir(parents=True, exist_ok=True)
pathlib.Path('_folder/fd/faa/fz_img').mkdir(parents=True, exist_ok=True)
pathlib.Path('_folder/fd/faa/fz_other').mkdir(parents=True, exist_ok=True)

# Move
copy_tree('_folder/fa/faa/', '_folder/fd/faa/fa_images')

## ['_folder/fd/faa/fa_images\\file_a.txt', '_folder/fd/faa/fa_images\\file_b.txt']

copy_tree('_folder/fa/faa/', '_folder/fd/faa/fb_images')

## ['_folder/fd/faa/fb_images\\file_a.txt', '_folder/fd/faa/fb_images\\file_b.txt']

copy_tree('_folder/fa/faa/', '_folder/fd/faa/fc_images')

## ['_folder/fd/faa/fc_images\\file_a.txt', '_folder/fd/faa/fc_images\\file_b.txt']

copy_tree('_folder/fa/faa/', '_folder/fd/faa/fz_img')

## ['_folder/fd/faa/fz_img\\file_a.txt', '_folder/fd/faa/fz_img\\file_b.txt']

copy_tree('_folder/fa/faa/', '_folder/fd/faa/fz_other')
# Empty Folder

## ['_folder/fd/faa/fz_other\\file_a.txt', '_folder/fd/faa/fz_other\\file_b.txt']

pathlib.Path('_folder/fd/faa/fd_images').mkdir(parents=True, exist_ok=True)
pathlib.Path('_folder/fd/faa/fe_images').mkdir(parents=True, exist_ok=True)

9.2.2.6 Search for Files in Folder

Find the total number of files in a folder.

from pathlib import Path

# the number of files in folder found with search critiera
st_fle_search = '*.txt'
ls_spn = [Path(spn).stem for spn in Path('_folder/fd/faa/fa_images').rglob(st_fle_search)]
print(ls_spn)

# count files in a non-empty folder

## ['file_a', 'file_b']

srn = '_folder/fd/faa/fa_images'
[spn for spn in Path(srn).rglob(st_fle_search)]

## [WindowsPath('_folder/fd/faa/fa_images/file_a.txt'), WindowsPath('_folder/fd/faa/fa_images/file_b.txt')]

bl_folder_is_empty = len([spn for spn in Path(srn).rglob(st_fle_search)])>0
print(bl_folder_is_empty)

# count files in an empty folder

## True

srn = '_folder/fd/faa/fd_images'
[spn for spn in Path(srn).rglob(st_fle_search)]

## []

bl_folder_is_empty = len([spn for spn in Path(srn).rglob(st_fle_search)])>0
print(bl_folder_is_empty)

## False

9.2.2.7 Search for Folder Names

python search for folders containing strings

Search for folders with certain search word in folder name, and only keep if folder actually has files.

import os

# get all folder names in folder
ls_spt = os.listdir('_folder/fd/faa/')
print(ls_spt)

# Select only subfolder names containing _images

## ['fa_images', 'fb_images', 'fc_images', 'fd_images', 'fe_images', 'fz_img', 'fz_other', '_img']

srt = '_folder/fd/faa/'
st_search = '_images'
ls_srt_found = [srt + spt 
                for spt in os.listdir(srt) 
                if st_search in spt]
print(ls_srt_found)

## ['_folder/fd/faa/fa_images', '_folder/fd/faa/fb_images', '_folder/fd/faa/fc_images', '_folder/fd/faa/fd_images', '_folder/fd/faa/fe_images']

9.2.2.8 Find Non-empty Folders by Name

Search:

Get subfolders in folder with string in name
Only collect if there are files in the subfolder

import pathlib

# Select only subfolder names containing _images
srt = '_folder/fd/faa/'
# the folder names must contain _images
st_srt_srh = '_images'
# there must be files in the folder with this string
st_fle_srh = '*.txt'

# All folders that have String
ls_srt_found = [srt + spt 
                for spt in os.listdir(srt) 
                if st_srt_srh in spt]
print(ls_srt_found)

# All folders that have String and that are nonempty

## ['_folder/fd/faa/fa_images', '_folder/fd/faa/fb_images', '_folder/fd/faa/fc_images', '_folder/fd/faa/fd_images', '_folder/fd/faa/fe_images']

ls_srt_found = [srt + spt
                for spt in os.listdir(srt)
                if ((st_srt_srh in spt) 
                    and 
                    (len([spn for spn 
                          in Path(srt + spt).rglob(st_fle_srh)])>0)) ]
print(ls_srt_found)

## ['_folder/fd/faa/fa_images', '_folder/fd/faa/fb_images', '_folder/fd/faa/fc_images']

9.2.2.9 Found Folders to new Folder

Search for subfolders by folder name string in a folder
Select nonempty subfolders
Move nonsempty subfolders to one new folder
Move this single combination folder

The results here are implemented as function in the pyfan package: fp_agg_move_subfiles.

import pathlib
import os
import shutil
from distutils.dir_util import copy_tree

# 1 Define Parameters

# Select only subfolder names containing _images
srt = '_folder/fd/faa/'
# the folder names must contain _images
st_srt_srh = '_images'
# there must be files in the folder with this string
st_fle_srh = '*.txt'

# new aggregating folder name
srt_agg = '_img'

# folders to move aggregation files towards
ls_srt_dest = ['_folder/fd/faa/', '_folder/']

# delete source
bl_delete_source = False

# 2 Gather Folders 
ls_ls_srt_found = [[srt + spt, spt]
                    for spt in os.listdir(srt)
                    if ((st_srt_srh in spt) 
                        and 
                        (len([spn for spn 
                              in Path(srt + spt).rglob(st_fle_srh)])>0)) ]
print(ls_ls_srt_found)

# 3 Loop over destination folders, loop over source folders

## [['_folder/fd/faa/fa_images', 'fa_images'], ['_folder/fd/faa/fb_images', 'fb_images'], ['_folder/fd/faa/fc_images', 'fc_images']]

for srt in ls_srt_dest:

  # Move each folder over
  for ls_srt_found in ls_ls_srt_found:

    # Paths
    srt_source = ls_srt_found[0]
    srt_dest = os.path.join(srt, srt_agg, ls_srt_found[1])
    
    # dest folders
    pathlib.Path(srt_dest).mkdir(parents=True, exist_ok=True)
    
    # move
    copy_tree(ls_srt_found[0], srt_dest)

# 4. Delete Sources

## ['_folder/fd/faa/_img\\fa_images\\file_a.txt', '_folder/fd/faa/_img\\fa_images\\file_b.txt']
## ['_folder/fd/faa/_img\\fb_images\\file_a.txt', '_folder/fd/faa/_img\\fb_images\\file_b.txt']
## ['_folder/fd/faa/_img\\fc_images\\file_a.txt', '_folder/fd/faa/_img\\fc_images\\file_b.txt']
## ['_folder/_img\\fa_images\\file_a.txt', '_folder/_img\\fa_images\\file_b.txt']
## ['_folder/_img\\fb_images\\file_a.txt', '_folder/_img\\fb_images\\file_b.txt']
## ['_folder/_img\\fc_images\\file_a.txt', '_folder/_img\\fc_images\\file_b.txt']

if bl_delete_source:
  for ls_srt_found in ls_ls_srt_found:
    shutil.rmtree(ls_srt_found[0])

9.2.3 Parse Yaml

Go back to fan’s Python Code Examples Repository (bookdown site) or the pyfan Package (API).

Use the PyYAML to parse yaml.

9.2.3.1 Write and Create a Simple YAML file

First, Yaml as a string variable:

# Create the Tex Text
# Note that trible quotes begin first and end last lines
stf_tex_contents = """\
- file: matrix_matlab
  title: "One Variable Graphs and Tables"
  description: |
    Frequency table, bar chart and histogram.
    R function and lapply to generate graphs/tables for different variables.
  core:
  - package: r
    code: |
      c('word1','word2')
      function()
      for (ctr in c(1,2)) {}
  - package: dplyr
    code: |
      group_by()
  date: 2020-05-02
  output:
    pdf_document:
      pandoc_args: '../_output_kniti_pdf.yaml'
      includes:
        in_header: '../preamble.tex'
  urlcolor: blue
- file: matrix_algebra_rules
  title: "Opening a Dataset"
  titleshort: "Opening a Dataset"
  description: |
    Opening a Dataset.
  core:
  - package: r
    code: |
      setwd()
  - package: readr
    code: |
      write_csv()
  date: 2020-05-02
  date_start: 2018-12-01
- file: matrix_two
  title: "Third file"
  titleshort: "Third file"
  description: |
    Third file description."""
# Print
print(stf_tex_contents)

## - file: matrix_matlab
##   title: "One Variable Graphs and Tables"
##   description: |
##     Frequency table, bar chart and histogram.
##     R function and lapply to generate graphs/tables for different variables.
##   core:
##   - package: r
##     code: |
##       c('word1','word2')
##       function()
##       for (ctr in c(1,2)) {}
##   - package: dplyr
##     code: |
##       group_by()
##   date: 2020-05-02
##   output:
##     pdf_document:
##       pandoc_args: '../_output_kniti_pdf.yaml'
##       includes:
##         in_header: '../preamble.tex'
##   urlcolor: blue
## - file: matrix_algebra_rules
##   title: "Opening a Dataset"
##   titleshort: "Opening a Dataset"
##   description: |
##     Opening a Dataset.
##   core:
##   - package: r
##     code: |
##       setwd()
##   - package: readr
##     code: |
##       write_csv()
##   date: 2020-05-02
##   date_start: 2018-12-01
## - file: matrix_two
##   title: "Third file"
##   titleshort: "Third file"
##   description: |
##     Third file description.

Second, write the contents of the file to a new tex file stored inside the *_file* subfolder of the directory:

# Relative file name
srt_file_tex = "_file/"
sna_file_tex = "test_yml_fan"
srn_file_tex = srt_file_tex + sna_file_tex + ".yml"
# Open new file
fl_tex_contents = open(srn_file_tex, 'w')
# Write to File
fl_tex_contents.write(stf_tex_contents)
# print

## 908

fl_tex_contents.close()

9.2.3.2 Select Subset of Values by Key

Load Yaml file created prior, the output is a list of dictionaries:

import yaml 
import pprint
# Open yaml file
fl_yaml = open(srn_file_tex)
# load yaml 
ls_dict_yml = yaml.load(fl_yaml, Loader=yaml.BaseLoader)
# type
type(ls_dict_yml)

## <class 'list'>

type(ls_dict_yml[0])
# display

## <class 'dict'>

pprint.pprint(ls_dict_yml, width=1)

## [{'core': [{'code': "c('word1','word2')\n"
##                     'function()\n'
##                     'for '
##                     '(ctr '
##                     'in '
##                     'c(1,2)) '
##                     '{}\n',
##             'package': 'r'},
##            {'code': 'group_by()\n',
##             'package': 'dplyr'}],
##   'date': '2020-05-02',
##   'description': 'Frequency '
##                  'table, '
##                  'bar '
##                  'chart '
##                  'and '
##                  'histogram.\n'
##                  'R '
##                  'function '
##                  'and '
##                  'lapply '
##                  'to '
##                  'generate '
##                  'graphs/tables '
##                  'for '
##                  'different '
##                  'variables.\n',
##   'file': 'matrix_matlab',
##   'output': {'pdf_document': {'includes': {'in_header': '../preamble.tex'},
##                               'pandoc_args': '../_output_kniti_pdf.yaml'}},
##   'title': 'One '
##            'Variable '
##            'Graphs '
##            'and '
##            'Tables',
##   'urlcolor': 'blue'},
##  {'core': [{'code': 'setwd()\n',
##             'package': 'r'},
##            {'code': 'write_csv()\n',
##             'package': 'readr'}],
##   'date': '2020-05-02',
##   'date_start': '2018-12-01',
##   'description': 'Opening '
##                  'a '
##                  'Dataset.\n',
##   'file': 'matrix_algebra_rules',
##   'title': 'Opening '
##            'a '
##            'Dataset',
##   'titleshort': 'Opening '
##                 'a '
##                 'Dataset'},
##  {'description': 'Third '
##                  'file '
##                  'description.',
##   'file': 'matrix_two',
##   'title': 'Third '
##            'file',
##   'titleshort': 'Third '
##                 'file'}]

Select yaml information by file name which is a key shared by components of the list:

ls_str_file_ids = ['matrix_two']
ls_dict_selected = [dict_yml for dict_yml in ls_dict_yml if dict_yml['file'] in ls_str_file_ids]
pprint.pprint(ls_dc_selected, width=1)

## [{'date': datetime.date(2020, 5, 2),
##   'description': 'Frequency '
##                  'table, '
##                  'bar '
##                  'chart '
##                  'and '
##                  'histogram',
##   'file': 'mat_matlab',
##   'title': 'One '
##            'Variable '
##            'Graphs '
##            'and '
##            'Tables',
##   'val': 1}]

9.2.3.3 Dump List of Dictionary as YAML

py yaml dump pipe

Given a list of dictionaries, dump values to yaml. Note that dumped output does not use pipe for long sentences, but use single quote and space line, which works with the rmdparrse.py function without problem.

ls_dict_selected = [dict_yml for dict_yml in ls_dict_yml 
                    if dict_yml['file'] in ['matrix_two','matrix_matlab']]
print(yaml.dump(ls_dict_selected))

## - core:
##   - code: 'c(''word1'',''word2'')
## 
##       function()
## 
##       for (ctr in c(1,2)) {}
## 
##       '
##     package: r
##   - code: 'group_by()
## 
##       '
##     package: dplyr
##   date: '2020-05-02'
##   description: 'Frequency table, bar chart and histogram.
## 
##     R function and lapply to generate graphs/tables for different variables.
## 
##     '
##   file: matrix_matlab
##   output:
##     pdf_document:
##       includes:
##         in_header: ../preamble.tex
##       pandoc_args: ../_output_kniti_pdf.yaml
##   title: One Variable Graphs and Tables
##   urlcolor: blue
## - description: Third file description.
##   file: matrix_two
##   title: Third file
##   titleshort: Third file

9.3 Install Python

9.3.1 Core Installations

Go back to fan’s Python Code Examples Repository (bookdown site) or the pyfan Package (API).

Use the PyYAML to parse yaml.

9.3.1.1 Git Bash

Download and install git

9.3.1.2 Conda Install

Download Anaconda for Python 3. For more involved conda instructions see here
Get where you installed conda: open up anaconda prompt with admin rights (press windows button, and search for anaconda prompt, right click on the resulting terminal icon, choose as admin, a terminal opens up).

where python
where anaconda

# C:/ProgramData/Anaconda3/Scripts/anaconda.exe
# C:/ProgramData/Anaconda3/python.exe

Add to Path: open up windows Path and copy the paths found above inside.

9.3.1.2.1 Add To Path Details

To Add Anaconda to Path, In Windows

Search for: Environment Variables
Edit Environment Variables
Add new to Path (lower half):
- C:/ProgramData/Anaconda3/Scripts/
- C:/ProgramData/Anaconda3/
Now open up regular windows command Prompt, Type in: conda –version
Close and Open up Git Bash: conda –version

Alternatively, in windows, directly search for Path, and add the python and anaconda exe paths to paths.

9.4 Documentation

9.4.1 Numpy Doc Documentation Guide

Go back to fan’s Python Code Examples Repository (bookdown site) or the pyfan Package (API).

Check types:

print(type(111))
print(type('111'))
import logging
print(type(logging.WARNING))

Style 1:

Parameters
----------
n : int
    The upper limit of the range to generate, from 0 to `n` - 1.
param1 : int
    The first parameter.
param1 : str
    Description of `param1`.
msg : str
    Human readable string describing the exception.
param1 : int
    The first parameter.
param2 : str
    The second parameter.
param3 : str, optional
    The second parameter.
param5: dict
    A dictionary
param6: bool
    boolean
arr1 : ndarray
    2D array containing data with `float` type.
arr2 : ndarray
    1D mask array(containing data with boolean type).

Style 2, this will add a link to the types in python doc:

Parameters
----------
param2 : :obj:`str`, optional
    The second parameter.
code : :obj:`int`, optional
    Numeric error code.
param3 : :obj:`int`, optional
    Description of `param3`.
param4 : :obj:`list` of :obj:`str`
    Description of `param2`. Multiple
    lines are supported.

For args and kwargs:

Parameters
----------
*args
    Variable length argument list.
**kwargs
    Arbitrary keyword arguments.

9.4.1.1 Returns

Returns
-------
numpy.array of shape (1, it_draws)
    A vector of sorted or unsorted random grid points, or equi-quantile
    points.

Returns
-------
:obj:`tuple` of :obj:`bool`

Returns
-------
None

9.4.1.2 Function Calls

To refer to functions in the same .py file, just need to use: :func:log_format to refer to function name. For function in different .py files, might need its full path

**kwargs
  Arguments for functions that is called, including :func:`log_format`

9.4.1.3 Examples

Array outputs.

Examples
--------
>>> fl_mu = 0
>>> fl_sd = 1
>>> it_draws = 5
>>> it_seed = 123
>>> fl_lower_sd = -1
>>> fl_higher_sd = 0.8
>>> it_draw_type = 0
>>> ar_draw_random_normal(fl_mu, fl_sd, it_draws,
...                       it_seed, it_draw_type,
...                       fl_lower_sd, fl_higher_sd)
[-1.          0.8         0.2829785 - 1. - 0.57860025]
>>> it_draw_type = 1
>>> ar_draw_random_normal(fl_mu, fl_sd, it_draws,
...                       it_seed, it_draw_type,
...                       fl_lower_sd, fl_higher_sd)
[-1. - 0.47883617 - 0.06672597  0.3338994   0.8]
>>> it_draw_type = 2
>>> ar_draw_random_normal(fl_mu, fl_sd, it_draws,
...                       it_seed, it_draw_type,
...                       fl_lower_sd, fl_higher_sd)
[-1. - 1. - 0.57860025  0.2829785   0.8]

String outputs.

Examples
--------
>>> log_vig_start(spt_root = proj_sys_sup.main_directory(),
...               main_folder_name='logvig', sub_folder_name='parameters',
...               subsub_folder_name='combo_type',
...               file_name='fs_gen_combo_type',
...               it_time_format=8, log_level=logging.INFO)
C:\\Users\\fan\\logvig\\parameters\\combo_type\\fs_gen_combo_type_20201030.log.py