The work-in-progress Py4Econ python examples repository.

bookdown site and bookdown pdf.

Files are written with RMD. Materials are gathered from various projects in which python code is used for research and paper-administrative tasks. Bullet points show which python packages/functions are used to achieve various objectives. This is not a python package, but a set of example files. The pyfan package repository provides some associated programs.

From other repositories: for research support toolboxes, see matlab toolbox, r toolbox, and python toolbox; for code examples, see matlab examples, stata examples, r examples, python examples, and latex examples; for packaging example, see pkgtestr for developing r packages; for teaching, see intro mathematics for economists, and intro statistics for undergraduates. see here for all of fan’s public repositories.

Please contact FanWangEcon for issues or problems.

1 Data Structures

1.1 Numbers, Strings, Lists and Tuples

1. Basic Number Numeric Manipulations: rmd | pdf | html
   • Loop over a list of numbers where the first and second digits have different interpretations.
   • py: int(np.floor(it_num/10)) + it_num%10
   • numpy: floor + arange
2. Define and Unpack Tuple: rmd | pdf | html
   • Define/deal multiple variables on the same line
   • Define tuple in python with and without parenthesis, unpack tuple, get subset of elements.
   • Access tuple element and fail to mutate tuple element.
   • py: isinstance(tp_abc, tuple)
3. List Manipulations and Defaults: rmd | pdf | html
   • Conditional statements based on list length and element value.
   • Provide default for element of a list when list does not have that element.
   • py: lambda + join + append() + if len(X) >= 3 and X[2] is not None + if elif else
4. Python String Manipulation Examples: rmd | pdf | html
   • Count unique elements of a string array, generate frequency list.
   • Search for substring, replace string, wrap string.
   • Display and format numeric string with fstring.
   • Change the decimal rounding given a list of estimates and standard error string arrays.
   • py: zip() + upper() + round() + float() + split() + replace() + ascii_lowercase() + set() + join() + join(filter(None, ls_st)) + all([st_i in ‘1234567890’ for st_i in st])
   • textwrap: fill(st, width = 20)
   • fstring: f + f’{fl_esti_rounded:.{it_round_decimal}f}’
   • random: choice

1.2 Dictionary

1. Python Dictionary Examples and Usages: rmd | pdf | html
   • Generate a dictionary, loop through a dictionary.
   • List comprehension with dictionary.
   • py: dc = {‘key’: “name”, ‘val’: 1} + update() + dc.items() + zip()
   • copy: deepcopy

1.3 Numpy Arrays

1. Numpy Combine Arrays to Matrix: rmd | pdf | html
   • Arrays to matrix.
   • numpy: column_stack() + random.choice() + reshape()

2 Pandas

2.1 Panda Basics

1. Python Pandas Conditional Selection of Selectiotn Rows and Columns: rmd | pdf | html
   • Select subset of rows or columns based on cell value conditions.
   • pandas: pd.DataFrame() + replace([‘Zvcss’, ‘Dugei’], ‘Zqovt’) + df.loc[df[‘c5’] == ‘Zqovt’]
2. Convert Data Structures to Pandas Dataframes: rmd | pdf | html
   • Converted nested dictionary to pandas dataframe.
   • Convert from pandas dataframe to a nested dictionary.
   • pandas: DataFrame.from_dict() + reset_index() + set_index() + to_dict()
3. Dataframe Export as CSV with Automatic File Path and Name: rmd | pdf | html
   • Export a pandas dataframe to csv, store automatically in user home download folder.
   • File name contains the variable name, use fstring to get variable name as file string.
   • pandas: df2export.to_csv(spn_csv_path, sep=”,”)
   • pathlib: home() + joinpath() + mkdir(parents=True, exist_ok=True)
   • fstring: f’{mt_abc=}’.split(‘=’)[0]
   • time: strftime(“%Y%m%d-%H%M%S”)

3 Functions

3.1 Function Arguments and Returns

1. Python Function Data Type Handling: rmd | pdf | html
   • Check if parameter is string or integer, conditional execution and exception handling.
   • Check if parameter is string or an integer between some values.
   • py: type + isinstance(abc, str) + isinstance(abc, int) + raise + try except
2. Tuple and Dictionary as Arguments with args and kwargs: rmd | pdf | html
   • Update default parameters with dictionary that replaces and appends additional key-value pairs using kwargs.
   • Pass a dictionary for named arguments to a function.
   • Python function None as default for mutable list argument.
   • python: dict3 = {dict1, dict2} + dict1.update(dict2) + func(par1=’val1’, kwargs)
3. Command Line Argument Parsing Positional and Optional Arguments: rmd | pdf | html
   • Parse parameters entered via command line to call a python script.
   • Optional and positional arguments of different data types (int, str, etc.).
   • Default values, allowed list of values.
   • argparse: parser.add_argument() + parser.parse_args()
4. Function value returns: rmd | pdf | html
   • Return one or multiple values from function.
   • python: return a, b, c

3.2 Operators

1. Python Operators: rmd | pdf | html
   • Python single line conditional tenary opoerators.
   • python: st_a if st_z in st_x else st_b

3.3 Exceptions

1. Python Raise, Try and Catch Exceptions: rmd | pdf | html
   • Raise an Exception in a python function, try and catch and print to string.
   • Trace full exception stack.
   • python: raise + try except + ValueError + TypeError
   • traceback: print_exc()

4 Statistics

4.1 Linear Regression

1. Regression in Python with Statsmodels: rmd | pdf | html
   • Linear regression in Python with Statsmodels.
   • Not full ranked regression, more coefficients than observations.
   • statsmodels: OLS() + add_constant()

4.2 Markov Process

1. Markov Transition Conditional Probability Check Sum to 1: rmd | pdf | html
   • Generate a sample 50 by 50 markov transition matrix.
   • Check row sums for approximate equality to 1.
   • numpy: allclose + reshape + sum

5 Mathematics

5.1 Differentiate

1. Derivatives Involving Normal CDF and PDF Analytically and with Sympy: rmd | pdf | html
   • Sympy derivative examples, the chain rule. Analytically and numerically check against sympy results.
   • Derivatives of standard normal CDF and normal PDF.
   • py: lambda
   • numpy: sqrt + exp + column_stack(()).astype(float) + array + empty + linspace
   • scipy.stats: norm.pdf + norm.cdf
   • sym: diff + subs + simplify

5.2 Production Functions

5.3 CES Production Functions

1. Constant Elasticity of Substitution Production Function with Multiple Inputs: rmd | pdf | html
   • Optimal expenditure minimizing input choices and aggregate marginal cost given a Constant Elasticity of Substitution production function with N inputs.
   • numpy: dot + outer + multiply + matmul + dot

6 Tables and Graphs

6.1 Matplotlib Base Plots

1. Mabplotlib Scatter and Line Plots: rmd | pdf | html
   • Plot several arrays of data, grid, figure title, and line and point patterns and colors.
   • Plot out random walk and white noise first-order autoregressive processes.
   • matplotlib: subplots() + ax.plot() + ax.legend() + ylabel() + xlabel() + title() + grid() + show()
   • numpy: random.normal() + random.seed() + cumsum() + arange()
2. Mabplotlib Text Plots: rmd | pdf | html
   • Print text as figure.
   • matplotlib: ax.text()
   • textwrap: fill()
   • json: dump()

7 Amazon Web Services

7.1 AWS Setup

1. AWS Account Set-up and Start Instance: rmd | pdf | html
   • Generate keypair on AWS, launch instance, launch security, ssh access, and AWSCLI.
   • ssh: ssh-agent + ssh-keygen + ssh ec2-user@ec2-52-23-218-117.compute-1.amazonaws.com
   • aws: aws ec2 start-instances + aws ec2 stop-instances + systemctl start amazon-ssm-agent
2. Boto3 Client Service Communications: rmd | pdf | html
   • Start AWS services, send requests etc via boto3.
   • boto3: boto3.client(service, aws_access_key_id, aws_secret_access_key, region_name)

7.2 S3

1. AWS S3 Uploading, Downloading and Syncing, Locally, EC2 and in Docker Container: rmd | pdf | html
   • From EC2 or local computer upload files to S3 folders.
   • Download sync folders with exclusions between local and S3 folders.
   • Download file from S3 to local computer, an EC2 Linux computer, or into a Docker Container.
   • py: platform.release()
   • boto3: boto3.client(‘s3’) + s3.upload_file() + s3.download_file()
   • os: sep

7.3 Batch

1. AWS Batch, Batch Array: rmd | pdf | html
   • Set up python function that uses AWS_BATCH_JOB_ARRAY_INDEX.
   • Register batch task and submit batch array tasks using ECR image, and save results to S3.
   • Batch Array status check until success.
   • yaml: load()
   • boto3: client() + register_job_definition(jobDefinitionName, type, containerProperties, retryStrategy) + aws_batch.submit_job(jobName, jobQueue, arrayProperties={‘size’:10}, jobDefinition) + aws_batch.describe_jobs()

8 Docker Container

8.1 Docker Setup

1. Docker Container Set-Up and Run on AWS: rmd | pdf | html
   • Install Docker on AWS and build Docker image.
   • Start docker container and run programs inside Docker.
   • aws: ssh + yum update -y + amazon-linux-extras install docker -y
   • docker: service docker start + service docker status + docker build + docker images + docker image prune + docker run -t -i fanconda /bin/bash + python /fanProg/invoke/run.py + docker ps -a + docker system df + docker container ls -a
2. AWS Docker Elastic Container Registery (ECR) Update and Push: rmd | pdf | html
   • Update and push to Elastic Container Registry (ECR) with newly built Docker image.
   • Pull from Elastic Container Registry docker image.
   • scp: scp -o StrictHostKeyChecking=accept-new -i
   • aws: aws ecr get-login
   • docker: docker login + docker build + docker tag + docker push + docker pull

9 Get Data

9.1 Environmental Data

1. CDS ECMWF Global Enviornmental Data Download: rmd | pdf | html
   • Using Python API get get ECMWF ERA5 data.
   • Dynamically modify a python API file, run python inside a Conda virtual environment with R-reticulate.
   • r: file() + writeLines() + unzip() + list.files() + unlink()
   • r-reticulate: use_python() + Sys.setenv(RETICULATE_PYTHON = spth_conda_env)

10 System and Support

10.1 Command Line

1. Execute Python from Command Line and Run Command Line in Python: rmd | pdf | html
   • Run python functions from command line.
2. Run Matlab Command Line Operations: rmd | pdf | html
   • Generate a matlab script and run the script with parameters.
   • subprocess: cmd = Popen(ls_str, stdin=PIPE, stdout=PIPE, stderr=PIPE) + cmd.communicate()
   • decode: decode(‘utf-8’)
   • os: chdir() + getcdw()

10.2 File In and Out

1. Searching for Programs, Reading and Writing to File Examples: rmd | pdf | html
   • Check the path to a particular installed program.
   • Get the parent folder of the current file.
   • Reading from file and replace strings in file.
   • Convert text file to latex using pandoc and clean.
   • py: open() + write() + replace() + [c for b in [[1,2],[2,3]] for c in b]
   • subprocess: call()
   • pathlib: Path().rglob() + Path().stem + Path(spn).parents[1]
   • os: remove() + listdir() + path.isfile() + path.splitdrive() + path.splitext() + path.split()
   • shutil: which()
2. Python Directory and Folder Operations: rmd | pdf | html
   • Join folder names to form absolute path.
   • Folder path slash conversion from system os.sep to forward slash.
   • Generate new folders and files, with existing folder substrings.
   • Generate subfolder recursively.
   • py: open(srt, ‘w’) + write() + close()
   • os: os.sep + os.listdir() + os.path.abspath() + os.path.abspath(os.path.join(os.sep, ‘users’, ‘fan’)) + os.path.join(‘/’, ‘c:’, ‘fa’, ‘fb’) + spn_path.replace(os.sep, ‘/’)
   • pathlib: Path(srt).mkdir(parents=True, exist_ok=True) + [Path(spn).stem for spn in Path(srt).rglob(st)]
   • shutil: shutil.copyfile(‘/fa/fl.txt’, ‘/fb/fl.txt’) + shutil.copy2(‘/fa/fl.txt’, ‘/fb’) + shutil.rmtree(‘/fb’)
   • distutils: dir_util.copy_tree(‘/fa’, ‘/fb’)
3. Python Yaml File Parsing: rmd | pdf | html
   • Parse and read yaml files.
   • yaml: load(fl_yaml, Loader=yaml.BaseLoader) + dump()
   • pprint: pprint.pprint(ls_dict_yml, width=1)

10.3 Install Python

1. Basic Conda Setup Instructions: rmd | pdf | html
   • Conda and git installations
   • bash: where

10.4 Documentation

1. Python Documentation Numpy Doc: rmd | pdf | html
   • Numpy documentation examples.

---

Please contact for issues or problems.