Compute Covariance, Correlation for cov(x,y) given X(a,z), Y(a,z) and f(a,z)

back to Fan's Dynamic Assets Repository Table of Content.

FFT_DISC_RAND_VAR_MASS2COVCOR find cov(x,y) given X(a,z), Y(a,z) and f(a,z)
Default
Parse Parameters
1. Compute Covariance
2. Compute Correlation
Display

function [fl_cov_xy, fl_cor_xy] = fft_disc_rand_var_mass2covcor(varargin)

FFT_DISC_RAND_VAR_MASS2COVCOR find cov(x,y) given X(a,z), Y(a,z) and f(a,z)

Having computed elsewhere E(X), E(Y), and SD(X), SD(Y), and given X(a,z) and Y(a,z), which are the optimal choices along the endogenous state space grid a, and the exogenous state space grid z, and given also f(a,z), the probability mass function over (a,z), we compute covariance and correlation between outcomes X and Y.

Covariance

$\mathrm{Cov}\left(x,y\right) = \sum_{a} \sum_{z} f(a,z) \cdot \left( x(a,z) - \mu_x \right) \cdot \left( y(a,z) - \mu_y \right)$

Correlation

$\rho_{x,y} = \frac{\mathrm{Cov}\left(x,y\right)}{\sigma_x \cdot \sigma_y}$

@param st_var_name string name of the variable (choice/outcome) been analyzed

@param mt_choice_bystates matrix N by M of choices along two dimensions, N could be endogenous states, M could be exogenous shocks, or vice-versa

@param mt_dist_bystates matrix N by M of probability mass on states, N could be endogenous states, M could be exogenous shocks, or vice versa

@return tb_choice_drv_cur_byY table table containing two columns, unique outcomes/choices y from y(a,z) and probability mass associated with each y f(y)

@return ar_choice_prob_byY table array probability mass associated with each y f(y), second column from tb_choice_drv_cur_byY, dimension unknown, determined by y(a,z) function

@return ar_choice_unique_sorted_byY table array unique Ys, dimension unknown, determined by y(a,z) function

@return mt_choice_prob_byYZ matrix f(y,z), meaning for y outcomes along the column dimension.

@return mt_choice_prob_byYA matrix f(y,a), meaning for y outcomes along the row dimension.

@seealso

Default

use binomial as test case, z maps to binomial win prob, remember binom approximates normal.

if (~isempty(varargin))

    % if invoked from outside overrid fully
    [covvar_input_map] = varargin{:};
    bl_display_drvm2covcor = false;

else

    clear all;
    close all;

    it_states = 6;
    it_shocks = 5;
    fl_binom_n = it_states-1;
    ar_binom_p = (1:(it_shocks))./(it_shocks+2);
    ar_binom_x = (0:1:(it_states-1)) - 3;

    % f(z)
    ar_binom_p_prob = binopdf(0:(it_shocks-1), it_shocks-1, 0.5);
    % f(a,z), mass for a, z
    mt_dist_bystates = zeros([it_states, it_shocks]);
    for it_z=1:it_shocks
        % f(a|z)
        f_a_condi_z = binopdf(ar_binom_x - min(ar_binom_x), fl_binom_n, ar_binom_p(it_z));
        % f(z)
        f_z = ar_binom_p_prob(it_z);
        % f(a,z)=f(a|z)*f(z)
        mt_dist_bystates(:, it_z) = f_a_condi_z*f_z;
    end

    % x(a,z), some non-smooth structure
    rng(123);
    mt_choice_x_bystates = ar_binom_x' - 0.01*ar_binom_x'.^2  + ar_binom_p - 0.5*ar_binom_p.^2 + rand([it_states, it_shocks]);
    mt_choice_x_bystates = round(mt_choice_x_bystates*3);

    % y(a,z), some non-smooth structure
    rng(456);
    mt_choice_y_bystates = 10 -(mt_choice_x_bystates) + 15*(rand([it_states, it_shocks])-0.5);

    % Obtain mean and sd
    st_cur_output_key = 'x_outcome';
    [ar_choice_prob_byX, ar_choice_unique_sorted_byX, ~, ~] = ...
        fft_disc_rand_var_mass2outcomes(st_cur_output_key, mt_choice_x_bystates, mt_dist_bystates);
    [ds_stats_x_map] = fft_disc_rand_var_stats(st_cur_output_key, ar_choice_unique_sorted_byX', ar_choice_prob_byX');
    fl_choice_x_mean = ds_stats_x_map('fl_choice_mean');
    fl_choice_x_sd = ds_stats_x_map('fl_choice_sd');

    st_cur_output_key = 'y_outcome';
    [ar_choice_prob_byY, ar_choice_unique_sorted_byY, ~, ~] = ...
        fft_disc_rand_var_mass2outcomes(st_cur_output_key, mt_choice_y_bystates, mt_dist_bystates);
    [ds_stats_y_map] = fft_disc_rand_var_stats(st_cur_output_key, ar_choice_unique_sorted_byY', ar_choice_prob_byY');
    fl_choice_y_mean = ds_stats_y_map('fl_choice_mean');
    fl_choice_y_sd = ds_stats_y_map('fl_choice_sd');

    % display
    bl_display_drvm2covcor = true;

    % Collect
    covvar_input_map = containers.Map('KeyType','char', 'ValueType','any');
    covvar_input_map('mt_choice_x_bystates') = mt_choice_x_bystates;
    covvar_input_map('mt_choice_y_bystates') = mt_choice_y_bystates;
    covvar_input_map('mt_dist_bystates') = mt_dist_bystates;
    covvar_input_map('fl_choice_x_mean') = fl_choice_x_mean;
    covvar_input_map('fl_choice_x_sd') = fl_choice_x_sd;
    covvar_input_map('fl_choice_y_mean') = fl_choice_y_mean;
    covvar_input_map('fl_choice_y_sd') = fl_choice_y_sd;
end

----------------------------------------
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Summary Statistics for: x_outcome
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
----------------------------------------
fl_choice_mean
    0.2019

fl_choice_sd
    4.2414

fl_choice_coefofvar
   21.0090

fl_choice_prob_zero
    0.0717

fl_choice_prob_below_zero
    0.4127

fl_choice_prob_above_zero
    0.5156

fl_choice_prob_max
   3.7187e-06

tb_disc_cumu
    x_outcomeDiscreteVal    x_outcomeDiscreteValProbMass     CDF      cumsumFrac
    ____________________    ____________________________    ______    __________

             -8                         0.0754                7.54     -2.9879  
             -7                       0.022967              9.8367     -3.7842  
             -6                      0.0036146              10.198     -3.8916  
             -5                       0.024097              12.608     -4.4884  
             -4                       0.024097              15.018     -4.9659  
             -3                        0.17865              32.882     -7.6205  
             -2                       0.075861              40.468     -8.3721  
             -1                      0.0080324              41.271     -8.4119  
              0                       0.071696              48.441     -8.4119  
              1                        0.12986              61.427     -7.7686  

    x_outcomeDiscreteVal    x_outcomeDiscreteValProbMass     CDF      cumsumFrac
    ____________________    ____________________________    ______    __________

             -2                        0.075861             40.468     -8.3721  
             -1                       0.0080324             41.271     -8.4119  
              0                        0.071696             48.441     -8.4119  
              1                         0.12986             61.427     -7.7686  
              2                         0.02975             64.402     -7.4739  
              4                         0.22401             86.803     -3.0355  
              5                        0.063069              93.11     -1.4734  
              7                        0.051853             98.295     0.32449  
              8                        0.017043                100     0.99983  
              9                      3.7187e-06                100           1  

tb_prob_drv
    percentiles    x_outcomeDiscreteValPercentileValues    fracOfSumHeldBelowThisPercentile
    ___________    ____________________________________    ________________________________

        0.1                         -8                                 -2.9879             
          1                         -8                                 -2.9879             
          5                         -8                                 -2.9879             
         10                         -6                                 -3.8916             
         15                         -4                                 -4.9659             
         20                         -3                                 -7.6205             
         25                         -3                                 -7.6205             
         35                         -2                                 -8.3721             
         50                          1                                 -7.7686             
         65                          4                                 -3.0355             
         75                          4                                 -3.0355             
         80                          4                                 -3.0355             
         85                          4                                 -3.0355             
         90                          5                                 -1.4734             
         95                          7                                 0.32449             
         99                          8                                 0.99983             
       99.9                          8                                 0.99983             

----------------------------------------
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Summary Statistics for: y_outcome
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
----------------------------------------
fl_choice_mean
    7.3363

fl_choice_sd
    4.6961

fl_choice_coefofvar
    0.6401

fl_choice_prob_zero
     0

fl_choice_prob_below_zero
    0.0398

fl_choice_prob_above_zero
    0.9602

fl_choice_prob_max
    0.0228

tb_disc_cumu
    y_outcomeDiscreteVal    y_outcomeDiscreteValProbMass     CDF      cumsumFrac
    ____________________    ____________________________    ______    __________

           -2.7383                     0.015232             1.5232    -0.0056852
           -2.5789                   3.7187e-06             1.5235    -0.0056865
           -1.3296                     0.018593             3.3829    -0.0090564
          -0.89557                    0.0059499             3.9779    -0.0097828
           0.57961                     0.085679             12.546    -0.0030136
             1.635                      0.02975             15.521     0.0036167
            3.2354                     0.057119             21.233      0.028807
            4.2412                     0.023242             23.557      0.042243
            4.9612                     0.011621             24.719      0.050102
            5.2154                     0.096388             34.358       0.11862

    y_outcomeDiscreteVal    y_outcomeDiscreteValProbMass     CDF      cumsumFrac
    ____________________    ____________________________    ______    __________

           10.323                     0.074374              84.105     0.68725  
           10.352                     0.024097              86.515     0.72125  
           11.593                     0.024097              88.924     0.75933  
           12.046                     0.028917              91.816     0.80681  
           13.029                    0.0013387               91.95     0.80919  
           14.724                     0.046484              96.598     0.90248  
            17.26                    0.0074374              97.342     0.91998  
           19.402                    0.0036146              97.703     0.92954  
           20.991                     0.000119              97.715     0.92988  
           22.515                     0.022848                 100           1  

tb_prob_drv
    percentiles    y_outcomeDiscreteValPercentileValues    fracOfSumHeldBelowThisPercentile
    ___________    ____________________________________    ________________________________

        0.1                      -2.7383                              -0.0056852           
          1                      -2.7383                              -0.0056852           
          5                      0.57961                              -0.0030136           
         10                      0.57961                              -0.0030136           
         15                        1.635                               0.0036167           
         20                       3.2354                                0.028807           
         25                       5.2154                                 0.11862           
         35                        5.409                                 0.12455           
         50                       6.6874                                 0.26953           
         65                       9.7057                                 0.57988           
         75                       9.7057                                 0.57988           
         80                       10.323                                 0.68725           
         85                       10.352                                 0.72125           
         90                       12.046                                 0.80681           
         95                       14.724                                 0.90248           
         99                       22.515                                       1           
       99.9                       22.515                                       1

Parse Parameters

% probability over a and z
params_group = values(covvar_input_map, {'mt_dist_bystates'});
[mt_dist_bystates] = params_group{:};

% x and y outcomes
params_group = values(covvar_input_map, {'mt_choice_x_bystates', 'mt_choice_y_bystates'});
[mt_choice_x_bystates, mt_choice_y_bystates] = params_group{:};

% x and y stats
params_group = values(covvar_input_map, {'fl_choice_x_mean', 'fl_choice_x_sd', ...
                                         'fl_choice_y_mean', 'fl_choice_y_sd'});
[fl_choice_x_mean, fl_choice_x_sd, fl_choice_y_mean, fl_choice_y_sd] = params_group{:};

1. Compute Covariance

mt_x_devi_from_mean = (mt_choice_x_bystates - fl_choice_x_mean);
mt_y_devi_from_mean = (mt_choice_y_bystates - fl_choice_y_mean);
mt_x_y_multiply = (mt_x_devi_from_mean).*(mt_y_devi_from_mean);
mt_cov_component_weighted = mt_dist_bystates.*(mt_x_y_multiply);
fl_cov_xy = sum(mt_cov_component_weighted, 'all');

2. Compute Correlation

fl_cor_xy = fl_cov_xy/(fl_choice_x_sd*fl_choice_y_sd);

Display

if (bl_display_drvm2covcor)

    fft_container_map_display(covvar_input_map, 25, 15);

    covvar_output_map = containers.Map('KeyType','char', 'ValueType','any');
    covvar_output_map('mt_x_devi_from_mean') = mt_x_devi_from_mean;
    covvar_output_map('mt_y_devi_from_mean') = mt_y_devi_from_mean;
    covvar_output_map('mt_x_y_multiply') = mt_x_y_multiply;
    covvar_output_map('mt_cov_component_weighted') = mt_cov_component_weighted;

    fft_container_map_display(covvar_output_map, 25, 15);

    disp('fl_cov');
    disp(fl_cov_xy);

    disp('fl_cor');
    disp(fl_cor_xy);

end

----------------------------------------
----------------------------------------
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Begin: Show all key and value pairs from container
CONTAINER NAME: COVVAR_INPUT_MAP
----------------------------------------
  Map with properties:

        Count: 7
      KeyType: char
    ValueType: any

xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
----------------------------------------
----------------------------------------
pos = 1 ; key = fl_choice_x_mean ; val = 0.20188
pos = 2 ; key = fl_choice_x_sd ; val = 4.2414
pos = 3 ; key = fl_choice_y_mean ; val = 7.3363
pos = 4 ; key = fl_choice_y_sd ; val = 4.6961
pos = 5 ; key = mt_choice_x_bystates ;rown= 6 ,coln= 5
mt_choice_x_bystates :mu= 0.8 ,sd= 5.2614 ,min= -8 ,max= 9
               zi_1_c1    zi_2_c2    zi_3_c3    zi_4_c4    zi_5_c5
               _______    _______    _______    _______    _______

    zi_1_r1      -8         -8         -7         -6         -7   
    zi_2_r2      -5         -3         -3         -4         -2   
    zi_3_r3      -1         -2          1          0          0   
    zi_4_r4       1          2          4          4          4   
    zi_5_r5       4          5          7          5          4   
    zi_6_r6       9          7          8          7          8   

pos = 6 ; key = mt_choice_y_bystates ;rown= 6 ,coln= 5
mt_choice_y_bystates :mu= 8.0791 ,sd= 6.6417 ,min= -2.7383 ,max= 22.5154
               zi_1_c1    zi_2_c2     zi_3_c3    zi_4_c4    zi_5_c5
               _______    ________    _______    _______    _______

    zi_1_r1     12.046      14.724    22.515      19.402     20.991
    zi_2_r2     11.593      6.4991    8.7834      10.352      10.12
    zi_3_r3      5.409      10.323    9.7057      6.6874      17.26
    zi_4_r4     13.029       1.635    5.2154     0.57961    -1.3296
    zi_5_r5     7.2659    -0.89557    7.6379      3.2354     4.2412
    zi_6_r6    -2.5789      10.226    5.4774     -2.7383     4.9612

pos = 7 ; key = mt_dist_bystates ;rown= 6 ,coln= 5
mt_dist_bystates :mu= 0.033333 ,sd= 0.035743 ,min= 3.7187e-06 ,max= 0.12852
                zi_1_c1       zi_2_c2       zi_3_c3      zi_4_c4      zi_5_c5 
               __________    __________    _________    _________    _________

    zi_1_r1      0.028917      0.046484     0.022848    0.0036146     0.000119
    zi_2_r2      0.024097      0.092967     0.085679     0.024097    0.0014875
    zi_3_r3     0.0080324      0.074374      0.12852     0.064259    0.0074374
    zi_4_r4     0.0013387       0.02975     0.096388     0.085679     0.018593
    zi_5_r5    0.00011156     0.0059499     0.036146     0.057119     0.023242
    zi_6_r6    3.7187e-06    0.00047599    0.0054218     0.015232     0.011621

----------------------------------------
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Matrix in Container and Sizes and Basic Statistics
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
                            i    idx    rowN    colN      mean        std          min          max  
                            _    ___    ____    ____    ________    ________    __________    _______

    mt_choice_x_bystates    1     5      6       5           0.8      5.2614            -8          9
    mt_choice_y_bystates    2     6      6       5        8.0791      6.6417       -2.7383     22.515
    mt_dist_bystates        3     7      6       5      0.033333    0.035743    3.7187e-06    0.12852

----------------------------------------
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Scalars in Container and Sizes and Basic Statistics
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
                        i    idx     value 
                        _    ___    _______

    fl_choice_x_mean    1     1     0.20188
    fl_choice_x_sd      2     2      4.2414
    fl_choice_y_mean    3     3      7.3363
    fl_choice_y_sd      4     4      4.6961

----------------------------------------
----------------------------------------
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Begin: Show all key and value pairs from container
CONTAINER NAME: COVVAR_OUTPUT_MAP
----------------------------------------
  Map with properties:

        Count: 4
      KeyType: char
    ValueType: any

xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
----------------------------------------
----------------------------------------
pos = 1 ; key = mt_cov_component_weighted ;rown= 6 ,coln= 5
mt_cov_component_weighted :mu= -0.49051 ,sd= 0.77939 ,min= -2.8167 ,max= 0.24921
                 zi_1_c1       zi_2_c2      zi_3_c3      zi_4_c4      zi_5_c5  
               ___________    _________    _________    _________    __________

    zi_1_r1         -1.117      -2.8167      -2.4977     -0.27048     -0.011703
    zi_2_r2       -0.53357      0.24921       -0.397     -0.30535    -0.0091174
    zi_3_r3       0.018605     -0.48915      0.24304    0.0084172       -0.0149
    zi_4_r4      0.0060823     -0.30498     -0.77645      -2.1987      -0.61199
    zi_5_r5    -2.9828e-05       -0.235     0.074128      -1.1239      -0.27322
    zi_6_r6     -0.0003244    0.0093514    -0.078591      -1.0432      -0.21523

pos = 2 ; key = mt_x_devi_from_mean ;rown= 6 ,coln= 5
mt_x_devi_from_mean :mu= 0.59812 ,sd= 5.2614 ,min= -8.2019 ,max= 8.7981
               zi_1_c1    zi_2_c2    zi_3_c3    zi_4_c4     zi_5_c5 
               _______    _______    _______    ________    ________

    zi_1_r1    -8.2019    -8.2019    -7.2019     -6.2019     -7.2019
    zi_2_r2    -5.2019    -3.2019    -3.2019     -4.2019     -2.2019
    zi_3_r3    -1.2019    -2.2019    0.79812    -0.20188    -0.20188
    zi_4_r4    0.79812     1.7981     3.7981      3.7981      3.7981
    zi_5_r5     3.7981     4.7981     6.7981      4.7981      3.7981
    zi_6_r6     8.7981     6.7981     7.7981      6.7981      7.7981

pos = 3 ; key = mt_x_y_multiply ;rown= 6 ,coln= 5
mt_x_y_multiply :mu= -24.6478 ,sd= 33.3193 ,min= -109.3183 ,max= 19.6461
               zi_1_c1     zi_2_c2    zi_3_c3    zi_4_c4    zi_5_c5
               ________    _______    _______    _______    _______

    zi_1_r1      -38.63    -60.595    -109.32     -74.83    -98.343
    zi_2_r2     -22.142     2.6806    -4.6336    -12.672    -6.1295
    zi_3_r3      2.3163    -6.5769     1.8911    0.13099    -2.0034
    zi_4_r4      4.5433    -10.251    -8.0554    -25.663    -32.914
    zi_5_r5    -0.26737    -39.497     2.0508    -19.676    -11.755
    zi_6_r6     -87.235     19.646    -14.495    -68.488    -18.521

pos = 4 ; key = mt_y_devi_from_mean ;rown= 6 ,coln= 5
mt_y_devi_from_mean :mu= 0.74286 ,sd= 6.6417 ,min= -10.0745 ,max= 15.1791
                zi_1_c1     zi_2_c2    zi_3_c3    zi_4_c4     zi_5_c5
               _________    _______    _______    ________    _______

    zi_1_r1       4.7099     7.3879     15.179      12.066     13.655
    zi_2_r2       4.2566    -0.8372     1.4471      3.0157     2.7837
    zi_3_r3      -1.9272      2.987     2.3694    -0.64883     9.9237
    zi_4_r4       5.6926    -5.7012    -2.1209     -6.7566    -8.6659
    zi_5_r5    -0.070395    -8.2318    0.30167     -4.1009     -3.095
    zi_6_r6      -9.9151     2.8899    -1.8588     -10.075    -2.3751

----------------------------------------
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Matrix in Container and Sizes and Basic Statistics
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
                                 i    idx    rowN    colN      mean        std        min        max  
                                 _    ___    ____    ____    ________    _______    _______    _______

    mt_cov_component_weighted    1     1      6       5      -0.49051    0.77939    -2.8167    0.24921
    mt_x_devi_from_mean          2     2      6       5       0.59812     5.2614    -8.2019     8.7981
    mt_x_y_multiply              3     3      6       5       -24.648     33.319    -109.32     19.646
    mt_y_devi_from_mean          4     4      6       5       0.74286     6.6417    -10.075     15.179

fl_cov
  -14.7154

fl_cor
   -0.7388

end

ans =

  -14.7154