OTRecod.jl

Data recoding

When databases are constructed from heterogeneous sources, it is not unusual that different encodings are used for the same outcome. In such case, it is necessary to recode the outcome variable before merging two databases. The method proposed for the recoding is an application of optimal transportation where we search for a bijective mapping between the distributions of such variable in two databases. In this article, we build upon the work by Garés et al., where they transport the distributions of categorical outcomes assuming that they are distributed equally in the two databases. Here, we extend the scope of the model to treat all the situations where the covariates explain the outcomes similarly in the two databases. In particular, we do not require that the outcomes be distributed equally. For this, we propose a model where joint distributions of outcomes and covariates are transported. We also propose to enrich the model by relaxing the constraints on marginal distributions and adding an L1 regularization term. The performances of the models are evaluated in a simulation study, and they are applied to a real dataset.

Valérie Garès & Jérémy Omer (2022) Regularized Optimal Transport of Covariates and Outcomes in Data Recoding, Journal of the American Statistical Association, 117:537, 320-333, DOI: 10.1080/01621459.2020.1775615

pdf

Installation

In a Julia session switch to pkg> mode to add NPSMC:

julia>] # switch to pkg> mode
pkg> add https://github.com/otrecoding/OTRecod.jl

Alternatively, you can achieve the above using the Pkg API:

julia> using Pkg
julia> pkg"add https://github.com/otrecoding/OTRecod.jl"

When finished, make sure that you're back to the Julian prompt (julia>) and bring OTRecod into scope:

julia> using OTRecod

You can test the package with

julia>] # switch to pkg> mode
pkg> test OTRecod

To run an example from a dataset

OTRecod.run_directory — Function

run_directory(path, method)
run_directory(path, method, outname)
run_directory(path, method, outname, maxrelax)
run_directory(path, method, outname, maxrelax, lambda_reg)
run_directory(
    path,
    method,
    outname,
    maxrelax,
    lambda_reg,
    nbfiles
)
run_directory(
    path,
    method,
    outname,
    maxrelax,
    lambda_reg,
    nbfiles,
    norme
)
run_directory(
    path,
    method,
    outname,
    maxrelax,
    lambda_reg,
    nbfiles,
    norme,
    percent_closest
)
run_directory(
    path,
    method,
    outname,
    maxrelax,
    lambda_reg,
    nbfiles,
    norme,
    percent_closest,
    observed
)

Run one given method on a given number of data files of a given directory The data files must be the only files with extension ".txt" in the directory

path : name of the directory
method : :group or :joint
maxrelax: maximum percentage of deviation from expected probability masses
lambda_reg: coefficient measuring the importance of the regularization term
nbfiles: number of files considered, 0 if all the data files are tested
norme : 0, 1 or 2, norm used for distances in the space of covariates
percent_closest: percent of closest neighbors taken in the computation of the costs (both distance and regularization related)
observed: if nonempty, list of indices of the observed covariates; this allows to exclude some latent variables.

source

julia> using OTRecod

Instance

OTRecod.Instance — Type

struct Instance

Definition and initialization of an Instance structure

datafile : file name
norme : ( 1 : Cityblock, 2 : Euclidean, 3 : Hamming )
indXA : indexes of subjects of A with given X value
indXB : indexes of subjects of B with given X value

source

Solution

OTRecod.Solution — Type

mutable struct Solution

tsolve : solution time
jointYZA : joint distribution of Y and Z in A
jointYZB : joint distribution of Y and Z in B
estimatorZA : estimator of probability of Z for individuals in base A
estimatorYB : estimator of probability of Y for individuals in base B

source