My goal is to generate EDA reports for new datasets.

Step 1 - python script

To accomplish this, I have written a function in python which is embedded in a script that 1. loads pandas, 2. loads a data set, 3. calls the function on that dataset.

import pandas as pd





def EDA(df, name):



    '''

    name == string version of df's name

    provides count, unique

        to use:

        1. run cmd as administrator

        2. in cmd run 

            chcp 65001                   # enables utf-8 to display in the cmd console

            set PYTHONIOENCODING=utf-8   # enables utf-8 output

        3. python scriptpath.py > Output/dfnameEDA.txt

    '''



    df.name = name

    print('#{}n'.format(df.name))

    for col in df.columns:

        print('#{}n'.format(col))

        print(df[col].describe())

        print('n')

        print(df[col].value_counts(dropna=False))

        print('n')



path = 'Data/active.csv'

active = pd.read_csv(path, encoding='utf-8')



# runs for 'active' dataset. must change for each dataset in directory

if __name__=='__main__':

    EDA(active, name='active')

Step 2 - Administrator CMD commands

two parts here: 1. change output encoding to utf-8, 2. run script and save output as a txt file

My data come from a global database and often have lettering that will not work with the default codec

chcp 65001

set PYTHONIOENCODING=utf-8



python Syntaxnameofscript.py > OutputdatasetEDA.txt

Issues

1. have to create customized script for each dataset


2. have to run cmd commands for each dataset


3. output is one txt per dataset
   Ideally I could generate a .doc file or at least a single .txt file for all datasets in the directory.
   The reason for this is that I would like to expand the function to include histograms and other graphics that will not work with the .txt format.

Overall, it just seems like I shouldn't be doing so many repetitive steps to get this to work. I would ultimately like to run a single command in a directory to create a single EDA report for all datafiles in that directory