Data Cleaning

Curated Learning Resources

• 

  Data Cleaning IS Analysis, Not Grunt Work

  Randy Au, Quant User Researcher, Google

  Cleaning data is considered by some people to be menial work that’s somehow “beneath” the sexy “real” data science work. Randy calls BS on this. The act of cleaning data imposes values/judgments/interpretations upon data intended to allow downstream analysis algorithms to function and give results. That’s exactly the same as doing data analysis. Data cleaning is a spectrum of reusable data transformations on the path towards doing a full data analysis. Once we accept that framework, the steps we need to take to clean data flow more naturally. We want to allow our analysis code to run, control useless variance, eliminate bias, and document for others to use, all in service to the array of potential analyses we want to run in the future.

  Mark as Read
• 

  So You’ve Got a Really Big Dataset. Here’s How You Clean It.

  Li-Lian Ang, Community and Operations Manager, BlueDot Impact

  Li-Lian provides a step-by-step guide to cleaning large datasets in Python, using the Pandas and Matplotlib libraries. She explains how to filter data, standardize missing data labels, clean dependent variables, remove duplicate entries, and check for missing values in each variable and row. She suggests auditing variables by type and providing suggestions for each type, including Boolean, datetime, numerical, categorical, and text.

  Mark as Read