To Messy Data with…Love or Frustration?
There are times when, as humans, we are inconsistent, frustrated, and attempting to make sense of what has been put in front of us. Well, the apple does not fall far from the tree regarding data - especially since most data are derived and created by humans. There’s even a term for that chaos – “messy data.” And just like us, there is a lot to love about data that can appear messy, but there is a method to its madness if you know how to speak its language:
Constructive Chaos - Messy data can point us to the unexpected. Inconsistencies and outliers can reveal hidden patterns, trends, and correlations you wouldn't have found in clean data.
Context Matters - Consider the “…value of 100—is that a perfect grade, a high Fahrenheit temperature, an age, or an invalid credit rating? You need context (supplied via documentation, help, policies, metadata, etc.) to understand the data correctly and to implement the right data quality checks and rules.”1 Your messy data may not be that messy after all. It just needs the right frame around it to make sense of the picture.
“MacGyver-ing” Your Data Analysis - Working with messy data requires creativity and resourcefulness. It encourages you to devise innovative cleaning methods and to think of novel ways to extract meaning (that don’t require nail polish remover, sunglasses, and a lamp).2 It’s a puzzle that encourages you to improve your problem-solving skills.
Does loving messy data mean leaving it how you found it? On the contrary, working with truly messy data will help you appreciate the patterns hidden in the chaos to create new ways to adapt your imperfect dataset to solve real-world problems. Not sure where to start working with your messy data? Send us an email, and we will help you find the meaning behind the madness.
1. Smith, H. (2018, August 2). Data is inherently messy. Is that really such a bad thing? InfoWorld. https://www.infoworld.com/article/3294937/data-is-inherently-messy-is-that-really-such-a-bad-thing.html
2. Zlotoff, L.D., Kane, M., & Sheppard, J. (Writers) & Vejar, M. (Director). (1990, November 19). Bitter harvest (Season 6, Episode 9) [Television series episode]. In S. Downing (Executive producer), MacGyver. Paramount Television; ABC.
