Pattern, Processes, Procedures, and Problem Solving
In our last article, we covered the investigation of inputs and discovery of data that feeds the information portions of analytics. It was the first step in our process and one that is highly dependent on this next step; aptly named — process.
analytics is only true/reliable if it can be repeated
Like it or not, analytics includes repetition. As noted in all the prior articles, it is a very recursive process. If you have no patience for repetition, you should probably find a different occupational pursuit. Repetition can be difficult unless you adopt good process.
Like all the steps in our process — there is an information (I) portion of process. It is probably the least loved step of them all — documentation. Documentation, definitions, data heritage, and other descriptive material can be very helpful in making sure your analysis is reliable and repeatable. Unfortunately, this sub-step is too often skipped, ignored, or just hastily done.
On the subject of hastily done, one of the most common mistakes of junior analysts is to combine and/or skip steps. Skipping steps leads to starting over… this is the sort of recursion that should be avoided. In analytics, patience and discipline speed the process.
Along the way, patterns emerge. There is recurring organization (O) to the analytic process. Seasoned purveyors learn what to look for over years of experience (and repetition). No matter what subject matter you are dealing with there are clear patterns in all data. The hardest problems/data sets to work with in analytics tend to have the highest complexity or shortest length of time available to create identifiable patterns.
What kind of patterns are we referring to?
- Patterns in the distribution
- And many others
All of which would require an article of their own to detail (stay tuned). Most rely on techniques like visual comparison, benchmark development, and decomposition (also worthy of solo articles).
Process has a nearly endless diversity. It is what can lead to analysis paralysis. There is always more to find. Analysts need to determine (D) the priority of various potential paths. Experienced analysts know how to project probabilities of numerous paths and patterns.
Finally, the analyst should begin to have suspicions (S). A story emerges from this process, allowing the analyst to make predictions and to postulate potential hypotheses for later testing. If too many emerge, there will be more decision-making (D) to follow. There is that recursion again.
But wait… along the way — problems often emerge with the data. Great analysts have a nose for forensic data analysis. As they detail (I) and iterate (P), various patterns (O) may signal a problem (D). In this situation an analyst must look for causality (S). Root cause, unlike more ephemeral causality should be purely technical. The why is a computer procedure or algorithm, not a complicated system (hopefully) or worse a behavior. Some of my frequent readers know that I am a huge cynic of causality claims in those latter two scenarios.
Once root cause is determined, a fix will need to be put in place and the analyst gets to… start again. Sigh. The process churns on, but at least the problem was upstream. Many organizations react poorly to these delays and fail to recognize the value analysts provide from and auditing and integrity perspective. My teams in the financial industry saved our companies billions… that’s a B, but not a new step in the process (just a real big value add).
At the completion of this series and our process, we will add more articles on various techniques in the process stage of analytics. Problem solving, pattern recognition, and prediction are all great topics — but require their own articles. But know this — each stage of this process can really only be understood and mastered by application. Articles like this can only set the stage.
Speaking of which, our stage is now set for the next phase of the process. Coming soon… once we get organized.
Stay tuned and thanks for reading!
Gurupriyan is a Software Engineer and a technology enthusiast, he’s been working on the field for the last 6 years. Currently focusing on mobile app development and IoT.