Let’s talk about data.

When hearing the word data as an engineer it, in turn, conjures the word information. It might be files in folders, in object stores, or it may be tables in databases, or ‘unstructured’ in some language or format describing complex possibly related objects. I have watched an engineer or developer more than once in my career use their hands to describe data and / or talk about its “shape” as if it were a physical object sitting on the table between us.

Data can be validated, interrogated, analyzed, assessed, aggregated, and understood, both for its content and its form. It can be quantized and transformed, and regardless of its content data can almost always be defined exactly.

But this isn’t the kind of data we are talking about.

Data with a capital ‘D’ by comparison to the description above is a magical thing. Whereas data can be found in files, folders, indexes, the natural habitat of data with a capital ‘D’ is meetings where it gravitates toward higher level abstract planning activities especially associated with management.

Unlike the kind of information used by engineers this kind of data cannot be quantified or understood easily—in fact—it actively resists attempts to do so. Asking too many questions about the Data or defining its exact content in any way may limit its ability to solve the next problem we imagine.

While regular data may be thought of as having a “shape” that can be defined, Data can change shape as required via ELT, munging (whatever that is), mapping, or some other means usually accompanied by a wave of the hand (like magic). Data can even change from being structured to unstructured as the problem it is being used to solve requires.

The phenomena isn’t just limited to data.

Years ago when working with digital premedia workflows in the printing industry I was pulled into a meeting with the rest of my team where the manager of the department told us that we had to enable a 50% reduction in time for some specific customer accounts. This was to be done using automation.

Native files would come from the customer, we needed to sort these, validate them and then output print ready files from them. When we mentioned that we could perhaps get close if a person preprocessed and reviewed the files (although at that point they could just output them) we were told “imagine that there are no people”. When we mentioned the customer would then need to prepare these files in a certain way we were told the customer wouldn’t be doing anything different.

Over the previous few years we had made gains in the department of this kind of magnitude but here we were told we had 2 months to complete this.

In our work as integrators and workflow specialists automation for us was as quantifiable as data in our first example. We would build process maps, determine which parts of workflows required human interaction, which parts could be configured and automated, and who / where / how these configurations would be selected. We would ask questions of customer service representatives, pushing back on requirements that were particularly inefficient, asking how they had come into existence and if they were still relevant.

As a team we had increased efficiency of page production by thousands of hours with standardized workflows and integrations with products like Dalim Twist (and later Esprit). We integrated Elpical Claro (now Pixometry) building our own web service and calling it via a custom Twist tool developed with Dalim’s Twist SDK—we had efficiency gains so high we had paid for the software, server, installation and development in the first 3 weeks of processing images. But this time every time we asked a question about how this could be done we were met with the same answer—just automate it!

I eventually asked if it were possible—did they think that we hadn’t done this because we simply hadn’t wanted to, or that we hadn’t thought just to “automate it” ourselves?

My boss told me that it didn’t seem like I wanted to help or be successful. The manager of the department confirmed later in my office in a conspiratorial whisper what you may have been thinking already—that the decision to lay off half of the staff that worked on these clients had already been made and therefore his idea of “fixing it” with automation now “had to work”—or more precisely WE had to make it work.

This wasn’t just any automation, this was Automation with a capital ‘A’ and just like Data with a capital ‘D’ the possibilities were endless.

So you may be expecting me to give a third example of this phenomenon—except two letters, in the current zeitgeist—but I will leave that one for you to piece together yourself. After seeing this a few times in various jobs I am certain that there are as many variations of this as there are industries where people gather in conference rooms to brainstorm the next phase of the roadmap.

So it is probably clear that these phenomena are some kind of communication issue. Years ago I would have told you that the exact problem was the language, the imprecision of it, the lack of a common nomenclature, and the inability on both sides to capture the exact ideas. That was before I spent a lot of time in these kinds of rooms having these kinds of conversations.

The reason I now call them “data” and “data with a capital D” is very straightforward—because they aren’t the same thing. Often in these conversations the two terms don’t even represent the same concepts. The issue isn’t the imprecision of the language—for both parties in these conversations—the language is very precise, just at describing two different ideas. The issues begin when either side assumes the other has the same conceptual picture of what is being discussed.

Let’s go back to the word “munge” which is often used to mean manipulate data. “Manipulate data” is quite a general term and can be interpreted differently by the visionaries in the room (the “Data” people) and the tacticians (or the “data” people).

Our “Data” person describes getting data from a certain “data” source and combining it with the existing system to solve a problem. The “data” tactician hears this and having some familiarity with the particular source says that the data in question won’t fit the system—meaning to them, that the content of the data doesn’t solve the problem. The “Data” strategist suggests “munging” the data—meaning to them that a new source can be selected (manipulating their ideas) to solve the problem. In turn, our tactician hears that we can change the structure of the exact source somehow (manipulating the data) and wonders how that is supposed to work.

The likely outcome here (which I have seen) is that everyone in the room gets frustrated and the tactician thinks that the strategist doesn’t understand the technical problems (which they probably don’t and aren’t trying to solve) and the strategist thinks the tactician is inflexible and won’t change their ideas to solve the problem.

If we put the damage of a frustrating interaction aside—the best result of this is that everyone walks away without a plan. With the worst result being that the tactician and the strategist agree on what seems like the same solution but is actually two different ideas, neither of which will solve the problem. Doing the wrong thing is often more expensive—in lots of ways—than doing nothing.

My description of how I discovered Automation with a capital ‘A’ highlights a different problem than the “munging” example.

In this case management were presupposing a tactic (we’ll automat it) to negate the results of a strategic decision they had already made ie: laying off half the staff. This is still somewhat of a communication issue because in this case tacticians hear: make the decision we made make sense (ie: mitigate it 100%) and what is probably closer to the truth is: help us feel like we did something to prepare for the effects of this with the minimal disruption for our customers.

Note: I am being generous when I say “somewhat” in this case. The manager telling me they already decided to lay people off suggests that the reason we’d never saved half the production cost on this segment of work was because we had never been asked to do that or we had never been sufficiently motivated. Neither of those things were true.

So is there a solution to these kinds of problems?

As much as the engineer in me wants to say that the solution is to ensure that visionaries and engineers all use the same language and decide on what problem is being solved—or to put that another way: make the terms “data” and “Data” mean the same thing. I don’t think that is the correct answer.

Data in all its forms, meaning, and language—is magical—both to engineers and visionaries. Data enables customers, solves business problems, generates revenue, and facilitates change and growth. Data solves technical problems, enables scaling, queuing, caching, it reduces costs, and brings insights we may have otherwise not seen.

While the layoffs in my automation example were unfortunate, the reduced costs and increased efficiencies of improving the process did extend the lifespan of that business in an industry that was and is not largely in growth. In that case the systems in place did help the remaining staff complete the work as expected to fulfill the customer needs.

Strategists need to be able to have and talk about big ideas and tacticians need to be able to get down in the weeds and search for viable concrete solutions to well defined problems. So the solution for me starts with knowledge (information if you will)—just like a well defined engineering, business problem, or customer need—awareness is the key.

If everyone in the room understands that we are using the same language or similar sounding language to describe two different concepts we can avoid the usual pitfalls.

A tactician (‘data’ or ‘automation’) who is aware that a strategist (‘Data’ or ‘Automation’) is not trying to solve an engineering problem but define a business case or the germ of a customer need or want can resist the urge to encumber their ideation by trying to figure out exactly how to do that.

A fantastic Solutions Architect I work with often has always done this intuitively. When asked about strategic ideas in meetings he will say “we will look at the actual problem when we come to solving it”. He lets the strategist work on their problem and expects them to let him do the same later, even if we are using the same words to describe the two different things.

Ideas can develop when challenged but there is a difference between saying “does that idea serve the customer’s needs?” and “I am not sure how we would do that?” The first challenges and refines the idea while the second jumps straight to tactics before the problem is fully formed.

Conversely a strategist needs to be aware that often with technical problems no amount of looking at the problem differently, thinking outside of the box, or other methods of manipulating ideas may overcome a hard technical limit. This also applies regardless of how important the issue is to the business (or the strategist themselves) or what decisions have already been made based on these ideas.

I had a real example of this in action: pulled into a meeting by senior management because “I don’t believe with all the smart people we have here we can’t solve this problem”—which was that we needed to use data from a supplier that would take approximately 3 months to download and being as this data updated every half an hour that timeframe was unacceptable (amongst other reasons). Our current data was over a year old and was no longer updated.

After explaining that the suggestion that “we do it at night” would not work because technically half of that 3 month estimate was already at night, then discussing that we were unable to change the laws of physics (that hard technical limit). We eventually decided that using their API to access the data and not downloading it would inform the strategy.

The supplier had their own app built on this API and the strategy was to outperform that application. I think I described this as “trying to build a faster car with the same engine”—but with careful caching (of the things we could), background and long running incremental calls to the API, and a UX that made more sense to our common customers we did actually build something that challenged the supplier’s own implementation. So much so that they inquired about buying our application to replace their own.

So ‘Data’ and certainly ‘data’ may not be truly magical but if we have the clarity to know that we are talking about two different things that relate to the same idea, we stop taking “sides” around an issue and hopefully find that place where either the strategy informs the tactics or the use of a specific tactic gives a strategic advantage and everything just falls into place.

As for the third example of this—the two initials that I am still not going to mention. Until the “magic” becomes a little less magical and the reality becomes a little more real, I’m going to skip writing about that but I wish your tacticians and your strategists the best of luck.

And let me know if you need me in that meeting.