For More Effective Data Integration, Integrate Your Organizational Teams
An old mantra holds that there are three essential ingredients in building an effective organization: People, tools, and processes. When you get each of these ingredients right, success naturally follows.
When it comes to data integration, however, it can be easy to overlook the importance of the first item on that list—people. It is simple enough to implement the right data integration tools and processes through careful research, design and engineering. But setting up your people for data integration success can be more challenging.
That is true due especially to the fact that at most organizations today, it’s not just data engineers and scientists who play important roles in data integration. These days, virtually everyone interacts with data in one way or another.
This means that, to make data integration as efficient and cost-effective as possible, it’s crucial to organize your people in the right way and empower them to communicate and collaborate effectively. This article explains how.
The Siloed Organization
Setting up your people for successful data integration begins by identifying the silos that separate different individuals and teams from working together effectively.
Here, I’m drawing on the “silo” concept made popular by the DevOps movement. Advocates of DevOps believe that software development and deployment is most effective when different individuals and groups are able to communicate and collaborate constantly, rather than existing inside their respective silos and rarely talking to each other.
The same type of thinking can be applied to data integration. Data integration typically involves passing information and processes between multiple groups; it is rare for a single group to bear total responsibility for collecting, storing, transforming and analyzing data. If your organization is siloed, and the groups within it cannot effectively communicate or hand tasks off to each other, your data integration process will fail.
For example, your organization might have an HR department that is responsible for collecting data about new employees. If the HR department exists in a silo—that is, if its members have little visibility into what other departments are doing, or little understanding of what other departments even do—it will be difficult to ensure that employee data is properly stored, archived and secured.
The developers who write the applications that process the data will have little means of understanding the special needs associated with it, so they will not able to create applications that provide the right level of data security, because they won’t know what that security is.
Similarly, in a siloed organization, data engineers who are responsible for managing data are unlikely to be able to communicate with HR in order to understand the special data management challenges associated with employee data. They may not know how long the data should be retained, or how to deal with data quality problems (such as missing or misspelled names) that they discover in the data sets.
On top of all of this, the HR department’s lack of visibility into what developers and data engineers are doing with the data collected by HR also means that HR employees are unlikely to be able to collect and record data in ways that streamline data integration. They won’t know which particular practices make data integration easier for the teams to which they hand off data.
De-Silo’ing Your Organization
What does breaking down data integration silos look like in practice? It doesn’t mean abandoning all organizational structure; in the example above, your HR department should still continue to operate as a distinct unit.
Instead, breaking down the silos entails allowing each of your departments or groups to communicate effectively with others and understand the challenges they face. It might mean having members of one department spend a few hours each month seeing what other departments do with the data that they collect. It could also mean establishing clear communication channels so that when a glitch occurs at some point along the data integration pipeline, the teams involved can easily talk about it, even if the business responsibilities of those teams do not normally overlap. Even something as simple as asking each department to identify its data integration painpoints, then sharing that information with other departments, could help to eliminate silos.
DevOps advocates would make similar points about about de-siloing software developers from IT Ops teams, by the way. The goal of DevOps is not to remove all barriers between teams or destroy specialization, but rather to build effective communication and visibility bridges between groups that would otherwise not communicate with each other.
Conclusion: De-Silo’ing Your People
Silos are the enemy of effective data organization. They prevent your people—those who work in technical roles, as well as everyone else who helps in some way to collect, store, back up, recover or interpret data—from understanding how they can help each other perform better data integration. If you want to make data integration faster and less expensive, start by eliminating the silos that separate your people form each other.
Learn how to unleash the power of data – Download our eBook: The New Rules for Your Data Landscape.