We all know companies are facing growing rivers, lakes and oceans of data as the sheer number of applications they operate (both in-cloud and on-premise) proliferate and the volumes of data flooding through their businesses grows unabated. As Keith Block, COO of SalesForce.com points out “90% of the world’s data was collected in the last few years”.
As you can imagine this is often overwhelming IT departments. They are under pressure from their businesses to make all this data flow smoothly between systems and be accessible for users in a timely fashion so its inherent value can be unlocked. And of course, it needs to be done in a cost effective manner.
Data integration is the pipework and plumbing systems that allow data to flow smoothly between applications and databases to users and systems across the organization. It allows data stored in disparate software systems, each under their unique government of data schemas, security models, etc. to be shared with each other seamlessly or directed to a data warehouse for consumption by a reporting or analytics tool. It is easy to understand that as a company’s application and data environments get more complex and data is growing exponentially the need for better data integration capabilities grow exponentially.
This is where things become interesting, as data integration is not a settled science in any way. There are many options available to solve a wide range of data integration requirements including hand coding, ETL (Extract, Transform, Load) tools, data warehouses, enterprise application integration tools, data federation, data virtualization, and iPaaS platforms (Integration Platform as a Service). Each company has to look at its own unique configuration of software, expertise and budget to determine what is the best fit for it.
There are new data integration tools in the marketplace designed to handle to the needs that arise from company’s complex software application environments. A common feature is that they consolidate functionality that was previously provided by several different tools into one tool. For example, data integration tools and application integration tools were distinctly different historically. Now certain tools combine the functionality of both. This means not needing to shoe horn an ETL tool, designed to aggregate data from multiple sources in batch fashion into a data warehouse, into an application integration tool.
Another capability certain of these new tools have is being able to call row level data from an application in real time. With this ability, only required data is exchanged during integration and systems can be tightly synchronized greatly enhancing the capabilities and performance of a company’s applications. It can also reduce monthly subscription fees payable to software as a service (SaaS) vendors as their rates typically go up if the volume of records a company queries from their software exceeds a certain threshold.
The final aspect to touch upon is these next generation tools allow users to build complex integrations that can pull, push, transform and sync data across multiple applications and data repositories to meet the needs of their business without the need for writing code. This is typically achieved with easy to use drag and drop capabilities integrated with the ability to compose integrations using visual data flow. Not only does this make IT teams more productive, it begins to allow the data integration function to move out from the IT department to business analysts.
IT teams may be experiencing unnecessary frustration as their existing data integration tools have not kept up with the evolution and growing complexity of their data and software application environments. Because of this it may well be time to evaluate the emerging new crop of tools which are designed to handle the complexity of today’s data and application environment.