Cloud Data Warehouse: An Smb’S Perfect Launch Point For Cloud Transformation

ico share

The process of modernizing and moving a company’s IT applications and infrastructure to the cloud, known as “cloud transformation”, continues to accelerate and will have a profound impact on businesses of every size and in every industry over the coming 5-10 years.  The compelling benefits to businesses of shifting the burden of hosting and managing their data and software applications to a 3rd party cloud service provider such as AWS, Microsoft, or Google are driving this trend.  At a high level, these include significant cost and operational savings, enhanced performance, reliability, scalability, security and a better foundation to become data driven for the benefit of customers and operations.

The Challenge: Where to Start?

Most Fortune 2000 companies are all-in on the cloud.  They have invested incredible resources over the past decade and are seeing significant results.  Most small to medium-sized businesses (SMBs), however, struggle with how to start this shift; they know it’s important to their continued growth and competitive survival, but the road to the cloud can be a daunting one.  The challenge for most SMBs is that they typically have a patchwork of legacy IT systems that are critical to their business; small IT teams that are spread thin and usually lack cloud expertise and experience; and have tight budgets.

Many have dipped their toe into the cloud by using a cloud backup service or replacing an existing on-premise application with a Software as a Service (“SaaS”) version of that app.  A good example of this is shifting from a server-based CRM application to a cloud-based one, such as Salesforce.com.  The majority of SMB’s applications and infrastructure, however, are still housed on-premise and they haven’t  tapped into the real power of the cloud.

A Cloud Data Warehouse: The Perfect Starting Point to Realize Value

A cloud data warehouse can be a great launch point for a more strategic move by an SMB to the cloud.  It consolidates data from across an organization to the cloud enabling impactful analytics, dashboards and applications to be built quickly.  A cloud data warehouse enables a company to harness the cloud and can be implemented for a price that fits an SMB’s budget.

As a refresher, a data warehouse is a central repository of data drawn from applications and data sources throughout a company.  It takes work to set up but when done properly provides easy ongoing access to company-wide data for developing valuable reporting, analytics and dashboards, all critical tools for better management and operational insights. A data warehouse eliminates the painful and time consuming manual process of pulling data from various applications and copying it onto a massive spreadsheet in an effort to unify this disparate data for reporting and analysis.  A cloud data warehouse is simply a data warehouse that resides in the cloud.  It is populated by data from both the SMB’s on-premise and cloud applications.

 

 

graphic-cloud

To facilitate an SMB’s move to a cloud data warehouse a turnkey service can be provided by a third party, such as Accur8 Software.  Called Data Warehouse as a Service (“DWaaS”), this approach uses a proven technology platform in combination with cloud services to set up and maintain the business’s data warehouse on a cloud platform.  This turnkey service approach can be ideal for an SMB as they do not have to devote resources—if they even have them–to deal with the intricacies of building and maintaining a cloud data warehouse on their own.  Importantly, data warehouse technologies as a whole have advanced rapidly this decade.  They are more powerful and flexible than ever, performance is orders of magnitude better and the set up and ongoing integration of data into the data warehouse are much easier.  There are a number of steps used by Accur8 to set up a cloud data warehouse and get it quickly into production:

  1. Review the short and long term objectives of the data warehouse with the SMB’s business and IT  teams, to, select the data sources that will be included to begin; and, understand the performance requirements needed by the business.
  2. Utilize Accur8’s data virtualization technology, which is a way of abstracting data from source systems, providing easy access to the underlying data, schema, security model, format of each application that will be included in the data warehouse. Accur8’s tools then streamline the process of data discovery, data cleansing and the building of the composite data model that will govern the data warehouse.  This includes understanding the characteristics of the queries and data transformations that underlie the customer’s reports and dashboards and how they map to its data sources.  We save significant time by quickly iterating the composite data model with the customer, on a virtual basis, using live data and transforms, while avoiding traditionally time consuming processes such as prepping and loading the data warehouse with new data after each change.
  3. Determine the best data warehouse technology for the customer. Our tooling enables Accur8 to  quickly test the behavior and performance of the company’s data on various cloud databases (PostgreSQL, Aurora, RedShift, MySQL, Presto, etc.) to find the best fit given the customer’s requirements.
  4. Use our integration engine to map the pathways data will travel from each source system into the data warehouse.  This includes setting up row or table level integration that can be performed on a real time, triggered, or scheduled basis according to a customer’s need. For large, complex datasets our engine automates the mapping process to reduce time and limit human errors.
  5. Connecting the customer’s existing reporting system (Tableau, Sisense, Power BI, Crystal Reports, etc.) to the cloud data warehouse.
  6. Testing, optimizing and fine tuning is undertaken with reports and dashboards generated using live data from the new data warehouse.
  7. Ongoing care and maintenance is provided post-production, which includes including monitoring, bug fixes, data validation and backups.

Benefits Realized

Most Fortune 2000 companies are all-in on the cloud.  They have invested incredible resources over the past decade and are seeing significant results.  Most small to medium-sized businesses (SMBs), however, struggle with how to start this shift; they know it’s important to their continued growth and competitive survival, but the road to the cloud can be a daunting one.  The challenge for most SMBs is that they typically have a patchwork of legacy IT systems that are critical to their business; small IT teams that are spread thin and usually lack cloud expertise and experience; and have tight budgets.

With a cloud data warehouse and reporting system in place there are a host of business and technology benefits a company will realize:

  1. An SMB can leverage the advantages of the leading edge, multi-billion dollar infrastructure their cloud provider offers.  These include superior scalability, reliability, security and expertise compared to operating their own on-premise infrastructure.
  2. The cloud computing capacity for the data warehouse can be notched up or down as required. For example, a few days before and after month end the computing power can be increased to handle larger workloads and then reduced for the rest of the month.
  3. The cloud data warehouse can be seamlessly scaled in size and breadth (addition of new fields to the data model) as the SMB’s business needs and data volumes grow, so there’s no risk of outgrowing your database.
  4. Achieve up to 85% cost savings compared to an on-premise data warehouse. These savings come from reductions in the overhead costs of building and maintaining a data warehouse; leveraging low cost open source software for key components such as the underlying database; and provisioning, managing and maintaining servers and other related infrastructure.
  5. The ability to take advantage of an ever-growing range of web services that are natural extensions with a cloud data warehouse in place such as AI and machine learning, all of which can bring real value to the business.

For companies looking for a quick start there is the cloud data mart.  A data mart is in essence a small data warehouse that houses data from as little as two or three applications related to one functional area of a business such as sales or HR. The cloud data mart offers all of the above benefits to one specific area of a business.

A Real World Example

Accur8 recently built a cloud data warehouse for a leading revenue cycle management company headquartered in the Northeast. The company provides a range of services such as medical billing, coding and practice management for over several thousand medical service providers (i.e. doctors, clinics, hospitals) across the United States. They have experienced significant growth organically and through acquisition. As a result, their business operates an array of applications including eight different billing systems. They wanted to move their data warehouse from an on-premise legacy system to the cloud. They knew such a move would save costs, improve performance and free up the computing capacity of their existing hardware.

Accur8 used its data virtualization tooling to streamline the data discovery, data modeling and development of the composite data model and schemas that would govern the data warehouse. Accur8 then used its tooling to test, virtually, the behavior and performance of the company’s data on various data warehouse technologies. Given the real time requirement of integrating over 100 million records per day from dozens of applications and data sources with complex structures it was determined that supplementing Accur8’s integration engine with Kafka (an open source data streaming technology) and using PostgreSQL as the data warehouse on Amazon Web Services (AWS) was the best path for the company.

The process of mapping data from source systems to the data warehouse was automated using Accur8’s integration engine. In this instance the company connected Accur8’s reporting tool to the cloud data warehouse to generate its reports, analytics and dashboards. Accur8 then completed the testing and optimizing process with input from the customer’s team. The company did, indeed, realize significant benefits. It freed up over 50% of the workload capacity in its legacy systems, enabling them to extend their life span significantly. Importantly, the user experience of both staff and clients using their reporting system improved significantly with complex reports, analytics and dashboards being generated, in many instances, more than 10x faster. At month end they take advantage of the cloud’s elastic compute capacity by provisioning more server instances for a few days to to process the heavy loads on their data warehouse so user’s don’t see a slowdown in performance.

What’s Next: Apps, Apps, Apps

Companies are deploying applications of all kinds across their businesses that are taking how they operate and serve their customers to new levels. Machine learning and AI are becoming more commonplace and affordable to the point they are a growing reality for many SMBs. Examples include fraud detection, predicting new sales or handicapping customer churn. There are now web services that provide the platform which enables data scientists to explore a company’s data, find patterns and correlations and develop learning models for these types of applications quickly and cost effectively–as long as the underlying data is stored in the cloud. This makes a well built, cloud data warehouse a great starting point as it takes care of the arduous process of gathering and assembling large swaths of a company’s data in the cloud.

Accur8’s approach of building cloud data warehouses with data virtualization technology enables the development of custom applications quickly and cost effectively. Accur8’s rapid application development tooling uses the data virtualization layer to enable easy access to the customer’s underlying data stored in the data warehouse or other data sources. This greatly reduces the complexity of handling backend systems. It enables us to quickly build customer-specific applications where the security, business logic, and data access are all handled seamlessly. User interface (“UI”) requirements are developed by modifying pre-built UI templates to meet the customer’s requirements. This can include individual user’s views, workflows and analytics. With this approach, cloud-based applications can be built and deployed in weeks not months.

graphic-accur8-integration-engine

The range of custom applications that can be built to improve customer services or business operations is almost unlimited and depends on the nature of a particular business and its existing data and application infrastructure. Typically a good starting point is to build applications that bring together data that has historically been difficult to access or that automates a process that manually integrates data using spreadsheets. One simple example is an application that integrates data from a company’s help desk system and CRM system, such as SalesForce. With such an application in place sales reps can know immediately if one of their customers is experiencing issues and can take action to remedy the situation. There is no need to wait until the end of the week or month for a spreadsheet summary or report.

Conclusion

SMBs instinctively know they need to move their businesses more rapidly to the cloud given all of the benefits of doing so. The problem they face is determining how to chart a meaningful path to the cloud that meets their unique strategic needs, IT resources and budget. A cloud data warehouse can be an excellent starting point. It moves the center of gravity of a company’s data to the cloud, providing the foundation to become more data driven with leading edge analytics, dashboards and applications such as those driven by AI. Importantly it can be implemented inside a timeframe and budget that suits an SMB.