Getting started with data warehouse modernisation
Simon Spring, Account Director EMEA, at WhereScape outlines how we should begin our journey modernising our data warehouses.
In the last few years, data automation technology has really ‘crossed the chasm’ into the mainstream. Leaders no longer see automation as a nice-to-have luxury, but a necessity to compete in an increasingly data-centric market. A report by IDC predicts that by 2025 data creation will grow to an enormous 163 zettabytes (ZB), ten times the amount of data produced in 2017.
In anticipation of the surge in the amount of data that consumers and businesses are creating, data warehouse development is under pressure to modernise. We will start to see automation and machine-to-machine technologies shifting the bulk of data creation away from traditional sources. Consequently, organisations should start to consider realigning current business goals, provisioning data for existing and future business cases, and leveraging new platforms and data-driven tools.
Data is beginning to be viewed holistically with a company-wide strategy, and more often than not automation holds the key.
An ‘automation first’ philosophy
Automation is becoming central to this process, because it provides the modern tooling required for data warehouse design, development and administration. With an ‘automation first’ philosophy guiding data warehouse development, developers can fix outmoded development methodology and practices. In doing so, it becomes possible to address the shortcomings of traditional approaches where productivity, flexibility, reuse and adherence to standards are limited.
This isn’t just about how much data can be stored and processed or how quickly meaning can be derived from it. Adding data warehouse automation software to the mix can deliver even more efficiencies and value in a much faster timeframe than hand-coding or using native tools without automation – in fact, it simplifies development to minimise both effort and risk in data integration and infrastructure projects. This allows companies to focus their effort and resources on providing analytic value to the businesses.
The key infrastructure enabler, however, is the adoption of cloud data warehousing as a solution to cope with growing data volumes. A cloud data warehouse is a database service hosted online by a public cloud provider. It has the functionality of an on-premises database but is managed by a third party, can be accessed remotely and its memory and compute power can be reduced or increased instantly.
Trending
In complete contrast to legacy approaches, these cloud-based services have empowered organisations to pay only for the resources they need. As such, data warehouses built on cloud infrastructure offer a foundation for modernisation in businesses committed to data-driven excellence.
Cloud data warehousing brings additional benefits to analytical data infrastructures, from agility and cost-effectiveness to scalability and performance. And specifically, building a cloud-based data warehouse with the benefits of automation enables end-users to design or prototype new analytic components without having to spend large sums of money on infrastructure. This fast-tracks new infrastructure projects and increases development and operation capabilities.
What to consider when selecting a cloud data warehouse
Choosing a cloud data warehouse solution should begin with a cost analysis to estimate how much money it could save the business. Different cloud providers have different pricing structures, with more established names, such as Amazon and Microsoft, renting out nodes and clusters, so every user has a defined section of the server. This makes pricing predictable and constant, but can be a disadvantage in that shared servers sometimes require maintenance.
Other modern cloud-based data warehouse platforms, such as Snowflake, provide elastic compute functionality allowing companies to ensure that they can easily adjust where and how the resources, such as costs, are being used. The ultimate goal is to make the design, development, deployment and operation of data warehouses quicker and cheaper, so teams can deliver projects in hours and days.
Each cloud provider has its own suite of supporting tools for functions such as data management, visualisation and predictive analytics, so these needs should be factored when deciding on which provider to use.
Understanding the benefits and options available in modernising the data warehouse, organisations can create effective strategies that not only optimise the selection of tools and data migration processes but build effective coordination between teams and stakeholders. Modernising the data warehouse and looking to the cloud can have a transformational effect; supporting businesses in the influx of big data, automating manual processes and maximising the return on cloud investment.
