Getting started with data warehouse modernisation

Data Warehouse

Simon Spring, Account Director EMEA, at WhereScape outlines how we should begin our journey modernising our data warehouses.

In the last few years, data automation technology has really ‘crossed the chasm’ into the mainstream. Leaders no longer see automation as a nice-to-have luxury, but a necessity to compete in an increasingly data-centric market. A report by IDC predicts that by 2025 data creation will grow to an enormous 163 zettabytes (ZB), ten times the amount of data produced in 2017.

In anticipation of the surge in the amount of data that consumers and businesses are creating, data warehouse development is under pressure to modernise. We will start to see automation and machine-to-machine technologies shifting the bulk of data creation away from traditional sources. Consequently, organisations should start to consider realigning current business goals, provisioning data for existing and future business cases, and leveraging new platforms and data-driven tools.

Data is beginning to be viewed holistically with a company-wide strategy, and more often than not automation holds the key.

An ‘automation first’ philosophy


Automation is becoming central to this process, because it provides the modern tooling required for data warehouse design, development and administration. With an ‘automation first’ philosophy guiding data warehouse development, developers can fix outmoded development methodology and practices. In doing so, it becomes possible to address the shortcomings of traditional approaches where productivity, flexibility, reuse and adherence to standards are limited.

This isn’t just about how much data can be stored and processed or how quickly meaning can be derived from it. Adding data warehouse automation software to the mix can deliver even more efficiencies and value in a much faster timeframe than hand-coding or using native tools without automation –  in fact, it simplifies development to minimise both effort and risk in data integration and infrastructure projects. This allows companies to focus their effort and resources on providing analytic value to the businesses.

The key infrastructure enabler, however, is the adoption of cloud data warehousing as a solution to cope with growing data volumes. A cloud data warehouse is a database service hosted online by a public cloud provider. It has the functionality of an on-premises database but is managed by a third party, can be accessed remotely and its memory and compute power can be reduced or increased instantly.

In complete contrast to legacy approaches, these cloud-based services have empowered organisations to pay only for the resources they need. As such, data warehouses built on cloud infrastructure offer a foundation for modernisation in businesses committed to data-driven excellence.

Cloud data warehousing brings additional benefits to analytical data infrastructures, from agility and cost-effectiveness to scalability and performance. And specifically, building a cloud-based data warehouse with the benefits of automation enables end-users to design or prototype new analytic components without having to spend large sums of money on infrastructure. This fast-tracks new infrastructure projects and increases development and operation capabilities.

What to consider when selecting a cloud data warehouse


Choosing a cloud data warehouse solution should begin with a cost analysis to estimate how much money it could save the business. Different cloud providers have different pricing structures, with more established names, such as Amazon and Microsoft, renting out nodes and clusters, so every user has a defined section of the server. This makes pricing predictable and constant, but can be a disadvantage in that shared servers sometimes require maintenance.

Other modern cloud-based data warehouse platforms, such as Snowflake, provide elastic compute functionality allowing companies to ensure that they can easily adjust where and how the resources, such as costs, are being used. The ultimate goal is to make the design, development, deployment and operation of data warehouses quicker and cheaper, so teams can deliver projects in hours and days.

Each cloud provider has its own suite of supporting tools for functions such as data management, visualisation and predictive analytics, so these needs should be factored when deciding on which provider to use.

Understanding the benefits and options available in modernising the data warehouse, organisations can create effective strategies that not only optimise the selection of tools and data migration processes but build effective coordination between teams and stakeholders. Modernising the data warehouse and looking to the cloud can have a transformational effect; supporting businesses in the influx of big data, automating manual processes and maximising the return on cloud investment.


Bekki Barnes

With 5 years’ experience in marketing, Bekki has knowledge in both B2B and B2C marketing. Bekki has worked with a wide range of brands, including local and national organisations.

How E-commerce Marketers Can Win Black Friday

Sue Azari • 11th November 2024

As new global eCommerce players expand their influence across both European and US markets, traditional brands are navigating a rapidly shifting landscape. These fast-growing Asian platforms have gained traction by offering ultra-low prices, rapid product turnarounds, heavy investment in paid user acquisition, and leveraging viral social media trends to create demand almost in real-time. This...

Why microgrids are big news

Craig Tropea • 31st October 2024

As the world continues its march towards a greener future, businesses, communities, and individuals alike are all increasingly turning towards renewable energy sources to power their operations. What is most interesting, though, is how many of them are taking the pro-active position of researching, selecting, and implementing their preferred solutions without the assistance of traditional...

Is automation the silver bullet for customer retention?

Carter Busse • 22nd October 2024

CX innovation has accelerated rapidly since 2020, as business and consumer expectations evolved dramatically during the Covid-19 pandemic. Now, finding the best way to engage and respond to customers has become a top business priority and a key business challenge. Not only do customers expect the highest standard, but companies are prioritising superb CX to...

Automated Testing Tools and Their Impact on Software Quality

Natalia Yanchii • 09th October 2024

Test automation refers to using specialized software tools and frameworks to automate the execution of test cases, thereby reducing the time and effort required for manual testing. This approach ensures that automation tests run quickly and consistently, allowing development teams to identify and resolve defects more effectively. Test automation provides greater accuracy by eliminating human...

Custom Software Development

Natalia Yanchii • 04th October 2024

There is a wide performance gap between industry-leading companies and other market players. What helps these top businesses outperform their competitors? McKinsey & Company researchers are confident that these are digital technologies and custom software solutions. Nearly 70% of the top performers develop their proprietary products to differentiate themselves from competitors and drive growth. As...

The Impact of Test Automation on Software Quality

Natalia Yanchii • 04th October 2024

Software systems have become highly complex now, with multiple interconnected components, diverse user interfaces, and business logic. To ensure quality, QA engineers thoroughly test these systems through either automated or manual testing. At Testlum, we met many software development teams who were pressured to deliver new features and updates at a faster pace. The manual...