Serverless data warehousing background
Data warehouses are the places where organizations store and analyse the reams upon reams of data that they collect. Historically, these were eye-wateringly expensive operations that only the largest organizations could afford to build, but with the advent of serverless, third-part hosting sites, data warehousing has become a reality for businesses or all sizes.
These services combine all the necessary aspects of running a data warehouse, from the admin, security and reliability viewpoints, and offer cost-effective pricing plans so organizations can take advantage of their services.
Serverless building blocks
Understanding how a data warehouse works requires understanding of the ever-present building blocks which are integral to data warehouses of every variety. These can be identified as a centralized repository and a data pipeline.
SEE ALSO: Distributed Tracing: Modern Debugging in Times of Serverless
The centralized repository supports storage and analytics. In physical terms, it would be the bricks-and-mortar premises. So if the repository is the physical space, the data pipeline is your transport: how information is sent and received by the repository.
When deciding on the best data pipeline to use (known as the serverless ETL), there are a number of considerations, including:
- What the origins and intended destinations of data are, meaning cloud-based, or on-prem to cloud etc.
- The usual size of the data being sent.
- What type of configurations and usability are required.
- The pricing models which are given for the various required functions (and of course there are plenty).
Should you use a serverless data warehouse?
So the question of course is whether a serverless data warehouse is required for your business needs. Unless you are a massive corporation, then you almost certainly do not need to build your own, because then you need to consider the cost and manpower required to undertake such a task.
With many serverless options available that can store and manage your data, which can be accessed with quick queries, there really is no need to follow any other option, states Trevor Jeffries, a tech blogger at Writemyx and Britstudent.
Here are just some of the benefits:
It really could not be easier to manage. You do not need to hire (and pay) experts because the third-party service does all this for you. The individual building blocks can be managed independently and you can decide at any stage to add or reduce the services that you require.
It can be scaled on demand. As your data warehouse is stored on the cloud, there is no upfront commitment to what you will require, and if those requirements change over time, you have no already committed to them.
It’s cost-effective. For logical reasons already mentioned.
Drawbacks
Like everything, there may be the odd disadvantage. Perhaps the obvious one here is in the question of integration. As Annabell Pieters, a writer at Nextcoursework and 1day2write reasons, “if you have a number of serverless building blocks, integrating these effectively with your existing operation can occasionally be problematic.”
Although they remove the need to hire in-house data warehouse experts, just the setup process can be complex and long-winded. A possible solution here is hiring a project manager on contract who can run with this process, but an element of understanding, particular upon existing management and IT staff will be needed, and of course requirements change over time too.
SEE ALSO: Wide adoption of serverless will be led by tomorrow’s giants, not today’s
Finally, not all solutions are cost effective, as some solution do require up-front payments. And then there is the all-important issue of vendor lock-in which always provides an inherent risk.
What the future looks like?
Undoubtedly the future for most organizations is a serverless, or at least part-serverless approach. There are just too many benefits and not enough drawbacks to going at it this way to prevent serverless data warehouses becoming the norm.
As third-party vendors become more developed, the technology becomes more advanced and accessible, and in-house staff become more knowledgeable, the current drawbacks may not remain so for long. Indeed, even current cost plans should be greatly reduced upon in the not-too-distant future. That is what serverless data warehouses are here to stay, and why it could be time to migrate your company over to this approach.
The post Want to build a serverless data warehouse? These are the points to consider appeared first on JAXenter.
Source : JAXenter
