Data Lakes in the Cloud - Best of Both Worlds

February 27, 2018 Brett Carpenter

When you hear the term “data lake” what comes to mind? It probably brings to mind visions of Hadoop and a big repository of data that could be structured or unstructured. Ideally, this data would be accessible from anyone in your organization that would need it. Nowadays, data lakes can be deployed in a variety of environments, most notably on the cloud. Even if you have an on-premises data lake, migrating to the cloud can be beneficial for you. Here are some key considerations for either building a data lake in the cloud or migrating your existing one.

Choosing the “right” cloud provider

Do a search for cloud storage providers and you might be blown away at the number of options. Of course, there are the big dogs of AWS, GCP, and Azure which have been around for years. Now, however, there are new players coming on the scene which can offer faster speeds, seemingly lower costs, or different capabilities.

Choosing the “right” cloud provider comes down to organizational preferences. You should ensure the storage capabilities match your needs and that the vendor can connect to external systems you plan to use or are currently using.

Cloud, on-premises, multi-cloud, or hybrid?

Another benefit of cloud is being able to use multiple vendors. If you wanted to start out with a single cloud instance - you can. If you have multiple departments throughout your organization using multiple clouds, that’s fine too. Many organizations today have a hybrid environment, where they are using both on-premises and cloud storage. But without a common data management solution to manage these multiple stores, all you have created is a set of siloed data ponds.

Choosing a Data Platform

Storing your data in the cloud is one thing. Operationalizing that data is a different story. You’ll want to select a data management platform that can:

  • Migrate data from your legacy systems
  • Ingest new data and organize it at the speed it’s delivered
  • Govern the data you have spread across your entire big data ecosystem
  • Allow self-service access to the data from all who need it
  • And, most importantly, allow you to use any cloud vendor your organization wants

This checklist might seem straightforward, but you’d be surprised at the lack of full capability options available. The Zaloni Data Platform covers all your needs.

Cloud Security

Finally, ensuring your cloud vendor provides the security needed is an essential part of the selection process. There is a strong worry of data in the cloud not being secured. That might have been the case years ago, but today’s cloud vendors pride themselves on their ability to protect your data.

Building your next-gen data lake

After you’ve made some decisions based on these considerations, you’ll be on a great start to building your next-generation data lake in the cloud. Regardless of whether you currently have a data lake, the cloud can be a very strategic step for your organization to become a leaner big data machine and bring your data lake to the next level.

 

Building a Governed Data Lake in the Cloud                     Future-Proof your Data Lake with the Proper Architecture

About the Author

Brett Carpenter

Brett Carpenter is the Marketing Strategist for Zaloni. When he's not diving into the world of data lakes, creating engaging content, or leading community endeavors, he's either enjoying the great outdoors or exploring the food scene in the Raleigh-Durham area.

More Content by Brett Carpenter
Previous Article
Data Lake in the Cloud, Hybrid, or On-Premise?
Data Lake in the Cloud, Hybrid, or On-Premise?

Excerpt from "Architecting Data Lakes - Second Edition" by Ben Sharma. It doesn't matter where your data la...

Next Article
Don't Give Up on Your Data Lake
Don't Give Up on Your Data Lake

Turn around your underperforming data lake using a metadata-based approach to make it a successful part of ...

×

Subscribe to the latest data lake expertise!

First Name
Last Name
Company
I would like to subscribe to email updates about content and events.
Zaloni is committed to the best experience for you. Read more on our Privacy Policy.
Thank you!
Error - something went wrong!