Automate On-Prem Data Migration to Snowflake Data Warehouse without ETL tool

Hi,

This is about a strategy which I planned to automate the data flow from on-prem server to snowflake data warehouse, it can work for you if you have a similar kind of environment.

So, let's practice it with some sample data. 😊

First, prepare some data from the snowflake sample database to the on-prem system. I chose a Part table from the sample database which has 2 lakh records inside that.

Create a Part Table in your Database and copy some data into the Part table stage.

Now it has 200000 records inside.

So, we will be downloading the same data there using snowsql and will automate that from the on-prem system.

Let's prepare a Stored Proc and a task on the snowflake to write the copy into command.

snowflake stored proc and task

We are good at the Snowflake side now, let's move on-prem now.

Setup the snowsql CLI at on-prem system with your current user, prepare a batch file in visual studio code and set up a scheduler on the windows server.

snowsql put command using batch file

put command

and now setup it to the scheduler

windows scheduler at on-prem system

Please follow the video below to get a complete idea.

Happy learning 😊😊.