Automate On-Prem Data Migration to Snowflake Data Warehouse without ETL tool

Vidit tyagi
2 min readFeb 19, 2022

Hi,

This is about a strategy which I planned to automate the data flow from on-prem server to snowflake data warehouse, it can work for you if you have a similar kind of environment.

on-prem data migration on snowflake using windows scheduler

So, let's practice it with some sample data. ๐Ÿ˜Š

First, prepare some data from the snowflake sample database to the on-prem system. I chose a Part table from the sample database which has 2 lakh records inside that.

Create a Part Table in your Database and copy some data into the Part table stage.

create a table in snowflake and insert 2lakh records inside that

Now it has 200000 records inside.

snow table stage list command

So, we will be downloading the same data there using snowsql and will automate that from the on-prem system.

Let's prepare a Stored Proc and a task on the snowflake to write the copy into command.

snowflake stored proc and task
snowflake stored proc and task

We are good at the Snowflake side now, let's move on-prem now.

Setup the snowsql CLI at on-prem system with your current user, prepare a batch file in visual studio code and set up a scheduler on the windows server.

snowsql command file with batch script
snowsql put command using batch file
put command file
put command

and now setup it to the scheduler

windows scheduler for snowflake data warehouse
windows scheduler at on-prem system

Please follow the video below to get a complete idea.

Happy learning ๐Ÿ˜Š๐Ÿ˜Š.

--

--