This is about a strategy which I planned to automate the data flow from on-prem server to snowflake data warehouse, it can work for you if you have a similar kind of environment.
So, let's practice it with some sample data. 😊
First, prepare some data from the snowflake sample database to the on-prem system. I chose a Part table from the sample database which has 2 lakh records inside that.
Create a Part Table in your Database and copy some data into the Part table stage.
Now it has 200000 records inside.
So, we will be downloading the same data there using snowsql and will automate that from the on-prem system.
Let's prepare a Stored Proc and a task on the snowflake to write the copy into command.
We are good at the Snowflake side now, let's move on-prem now.
Setup the snowsql CLI at on-prem system with your current user, prepare a batch file in visual studio code and set up a scheduler on the windows server.
and now setup it to the scheduler
Please follow the video below to get a complete idea.
Happy learning 😊😊.