Connecting to Snowflake
Snowflake is cloud native enterprise data warehouse. They publish native python drivers, the snowflake-connector-python , or if you prefer to use SQLAlchemy, snowflake-sqlalchemy . This article goes through using both packages, as well as building them into your own image.
Snowflake Connector with Pandas
After having spinning up a JupyterLab instance and a Dask cluster connecting to your Snowflake data warehouse using snowflake-connector-python is simple.
import snowflake.connector import pandas as pd ctx = snowflake.connector.connect( user='YOUR_USER', password='YOUR_PASSWORD', account='YOUR_ACCOUNT' ) query = "SELECT * FROM SNOWFLAKE_SAMPLE_DATA.TPCH_SF1.CUSTOMER" pd.read_sql(query, ctx)
Snowflake also has an optimized routine for loading data into a pandas dataframe
cursor = ctx.cursor() cursor.execute(query) df = cursor.fetch_pandas_all()
If you’re used to working with SqlAlchemy you can use it with snowflake as well
from sqlalchemy import create_engine from snowflake.sqlalchemy import URL from sqlalchemy import sql from dask.dataframe.io.sql import read_sql_table engine = create_engine(URL( user='YOUR_USER', password='YOUR_PASSWORD', account='YOUR_ACCOUNT', schema ='TPCH_SF1', database = 'SNOWFLAKE_SAMPLE_DATA' )) engine_url = engine.url query = "SELECT * FROM SNOWFLAKE_SAMPLE_DATA.TPCH_SF1.CUSTOMER" pd.read_sql(query, engine)
The images that come with Saturn come with both Snowflake packages installed. If you are building your own images and want to work with Snowflake, you should install
snowflake-connector-python as well as