Snowflake basics
In the context of my work, Snowflake is often considered the "data warehouse".
When I hear "data warehouse" I think of a massive building with huge ceilings, forklifts moving here and there, people wearing hard hats, pallets and boxes... Kinda like an Amazon warehouse, I suppose. Except instead of cheap plastic stuff that ends up in landfills, a data warehouse stores bits of data.
Where does the data come from?
Many places....
- Logs from a webserver that show user's activity as they are browsing your website
- Names and email addresses of people who signed up for your newsletter
- Times and directions of tiny movements that a robot makes
- The mathematical representation of pixels from one million frames of a video
Computers do things and emit data in the form of logs, JSON, Parquet, and other file formats. This is considered data. This data is useful for some entity downstream.
How does data make it into the warehouse?
Through ingestion tools. It could be as simple as cp. Or it could be a robust and high-scale ingestion tool like Fivetran. It could be driven by an Airflow DAG that orchestrates hourly ingestions from an S3 bucket to Snowflake.