The goal of this series has always been to democratize knowledge about how to Efficiently operationalize data, and this involves understanding how to approach data optimization problems in an effective and meaningful way.

This point will focus on what I call the “Top Down” portion of one of the biggest players in industry.

Snowflake has rapidly become one of the top tools for data processing in the ecosystem. It is powerful, easy to use, SQL based, seems to scale up/down easily, and most importantly, mostly just works.

This won’t be an architecture post, as the Select.dev team has done an incredible job of that here. I’d strongly recommend reading that post before this one.

Rather, the goal of this post is to the clear and concrete drivers of cost in Snowflake. Then give you a few strategies that you can use to ensure that you’re getting your optimal ROI from your Snowflake investment.

Subscribe now

TLDR

“Vinoo, just tell me what to do. Don’t tell me why.”

Okay, here’s what I think you should do right now.

Disclaimer: These steps are very opinionated and work well on a majority of workloads. However, it is up to you to make sure that they work in your unique setup.

1. File a ticket on Snowflake to get access to GET_QUERY_STATS:

2. Minimize your warehouse AUTO_SUSPEND time:

   1. ALTER WAREHOUSE <warehouseName> SET AUTO_SUSPEND = 60;

3. For all multi-cluster Warehouses:

   1. ALTER WAREHOUSE <warehouseName> SET MIN_CLUSTER_COUNT = 1;

   2. ALTER WAREHOUSE <warehouseName> set SCALING_POLICY = ECONOMY;

4. Modify STATEMENT_TIMEOUT_IN_SECONDS to be a value lower than the default of 2 days.

   1. ALTER WAREHOUSE <warehouseName> set STATEMENT_TIMEOUT_IN_SECONDS=36000

Now, let me tell you why.

Snowflake + Driving

If you’ve spoken to me about this space in the past, you’ve likely heard me the Snowflake ecosystem as a busy city.

You as a query author would like to get from Point A to Point B in the most gas-efficient way possible, but you do have a few limitations:

1. Your car’s overall gas-efficiency

   1. Hummers tend to be less cost effective than say a Hyundai Elantra

2. Your skill as a driver

   1. Slamming the brakes at stop lights or hitting the gas as soon as the light turns green eats up a lot of gas

3. How busy the roads are

   1. Sitting in traffic eats up gas

4. The route you take

   1. Taking a non-optimal route results in burning more gas than necessary

In Snowflake land, you have roughly the same set of challenges. Your ability to write a query to retrieve results in an optimal way have to do with:

1. Your choice of warehouse for the query you’re running

   1. Big warehouses are like Hummers, they consume a lot of gas and may not be the optimal choice unless you need them.

2. Your skill as a query author

   1. There are folks that know every bit of the optimal SQL query authorship model in and out.

3. How saturated your Warehouse is

   1. If there are a number of other queries running on your warehouse, it can slow down your query’s time to results

4. How you construct your set of queries to yield results

   1. This has to do with how “easy” your data is to query - schema, layout, partitioning, etc… and how well your query takes advantage of them.

Top down vs. Bottom Up

It is clear that there are two areas to optimize looking at the world this way.

1. The “Environment”

2. The “Driver”

Meaning, there are things that we can do to optimize the usage of the car as well as the environment the car is driving in, without affecting the driver in any way.

I’m going to call optimizing the Environment the “Top Down” approach, since it involves us looking at the top elements (the core infrastructure) of the system. I’m going to call optimizing the Driver the “Bottom Up” approach, given that it involves the operations of the system.

Picking the ideal car for the job can be complicated, so let’s try and focus on optimizing our current car for now.

Specifically:

1. Let’s ensure we have the data we need to debug.

2. Let’s ensure the car is on for a minimal amount of time, when it is not needed.

3. Let’s ensure that our engine cylinder count is optimal.

4. Let’s ensure that trips that are unduly long are stopped at some point.

Subscribe now

Insight #1: Get Data

Solving problems requires data. While most of this post is about the Top Down portion, equally important is setting ourselves up for the subsequent bottom up portion.

There are two functions in particular you will want access to.

1. GET_QUERY_STATS - https://docs.snowflake.com/en/LIMITEDACCESS/get_query_stats.html

2. GET_QUERY_OPERATOR_STATS

Insight: File a Snowflake Ticket to get access to GET_QUERY_STATS. GET_QUERY_OPERATOR_STATS is a preview feature enabled on all accounts

Insight #2: Turn Your Car Off

A Warehouse is a logical grouping of cloud servers. That’s really it. What makes warehouses complicated though, is that they are the driver of Snowflake cost.

Think about a warehouse like a car. When the car is on, regardless of whether it is doing anything, you’re burning gas. As such, it’s a good idea to turn your car on, only when you need to drive somewhere.

However, there is one important detail. People generally don’t forget to turn their car off, but they do forget to turn their warehouses off.

Luckily, Snowflake has built an autosuspend time into the warehouses.

The autosuspend time is the time before your idle warehouse automatically suspends.

Insight: Your car (warehouse) should only be on when you need it. Minimize the time that it is on and doing nothing.

At any given moment, a warehouse is in one of the following states:

Let’s break this down:

1. Suspended - The warehouse is off. And, of course, not charging you anything

2. Running - The warehouse is on and you’re being charged.

   1. Idle - The warehouse is on, and no queries are running. You’re still being charged.

      1. Will Suspend - You have no queries running and you are within your AUTO_SUSPEND time. This is normally a bad state that we want to get rid of.

      2. Won’t Suspend - You are within your AUTO_SUSPEND time but a query will come in before the auto suspend time is hit (because of orchestration or something else). Removing this state generally involves some investigative work or changes to query orchestration schedule.

   2. Active - The warehouse is on, and queries are running. This is good.

What does this look like in practice?

Let’s say you have a consumption history that looks like this.

Let’s break this down a bit. Here’s what really happening.

This is the simplest case and matches the above. Let’s pick a slightly more complicated case.

In this situation, there is a period of time (the pink “No Queries Running”) when the warehouse is up/active, but with no queries running. In this period of time, the warehouse is, however, unable to suspend because the this time is less than the warehouse autosuspend time.

To optimize your Snowflake efficiency, your goal is to minimize the Red “No Queries Running”, while looking for ways to also minimize the Pink “No Queries Running.”

The minimum auto-suspend that Snowflake supports is 30 seconds. However, Snowflake will charge you the full minute. For that reason, it’s best to keep auto_suspend at 60 seconds.

ALTER WAREHOUSE <warehouseName> SET AUTO_SUSPEND = 60;

Subscribe now

Insight #3: Car Engines + Cylinders

Car engines come in a variety of options: 2 cylinders, 4 cylinders, 6 cylinders, and 8+ cylinders for some trucks.

The more cylinders you have, the more power the car has, and the more gas that is consumed. A V6 (6 cylinder) car will have more power than a V4 (3 cylinder) car, but the latter will generally have better fuel economy.

Snowflake has a somewhat similar concept - called clusters. Clusters though, are effectively full copies of your warehouse. In fact, Snowflake bills at # Warehouse * # Clusters. In Snowflake land, the most clusters you have the most you can do at a time under one warehouse.

Now, Snowflake offers one pretty powerful feature, the ability to scale up the number of clusters in a warehouse. Initiatively, you can think of this as only adding an additional cylinder when you need it.

In the Efficiently case, we want to maximize our cost efficiency, meaning we only want resources when we need them and want to scale only when necessary.

First, let’s make sure we always start with only one cluster. No point in starting with more resources than we need.

ALTER WAREHOUSE <warehouseName> SET MIN_CLUSTER_COUNT = 1;

Next, let’s make sure to only scale up when we really need to. You can read more about scaling policies here.

ALTER WAREHOUSE <warehouseName> set SCALING_POLICY = ECONOMY;

Insight #4: Restrict Trip Distance

Our last top down insight focuses on the “something is clearly wrong” case. The Federal Motor Carrier Safety Administration has a clear rule about drivers who are driving too long.

Thinking about driving for 10 hours straight without a break is a fairly large ordeal for many drivers, so why let our queries operate this way?

Luckily, Snowflake has a session called STATEMENT_TIMEOUT_IN_SECONDS. This specifies the maximum time that a query is allowed to run before it dies. Meaning, the longest amount of time our driver is allowed to drive.

Strangely though, Snowflake’s default value for this parameter is 172,800 seconds… which is 2 days. These poor drivers!

Let’s give them (and your wallet) a break.

This value can be configured on a per-warehouse basis.

I prefer something more reasonable - let’s just use the same 10 hour (36,000 seconds) driving limit that passenger-carrying vehicles are permitted.

ALTER WAREHOUSE <warehouseName> set STATEMENT_TIMEOUT_IN_SECONDS=36000

Subscribe now

Conclusion

There is a prevalent idea floating around that Snowflake is expensive. That can be true, but as is the case in most of these systems, it really comes down to how effectively and Efficiently you use Snowflake.

Thoughts, ideas, hate my car analogy? Leave a comment and let me know.

1. We’ll read a CSV dataset into Spark

2. We’ll write the dataset into 5 Parquet files (treating each file as a block)

   1. We’re going to have 1 row group per block for simplicity’s sake.

3. We’ll introspect some of the metadata that exists on the files.

4. We’ll run queries that show the power of predicate pushdown.

You can follow along using the commands below.

Hands-On: Setup

I’m first going to pick a random example dataset.

$ wget https://raw.githubusercontent.com/curran/data/gh-pages/vegaExamples/airports.csv -O dataset.csv

$ head dataset.csv
iata,name,city,state,country,latitude,longitude
00M,Thigpen,Bay Springs,MS,USA,31.95376472,-89.23450472
00R,Livingston Municipal,Livingston,TX,USA,30.68586111,-95.01792778
00V,Meadow Lake,Colorado Springs,CO,USA,38.94574889,-104.5698933
01G,Perry-Warsaw,Perry,NY,USA,42.74134667,-78.05208056
01J,Hilliard Airpark,Hilliard,FL,USA,30.6880125,-81.90594389
01M,Tishomingo County,Belmont,MS,USA,34.49166667,-88.20111111
02A,Gragg-Wade,Clanton,AL,USA,32.85048667,-86.61145333
02C,Capitol,Brookfield,WI,USA,43.08751,-88.17786917
02G,Columbiana County,East Liverpool,OH,USA,40.67331278,-80.64140639

Next, I’m going to open my spark shell and read this data in, inferring it’s schema.

$ ./spark-shell
scala> val dataset = spark.read.option("header","true").option("inferSchema","true").csv("dataset.csv")
dataset: org.apache.spark.sql.DataFrame = [iata: string, name: string ... 5 more field

And let’s check out the inferred schema. We can see that spark has inferred the data type of the latitude and longitude column as doubles.

scala> dataset.printSchema
root
 |-- iata: string (nullable = true)
 |-- name: string (nullable = true)
 |-- city: string (nullable = true)
 |-- state: string (nullable = true)
 |-- country: string (nullable = true)
 |-- latitude: double (nullable = true)
 |-- longitude: double (nullable = true)

For purposes of this example, I’m going to force our hand a bit and write this dataset into 5 Parquet files. By writing the files in Parquet, extra metadata will be added to each file that that gives the readers information about the contents of the file.

scala> dataset.repartition(5).write.parquet("/root/parquet_dataset")

Let’s see how these files look on disk.

$ ls /root/parquet_dataset
part-00000-2c88417e-fb93-4b83-8912-0027ade7804c-c000.snappy.parquet
part-00001-2c88417e-fb93-4b83-8912-0027ade7804c-c000.snappy.parquet
part-00002-2c88417e-fb93-4b83-8912-0027ade7804c-c000.snappy.parquet
part-00003-2c88417e-fb93-4b83-8912-0027ade7804c-c000.snappy.parquet
part-00004-2c88417e-fb93-4b83-8912-0027ade7804c-c000.snappy.parquet
_SUCCESS

Let’s ignore the _SUCCESS file for now given that it’s mostly just a flag that our write job completed successfully (and yes, it’s very possible that write jobs can fail at times).

Unfortunately, we can use something like vim or emacs to open these files, given that they are in the Parquet file format.

To aid in our investigative process, we’re going to install 2 libraries that will help us introspect Parquet data.

pip3 install parquet-tools
pip3 install parquet-metadata

Now, let’s look at one of the files. Note: the names of your part files, if you are following along at home, may be different from the names of mine above.

$ parquet-tools show part-00000-53b27d15-b049-41db-a8aa-fa3033763836-c000.snappy.parquet

You should see the data contained in this file nicely printed out.

At this point, we have 5 parquet files (5 Blocks) each with a distinct subset of our initial file.

Hands-On: Query Plans

Let’s query this data and see how the magic of partitioning can help us.

Let’s first read our CSV dataset:

scala> val dataset = spark.read.option("header","true").option("inferSchema","true").csv("dataset.csv")

And run a simple filter operation.

scala> val simpleFilter = dataset.filter($"latitude" > 30)

You can see the results with the following command:

scala> simpleFilter.show()

You can see that all of our values for the latitude column are > 30.

Now, let’s dig a little deeper and see what’s going on behind the scenes.

res3: org.apache.spark.sql.execution.QueryExecution =
== Parsed Logical Plan ==
'Filter ('latitude > 30)
+- Relation[iata#16,name#17,city#18,state#19,country#20,latitude#21,longitude#22] csv

== Analyzed Logical Plan ==
iata: string, name: string, city: string, state: string, country: string, latitude: double, longitude: double
Filter (latitude#21 > cast(30 as double))
+- Relation[iata#16,name#17,city#18,state#19,country#20,latitude#21,longitude#22] csv

== Optimized Logical Plan ==
Filter (isnotnull(latitude#21) AND (latitude#21 > 30.0))
+- Relation[iata#16,name#17,city#18,state#19,country#20,latitude#21,longitude#22] csv

== Physical Plan ==
*(1) Filter (isnotnull(latitude#21) AND (latitude#21 > 30.0))
+- FileScan csv [iata#16,name#17,city#18,state#19,country...

This seems like a lot, so let me make it easier.

The parsed logical plan has our filter condition. So does the analyzed logical plan, but it has pushed the condition down a bit, and has cast our predicate value as a double (which makes sense, since schema inference has deemed that age is a double). Finally, the optimized logical plan has added some null checking—which also matches our predicate.

  == Parsed Logical Plan ==
  'Filter ('latitude > 30)
  ...

  == Analyzed Logical Plan ==
  ...
  Filter (latitude#21 > cast(30 as double))
  ...

  == Optimized Logical Plan ==
  Filter (isnotnull(latitude#21) AND (latitude#21 > 30.0))
  ..

Let's make our query a bit more complicated and look for a set of values in a range on the same column.

scala> val complexFilter = dataset.filter($"latitude" > 30).filter($"latitude" < 40)

Once again, let’s look at the query plan.

res4: org.apache.spark.sql.execution.QueryExecution =
== Parsed Logical Plan ==
'Filter ('latitude < 40)
+- Filter (latitude#21 > cast(30 as double))
   +- Relation[iata#16,name#17,city#18,state#19,country#20,latitude#21,longitude#22] csv

== Analyzed Logical Plan ==
iata: string, name: string, city: string, state: string, country: string, latitude: double, longitude: double
Filter (latitude#21 < cast(40 as double))
+- Filter (latitude#21 > cast(30 as double))
   +- Relation[iata#16,name#17,city#18,state#19,country#20,latitude#21,longitude#22] csv

== Optimized Logical Plan ==
Filter ((isnotnull(latitude#21) AND (latitude#21 > 30.0)) AND (latitude#21 < 40.0))
+- Relation[iata#16,name#17,city#18,state#19,country#20,latitude#21,longitude#22] csv

== Physical Plan...

Let’s simplify it again. As you can see below, the plan has combined both of our predicates into one step as part of the query process (meaning that what would previously take two passes over the data now only requires one). Meaning, we’re looking for values where latitude is > 30 AND < 40 during the same pass of the data.

  == Parsed Logical Plan ==
  'Filter ('latitude < 40)
  +- Filter (latitude#21 > cast(30 as double))
  ...

  == Analyzed Logical Plan ==
  ...
  Filter (latitude#21 < cast(40 as double))
  +- Filter (latitude#21 > cast(30 as double))
  ...

  == Optimized Logical Plan ==
  Filter ((isnotnull(latitude#21) AND (latitude#21 > 30.0)) AND (latitude#21 < 40.0))
  ...

Let’s exit out of our spark shell and play a bit more with Parquet.

Hands-On: Querying with Parquet

Remember, we have 5 Parquet files. Let’s inspect one of them.

$ parquet-metadata /root/parquet_dataset/part-00000-53b27d15-b049-41db-a8aa-fa3033763836-c000.snappy.parquet

You will get an output that looks like the following.

file    created_by      parquet-mr version 1.10.1 (build a89df8f9932b6ef6633d06069e50c9b7970bebd1)
file    columns 7
file    row_groups      1
file    rows    675
...
row_group       0       latitude        type    DOUBLE
row_group       0       latitude        num_values      675
row_group       0       latitude        compression     SNAPPY
row_group       0       latitude        encodings       BIT_PACKED,PLAIN,RLE
row_group       0       latitude        compressed_size 5476
row_group       0       latitude        uncompressed_size       5471
row_group       0       latitude        stats:min       14.1743075
row_group       0       latitude        stats:max       70.46727611
...

The two things I'd like you to focus on for now are the stats:min and stats:max attributes.

These attributes contain the minimum and maximum values inside each specified column of the specified row_group. That is, the smallest value in the columns of that group of rows is equal to stats:min and the largest value is equal to stats:max.

This information is a huge performance win! If we are running a query that falls outside of this range, this entire row group can be excluded.

Let’s say we have another file with the following metadata.

file    created_by      parquet-mr version 1.10.1 (build a89df8f9932b6ef6633d06069e50c9b7970bebd1)
file    columns 7
file    row_groups      1
file    rows    675
...
row_group       0       latitude        type    DOUBLE
row_group       0       latitude        num_values      674
row_group       0       latitude        compression     SNAPPY
row_group       0       latitude        encodings       BIT_PACKED,PLAIN,RLE
row_group       0       latitude        compressed_size 5176
row_group       0       latitude        uncompressed_size       5771
row_group       0       latitude        stats:min       44.4430157
row_group       0       latitude        stats:max       74.46727611
...

In our original query (where latitude is > 30 and < 40) we are able to exclude this whole file.

scala> val complexFilter = dataset.filter($"latitude" > 30).filter($"latitude" < 40)

In practice, this is called predicate pushdown. The requirements of the predicate (the query) have been pushed down, allowing the optimizers to look at the metadata on the row groups themselves to decide which row groups to read, and when they can be ignored.

Conclusion

There is a lot of magic that goes into our ability to query data quickly and Efficiently. In future posts, we’ll cover some of these in more detail as well.

In that post, I wrote a bit about how the layout of data on disk can impact the performance of your analytics jobs. This post will be focused on how that tactically works using open source technologies.

In this post, I’ll talk a bit about:

1. HDFS

2. Blocks + Block Size

3. Block sizes + tradeoffs

Background

Let’s take a step back and review what we have learned. We know that the way that data is laid out on disk can drastically affect the performance of analytics. We also know that there are a few strategies for laying data out on disk (we explored row-oriented, columnar, and hybrid models in the last post), but how is this actually relevant to us in our modern data stack?

Hadoop

To really understand what I’m about to describe, it’ll help to have some history. I’m going to start with Hadoop. If you’re reading this post, you’ve likely heard of Hadoop, but in the event you haven’t, I’m going to describe Hadoop in the Efficiently way.

Hadoop is an ecosystem of products that allow you to:

1. Store data in a distributed way (HDFS - Hadoop File System)

2. Query that data (MapReduce)

3. Manage compute resources for those queries (YARN)

4. [There is also a Commons Library and an Object store called Ozone].

For our purposes, we’re choosing to focus on HDFS, since it’s the most directly relevant piece of the partitioning ecosystem from the storage side.

HDFS

Let’s say you have an invaluable file that contains the names, addresses, and phone numbers of employees at your company.

You find this file very valuable and you want to access it from many places. Therefore, you make the logical decision to put this file somewhere that makes it easy to access. Now these are the days before Google Drive/Dropbox (kind of), so you had to build things your own way. You decide to upload this file on a server.

One day, you come into work and realize that the server is down! Meaning, you can’t access your file anymore. That's not good.

So, you decide that you can store 2 copies of that file on two different servers to avoid this situation from happening.

But wait, now you need to update the file… which file should be updated?

How do you keep the two in sync? Furthermore, let’s say you increase the number of people accessing/updating this file. The orchestration around all of this becomes extraordinarily difficult.

And now we have a problem.

Luckily, there are a bunch of mechanisms in HDFS that allow you to work through these challenges (which I’ll likely cover in subsequent posts).

But for now, let’s get back to partitioning.

Blocks

HDFS stores data in blocks. These are the same Blocks from the last post. They are indivisible segments of data. A block is also, just like in the last point, the minimum amount of data that Hadoop can read to do a single read. Meaning, any read requires me to read at minimum the whole block.

In order to store data into HDFS, Hadoop takes the files and divides them into blocks.

It then takes these blocks and distributes them across multiple nodes in its cluster.

… Technically it does some replication too, so it really looks more like this in practice (depending on how many replicas you set). In the case of the below image, we’re assuming two replicas.

Note: Those of you experienced with HDFS will notice that my architecture diagram is intentionally simple. No Name Node / Data Nodes separation, no clients, etc.. The purpose of this guide is a description of partitioning, not a deep dive into HDFS.

Once that process is complete, your file is officially stored on HDFS. This process seems pretty straightforward and simple, but these blocks play a critical role in the performance considerations associated with reads.

Subscribe now

Block Size

So far, we’ve glossed over a somewhat important concept. We know that files are divided into blocks… but exactly how many blocks? Are all of the blocks the same size? How big should a block be?

Let’s start with some details. The default block size in Hadoop is 128MB (it used to be 64MB). When I first heard this number, I remember thinking, “wow. that’s massive.” And at the time, it felt like it was. 128MB feels like a huge amount of data, especially when you’re working with tiny local CSVs.

That is, until data scale grew and we started seeing folks storing files that are 100s of GBs into HDFS.

The block size is configurable on a per-client basis.

Tradeoffs

Selection of the Block size affects the number of blocks that a file is “chunked” into and, as such, the number of I/O operations required to read or write the file.

Intuitively, a larger block size means the file is divided into fewer blocks, and thus requires fewer IO operations to read/write, but increases the amount of memory required to store each block in memory during processing.

Conversely, a smaller block size means the file is divided into more blocks. This means it requires more IO operations to read/write, but consumes less memory to store each block in memory during processing. This can be beneficial when working with small files (but not too small) or when performing random access on a large file. However, a smaller block size means more blocks are created, which increases the metadata overhead on the NameNode and can become a scalability bottleneck.

As an aside, there is a well-documented problem associated with having too many small files, called the small files problem. I don’t think I can do a better job of explaining why this is problematic than this article does: https://blog.cloudera.com/the-small-files-problem/

Block size also impacts HDFS’ fault tolerance, but that’s outside the scope of this article.

Tuning for Efficiency

As you’ve seen, reducing IO through partition pruning is a powerful way to query your data Efficiently. As such, selecting an optimal block size yields strong results.

Unfortunately, there isn’t a one-size-fits-all solution. Selecting a block size depends on the dataset, usage patterns, type of data, use case, and many other factors.

At a high level though, here’s what I’d recommend.

1. Files less than a few hundred MBs: a smaller block size, such as 64MB or even 128MB can minimize the amount of wasted space in a block.

2. Files that are several GB or more, a larger block size, such as 256MB or 512MB can help minimize the number of blocks generated by the “chunking” of a dataset and ideally, minimize unnecessary IO.

Disclaimer: These are just guidelines and selecting your optimal block size will likely require some testing on your side.

Share Efficiently

Feedback

Thanks for reading! I’d love any feedback. Feel free to comment below.

A few years ago, I gave a talk at Spark Summit 2020 about File Formats about Avro, ORC, and Parquet. I got a bunch of questions about that topic, which I promptly responded to point-to-point, ensuring that the knowledge lives only in those forums… and nowhere else.

That isn’t helpful for most people. This post is my attempt to fix that.

In this series, I’ll lay out some of the primitives of this topic and then dive into the hands on details.

Problem

In the efficiency space - minimizing “work” is key. Whether work requires compute, network, or storage, the goal of efficient data usage is to get the most accurate answer in the fastest and cheapest way possible.

One of the ways the ecosystem has developed to help data practitioners accomplish this task is though the development of File Formats. When you think of a file format, you may think of an extension that looks like .xlsx, .pdf, .pptx… and you’d be right.

Each of these formats tells the “reader” of the format, how to interpret the data that has been written to disk. Similarly, technologies like Parquet, Avro, and ORC help practitioners store their data in a way that minimizes work.

Background / Example Data

The word partition is derived from the Latin Let’s get to the actionable details.

A partition is a logical segment of data. In the big data world, this usually means a piece of a lager dataset. For our purposes, and to illustrate this, I’m going to use the example dataset below.

This dataset has 3 columns (Column A, Column B, and Column C) and 4 rows (Row 0, Row 1, Row 2, and Row 3).

This table should look just like something you’ve seen on Excel, in Pandas, etc.. Let’s take this example a bit further and split up the individual elements into their own logical “pieces.”

We can refer to each “cell” by it’s “<column><row>.” For example, the second row in Column B is called B1.

Storage

Let’s talk a bit about storage. In the data ecosystem, storage is thought of as cheap (at least cheaper than compute), and as such, few people look into ways of actually optimizing storage.

Storage - in this post - refers to the way that this data is actually stored on disk. Now, whatever represents “disk” may differ from system to system (for example, even though the concept of a disk isn’t exposed to you on S3, there data eventually ends up there).

For this example, and to learn more about partitioning however, I’m going to focus on the most simple storage mechanism - data that is stored on your local hard disk. From there, most everything else can be extrapolated.

Background

Data is stored on hard disks in what is called a block. A block is the minimum amount of data read during any read operation.

I’ve always thought about blocks as a suitcase. When you go on a trip, and have to check in a bag, you pay the same price regardless of how full or empty your suitcase is. That being said, it’s optimal to fill your suitcase with as many objects relevant to your trip as possible, in as easy of a way to find as possible.

Extending this analogy further, packing a bunch of unnecessary stuff in your suitcase isn’t great. Additionally, bringing a ton of suitcases on your trip (unless everything is strictly necessary), also isn’t great. Finally, inside of the suitcase, you want to “group” similar things together - ie. each of a pair of socks should likely be next to each other in the same suitcase, rather than one sock being in one suitcase and one sock being in the other.

In the land of hard drives, all of these insights apply. Reading unnecessary data is expensive. Reading fragmented data is expensive. Random seeks (sock example above) are expensive as well.

Our goal is to lay data out in a manner optimized for our workflows.

Row-wise Storage

In database land, the common way to storage data used to be row-wise. It’s pretty easy to understand why. Most people think about datasets as a list of rows.

Taking our dataset above, let’s store this in a row-wise method.

As you can see, I have taken each row in order and packed as much of the rows as I can into a block, before moving to the next block.

This method works great when my goal is to read the data sequentially. All that’s required is a simple linear scan of the block in order. It doesn’t work as well if, for example, I want to only look at Column C. In that case, I’m required to read all of the block (ie. read all of the data) and filter down to Column C.

This is row-wise storage methodology.

Columnar (Column-wise) Storage

Column-wise storage takes the opposite approach and orients around columns.

As you can see, we first take the entire column, pack it into a block, and then move onto the next column.

This method works great when the data is read as in a columnar way (ie one column at a time). It doesn’t work well if, if example, I want to reconstruct Row 0. In that situation, I’d need to read all of the data and filter down to the elements that make up Row 0.

Now, we’re in a dilemma - one approach seem too hot favor a row oriented of workflow, one approach seems to cold favor a column oriented workflow. Luckily for us (and Goldilocks), there’s a middle ground.

Hybrid Storage

A hybrid storage model gives us the best of both worlds. First, we group the a fixed number of Rows together and then the further group that by columns. We segment these and call these “Row Groups” (at least in the Parquet terminology)

In this example, we first selected two rows - Row 0 and Row 1. We then, grouped those rows by column, and inserted them into our first Row Group.

I called these logical Row Groups because this is more of how we should be thinking about them, rather than how they may necessarily end up on disk.

This representation of data is actually immensely powerful. It allows us to both optimize our workflows for row oriented workflows as well as column oriented workflows.

Let’s talk about how this works.

In the case of a row oriented workflow, let’s say I’d want to recreate Row 2. To do this, I would simply need to look at Block 1 and Block 2. If I were operating in a Columnar storage model, I would need to look at Block 1, Block 2, and Block 3. I’ve saved a whole Block!

In the case of a column oriented workflow, let’s say I’d want to recreate Column B. In this case, I would simply need to look at Block 1 and Block 2. If I were operating in a Row-wise storage model, I would need to look at Block 1, Block 2, and Block 3. I’ve once again saved a whole Block!

Our examples used very small data, you can imagine how this extrapolates further with larger data sets.

Data Workflows

Throughout this post, I’d referred to my data workflows as "row oriented” or “column oriented.” Luckily for us, the big data community has come up with some terminology that should help bring these two workflows to life.

OLTP

Online Transaction Processing (OLTP) workloads generally involve larger amounts of short queries/transactions. These tend to be more focused on processing than analytics and as such have more data updates and deleted. Roughly - we can consider OLTP workflows as “row oriented” workflows.

OLAP

Online Analytical Processing (OLAP) workloads are more analysis than processing focused. As such, there tends to be more analytical complexity per query and fewer CRUD transactions. Roughly - we can consider OLAP workflows as “column oriented” workflows.

Conclusion

Using data efficiently, relies on using all levels of the “data stack” (storage, network, compute) efficiently. Reducing the amount of unnecessary data read during a query process can have compounding effects on the speed and efficiency of your analytics process.

Coming soon

In subsequent parts of this series, I’ll be digging more into the details of how everything we have covered thus far can be applied in analytics workloads.

Subscribe now

My name is Vinoo and I’ll be your guide, writer, and (likely) ranter throughout this series.

0. What this is

Recently, I had a friend recommend Austin Kelon’s Show Your Work. I read it over the course of a short plane ride and realized… it’s time to start showing my work. Publicly.

This blog series, at it’s core, is a collection of the the things that I’ve learned while building analytical tools, creating data products, and consuming data products. Mostly, it’s intended as a technical series for technical audiences, but I’m not sure where it’ll actually go.

But, at it’s core, this blog is me challenging myself to extract meaningful and applicable lessons from the challenges I’ve faced working in the data ecosystem for over a decade.

In short, this blog is a public forcing function for me to democratize the best practices I’ve learned along the way.

1. Why this, why now

We grew up in the golden age of data. The cloud migration was well underway. Organizations transformed and decisions became data driven. The modern data stack was cool, powerful, and new. The potential for compute was endless. Money was cheap and plentiful. It was worse to do nothing than spend 7 figures on building a data platform.

Then, things got real. The economic downturn hit… and it hit hard. Layoffs started. Money got tight. Many startups had to look up the word “profit” for the first time (joking, but not really).

In these hard times though, companies tend to focus on cutting only what they can “see.” Specifically, they cut office space, employee travel, and employee expense budgets. They cut marketing budgets, third-party contractors, and of course, sadly, employees.

All the while, their compute bills are increasing by an order of magnitude year over year.

The compounding cost of clouds is an existential threat for enterprises. Without appropriate capital to fund business operations and without the means of assessing the current cloud/data investments, organizations are left in a precarious position. Data practitioners need to know about this methodology because it can be the very element that saves their company from at best an unexpected bill and at worst a cession of business.

2. Who may benefit from this

YC just sent this to it’s founders.

A lot of scary things going on in this post, but the high level is that founders need to $ave Dat Money. This series is for you if your enterprise would like to $ave Dat Money.

3. Why me

I’ve gotten it wrong… a lot. And as such, have had to learn a lot.

I have always been fascinated by the power of data to drive business decisions, inform research, and improve our understanding of the world around us. My career has taken me from the tech sector to the financial sector. I’ve built analytical tools, sold data as a broker, consumed data from brokers, and most recently, help folks optimize their internal tech stacks.

That's where this blog comes in. My goal is to explore the ways in which we can use data more efficiently, both in terms of the resources we use to collect and process it, and in terms of how we use it to drive meaningful results.

4. Where I’ll start

I’m likely going to start with some of the areas I’ve worked personally with and tell (more than) a few stories along the way. I'll be covering topics ranging from data storage and management to data analysis and visualization, and sharing case studies and best practices from my own experiences and those of others in the industry.

I'm thrilled to have you join me on this journey, and I hope you'll find value in the content we'll be sharing.