I ran ducklens over 69 million real Snowflake queries

I built a tool, ducklens, that reads a Snowflake account's query history and works out how much of the bill could run on one machine with DuckDB instead of a warehouse. The awkward thing about a tool like that is that it can produce whatever number flatters the pitch. So before pointing it at a real account, I wanted to run it over a large pile of queries I had no hand in generating and see what came out.

There is one good public source for this, and it is oddly underused. In 2018 Snowflake's engineering team exported telemetry for about 70 million queries from across their entire customer base over two weeks and released it as Snowset, alongside an NSDI paper. The paper gets cited. The trace itself mostly sits on a server, which is a shame, because it carries a column I have not seen in any other public query log: for each query, the number of bytes it spilled to disk when it ran out of memory.

That column is what makes the question answerable at all, so let me start there.

Why spill is the number

The obvious way to guess whether a query needs a cluster is to look at how much data it reads. It scans two terabytes, so surely it needs distributed compute. For an out-of-core engine that instinct is wrong. DuckDB does not pull a scan into memory; it streams through it and holds only the working set, the intermediate state an operator needs at one time. A query that scans two terabytes to produce a filtered sum keeps a few kilobytes of running totals in memory and is perfectly happy on a laptop. What actually strains a single machine is usually a much smaller query whose working set blows up, like a group-by on a high-cardinality key or a join that fans rows out. How much memory that needs has almost nothing to do with bytes scanned, which is why sizing off scan volume mostly misleads.

Spill is the engine measuring the working set for you. When an operator's state grows past the memory it is allowed, it writes the overflow to disk, and Snowflake recorded how many bytes went. That is a direct reading of the one quantity that decides single-machine fit, taken on real production queries under real concurrency, and it is not something you can reconstruct from an ordinary query log. Snowflake ran the experiment across their whole customer base and wrote down the answers.

What one machine actually holds

This is where single-node arguments usually go off the rails. They assume a query fits only if its working set fits in RAM. DuckDB does not work that way. When memory fills, it spills to local disk and keeps going, so the real ceiling is the machine's RAM plus its local disk.

Snowset splits spill into two columns: bytes that went to the node's local SSD, and bytes that spilled past the SSD to remote storage. A 2018 Snowflake node had a few hundred gigabytes of local SSD, so anything beyond that shows up as remote spill and looks fatal. A single box you can rent in 2026 has a different shape. An AWS i4i.8xlarge is 256GB of RAM and 7.5TB of local NVMe for around $1,200 a month reserved, and one node goes up to 30TB; bare metal is cheaper again. A query that spilled 800GB on a 2018 node, and so spilled remotely there, lands entirely on local NVMe on that box and never touches the network.

So the per-query test is whether the spill is bigger than a single machine's disk. Against 7.5TB, very little clears the bar.

The whole trace

Mapping Snowset into the schema ducklens reads is one SQL query. I ran the scorer over all of it. On the full trace, 68.7 million scored queries across roughly 2,050 warehouses, 71% of the query compute lands in single-machine range on a 256GB / 7.5TB box. By query count it is 80%, and the gap between the two is the expected one: the queries that do not fit are the expensive ones.

I owe you a detour here. I first scored a few million queries from the front of the trace and got 77%, and almost shipped that. It was wrong in the ordinary way samples go wrong. Snowset's files are not shuffled, and the spill-heavy and serving-heavy warehouses cluster later, so the front oversampled the easy customers. Across everyone, weighted by compute, it is 71%. If you refuse to let DuckDB spill at all and only count what stays in RAM, it is 60%. There is a spread, and I would rather hand you both ends of it than the flattering one.

What stays, and why it is concurrency

The residual is the interesting part. A thin slice of it is spill so large it overruns even 7.5TB of NVMe, and that is legitimate, some estates need distributed scale and the tool should say so. But most of what stays is one thing, high-concurrency serving, and it is worth being exact about why a bigger disk does nothing for it.

DuckDB is a single process with one shared thread pool and one memory budget, tuned to aim every core at one query. That is the opposite of what serving needs, where a warehouse is fielding dozens of small dashboard and API queries at once, each wanting its slice. You can push one instance some distance. The only controlled multi-stream benchmark I could find, from a competitor with no reason to be generous, has DuckDB surviving sixteen concurrent heavy streams with a tolerable slowdown. Past that you are arguing with the architecture. The clearest evidence is what the company selling DuckDB does about concurrency: MotherDuck gives each user their own instance rather than making one instance hold more. When the vendor scales out to serve concurrency, I am not going to claim one box swallows it.

This is the softest part of the estimate, and the place I nearly fudged. The concurrency threshold, how many sustained overlapping queries mark a workload as serving, is a knob. I moved it from sixteen to thirty-two and watched the headline jump to about 90%. It was the easiest twenty points in the whole project, and I dropped them, because a single DuckDB box does not hold thirty-two concurrent heavy queries, and shipping 90% would have meant quietly redefining the one thing DuckDB is worst at. The serving warehouses stay on Snowflake.

What this does not tell you

It is 2018 data. Workloads have grown since, which pushes fit down, and the drift toward serving and streaming pushes it down more, so read 71% as a measurement of a real population at a fixed point in time and not a forecast. There are no dollars in the trace either, so this is a fit result; the dollar figures ducklens prints on Snowset come from a rough size model, and the real cost only appears when you run it against an account's own metering. And it sees resource behavior rather than SQL, so it will miss a Snowflake-specific function or a governance surface that blocks a migration the shape analysis would pass. Clearing the single-machine bar is only the first gate.

The last caveat is the real price of the whole approach. Spilling to disk finishes the query, but slower than keeping it in memory would. This is a statement about what runs on one machine, and for batch and analytics work that trade is usually fine; for anything latency-sensitive it is not, and none of this claims a spilling query matches a warehouse's wall-clock.

Trying it

ducklens is on GitHub, MIT licensed, and the scorer is a single SQL file, so none of this needs taking on faith. Pull the Snowset parquet from its repository, run the one mapping query, and audit it. The --local-disk-gb flag sets the box's NVMe, so you can score the exact machine you would buy and watch the number track it. If you would rather not download anything, ducklens demo builds a synthetic account and scores it end to end. And pointing it at your own query history is the only version of the number that is actually about your bill.