Internal Engineering Talk

Profiling & Fixing
Scala 3 Compile Times

A field guide from our own monorepo: three compiler flags, six steps, and the mistakes that taught us to read the output correctly.

zsh — monorepo/
$ ./mill _.compile -Vprofile
Why this talk

Our builds spend most of their time in two phases

51.5s
total typer time across all 7 modules
13.8s
total inlining time across all 7 modules
43–86%
typer alone, as a share of a single module's compile time

Before we could fix anything, we had to learn to actually read what the compiler was telling us — that's what this talk hands you.

Agenda

Six steps, in the order you should actually run them

  • 01The three flags — cheapest to most detailed
  • 02Reading a single profiler line
  • 03The compiler phase cheat sheet
  • 04The trap: the typer dump can lie to you about macros
  • 05Prove a fix functionally, before you trust the timing
  • 06Establish the noise floor before believing any single number
The toolkit

Three flags. Each answers a different question.

-Vprofile

"How much time, per compiler phase, per module?"

Cheapest. Always start here.

-Vprofile-sorted-
by:complexity

"Which files, roughly, produce more than their source size suggests?"

A narrowing filter, not a verdict.

-Xprint:
typer,inlining

"What did the compiler actually generate for this file?"

Expensive, precise, and easy to misread.

Step 01 — Start Here

-Vprofile — aggregate phase timings

zsh — monorepo/
$ ./mill _.compile -Vprofile 422] typer,run ns = 12772435042,idle ns = 0,cpu ns = 12528639000, user ns = 12406649000,allocated = 5652060672,heap at end = 881729120, total allocated = 5711984016

One line like this, per compiler phase, per module. Works identically under mill, sbt, or bare scalac.

Step 01 — continued

Reading a profiler line, field by field

annotated
422] typer,run ns = 12772435042, idle ns = 0,cpu ns = 12528639000, user ns = 12406649000, allocated = 5652060672, heap at end = 881729120, total allocated = 5711984016
  • 422] — mill's task id for this module
  • typer — the compiler phase name
  • run ns — wall-clock time. Divide by 1e9 for seconds
  • allocated — bytes allocated during this phase only
  • total allocated — running cumulative across all phases so far — not phase-specific
Step 01 — continued

The compiler phase cheat sheet

PhaseWhat it doesWhen it's expensive
parserSource text → raw ASTRarely a bottleneck
typerType-checks everything: given/implicit search, type inference, overload resolution, compile-time macro branchesUsually cost #1
posttyperCleanup after typecheckingRarely a bottleneck
picklerSerializes the typed tree to TASTyScales with code size
inliningExpands regular inline def bodies — where a macro's chosen implementation actually gets substituted inUsually cost #2, esp. circe/Magnolia
erasureStrips generics, preps for bytecodeRarely dominant
genBCodeEmits JVM bytecodeScales with code size
wartremoverThird-party lint pluginPure overhead on every local build
Step 01 — payoff

Sum it, rank it, and you have your priority list

The recipe

Sum run ns per module per phase. Rank modules by typer + inlining time. That ranking is your priority list — no guessing required.

What we found

producer-api — 16.6s typer, largest single module
common-lib — 12.8s typer, 5.9s inlining, most files (672)
worker-service — 9.5s typer — 86% of its own compile time

Step 02 — Narrow the list

-Vprofile-sorted-by:complexity

zsh — monorepo/
$ ./mill common-lib.compile \ -Vprofile -Vprofile-sorted-by:complexity -Vprofile-details:20

sorted-by:complexity ranks files by a lines/tokens/TASTy-size ratio — a rough proxy for "this file produces more generated code than its source size suggests."

-Vprofile-details:N goes one level deeper: the N most complex methods, not just files.

The one that took several iterations to get right
Step 03 — The real evidence

-Xprint:typer,inlining

zsh — monorepo/
$ ./mill common-lib.compile -Xprint:typer,inlining [[syntax trees at end of typer]] // .../File.scala [[syntax trees at end of inlining]] // .../File.scala

Prints the full syntax tree of every file, twice — once per phase. This is where you see exactly what the compiler generated.

The trap

Two dumps. Two different truths.

typer dump shows

The macro's call site, plus the Mirror.SumOf[T] argument being constructed — a type-level listing of every case.

Its size scales with case count no matter what the macro's implementation does. Unavoidable, fast or slow.

inlining dump shows

What the macro's body actually expanded to — this is where you can tell a cheap implementation from an expensive one.

This is the dump that reflects your fix.

How we found out the hard way

Four attempts, one wrong assumption

  • 1Wrote a fast path. Diffed the typer dump. 0 of 164 call sites changed. Looked like total failure.
  • 2Rewrote it two more times, same measurement, same "0 changed" result every time.
  • 3Realized: the typer dump's size is dominated by the Mirror argument — it was never going to change, regardless of the fix.
  • 4Re-ran with -Xprint:typer,inlining and diffed the inlining dump instead.
Once we measured the right phase

The fix was real all along

2,737 105
lines generated for ValidationError (24 cases)
2,945 116
lines generated for SyncTaskType (20 cases)

✓ Confirmed identically in both the typer and inlining dumps — proof the new method is no longer inline at all.

Takeaway

The rule

If the method you're optimizing is inline, you must diff the inlining phase dump — not typer — to see if your fix took effect.

If you rewrite it as a plain def, typer and inlining will show the same (small) size — an instant, built-in confirmation that the fix is active.

Step 04 — Before you trust timing at all

Prove it functionally first

SimpleEnum.scala
summonFrom { case given (T <:< scala.reflect.Enum) => println(s"fast path: $label") fromRealEnum(m, label) case _ => println(s"slow path: $label") deriveSimple { ... } }

Compile just the smallest affected module — seconds, not minutes. Confirm in the output which branch actually fired for a known type. Remove the prints once confirmed.

Why: a silent no-op branch is easy to write — it happened to us three times before we landed on a version that actually worked.

Step 05 — Before you trust any single number

Establish the noise floor

+2.8% to -9.3%
observed deltas between builds of what should have been closely related code

Before trusting a before/after number:

  1. Run the same, unchanged code 2–3 times. Record the natural variance.
  2. Only trust a delta that clearly exceeds that noise band.
  3. For a confident final answer, average 3 runs on each side.
Step 06 — Turning data into priorities

Prioritize using the ratio, not just the size

PatternWhat it meansWhat to do
inlining >> typerMacro/derivation cost dominatesHand-write or reflect instead of deriving
inlining ≈ typerThe file's own code is just largeSplit the file — a structural fix, not a macro fix
High per-file allocation, few filesPossible wide-implicit-scope importNeeds a sampling profiler to confirm — see next slide
Limits of this method

Some costs never show up in a printed tree

One of our modules showed ~984MB of typer allocation per file — 10–100× higher than modules with far more files. The AST dumps showed nothing unusual.

The likely cause: implicit search exploring and discarding candidates from wide wildcard imports — work that's genuinely transient. It allocates heavily during typechecking but leaves no trace in the final tree, by definition.

Next tool: a sampling profiler (async-profiler / JFR) attached to the compiler process, with stack samples grouped by package.

Recap

The one-page cheat sheet

cheatsheet.sh
# 1. cheapest — always start here $ ./mill _.compile -Vprofile # 2. narrow down to suspect files $ ./mill <module>.compile -Vprofile -Vprofile-sorted-by:complexity -Vprofile-details:20 # 3. see exactly what got generated (inline → diff the INLINING dump) $ ./mill <module>.compile -Xprint:typer,inlining # golden rules inline def → diff inlining, not typer plain def → typer size == inlining size confirms the fix always → prove functionally (println) before trusting timing always → measure the noise floor before trusting a delta
Takeaways

Four things to carry out of this room

  • 1Start cheap (-Vprofile), escalate only once you have a real suspect.
  • 2The typer dump can lie about inline macros — diff inlining, not typer.
  • 3Prove a fix functionally (a println) before trusting a single timing number.
  • 4Measure the noise floor first — a single before/after build is not evidence.
Thank you

Questions?

zsh — monorepo/
$ ./mill _.compile -Vprofile # try it on your module