Compiler
Status: Implemented for a bounded subset of the Plan; the rest of this design is a working hypothesis.
The Compiler turns one analyzed Project graph into a deterministic application artifact. It starts from the graph generation that a valid AnalysisRun accepted for the current Head. It captures that graph once, read-only, and qualifies it into one immutable input. Renderers then return complete files without SQL or filesystem writes. The parent composes those files with the pinned Rails Core, checks path ownership, and packages one manifest with the submitted Plan and its reviewed GapSet. Meaning the target cannot lower is omitted and listed in that GapSet. The Compiler does not repair, complete, or reinterpret the Plan. This page owns capture, qualification, rendering, composition, packaging, dispatch, and the Compilation Record. Each subject's Rails lowering has its own target page. The support inventory (repository-only) records what the Rails target generates.
On this page
- Graph capture
- Pipeline
- Rendering boundary
- Output and packaging
- Rendering isolation
- Omissions and the GapSet
- Dispatch and the local lifecycle
- Program boundary
- Compilation Record
- Reproducibility
- Why compile early
- Code and checks
- Alternatives and open questions
- Decisions
Graph capture
Compilation begins only for the current Head and Project graph version accepted by a compatible target-specific AnalysisRun. Compile start deterministically serializes and validates the admitted graph, copies the submitted source bytes stored by the Head, stores both immutable representations and their SHAs, binds the frozen GapSet, and sets the Project's active Compilation pointer in one transaction.
The implemented PlanSerializer (repository-only) proves the
serialization substep for an empty Project or the current bounded application boundary. It consumes the sealed capture
through loaded associations only, sorts authored collections explicitly, emits only normalized Foundation Plan facts,
and reloads its exact UTF-8 bytes through the strict sketch/0.23 loader before exposing the bytes, SHA-256, and
loaded document. Strict loading plus independent query observation prove the service path issues no SQL. It does not
activate the process-global renderer SQL guard inside a threaded web or worker process. The snapshot serializes the
emitted Compilation boundary plus dependencies needed to keep retained meaning structurally loadable.
Otherwise-omitted enum and conditional-collision Fields, References, and Associations remain in the snapshot when
retained Account, Scaffold, query, Validation, Policy, state, or descriptor meaning refers to them. This includes
through/source Association and Predicate dependencies. Retaining a row does not admit its storage or behavior; target
limitations remain in the reviewed GapSet. Other unsupported Fields are omitted. The snapshot also omits some Field
modifiers, including normalization pipelines; the Head retains the complete authored form. Import, capture, and
serialization checks prove locator coherence and a byte fixpoint for the resulting subset, not preservation of
dependency selection or identical regenerated output from that lossy snapshot. Rails-target naming or table admission
failures and directly authored graph states the importer cannot produce (for example mutation inputs, bindings, or a
return destination attached to an index definition) still fail serialization loudly rather than disappearing; that
last Start-time edge is unreachable through import and is a candidate for the #406 ledger.
Within that otherwise admitted graph, the serializer preserves Entity-, Field-, and Reference-owned Validation
rows, the dependency-closed retained Predicate subset, and every retained Ordering root and term in deterministic
owner-local order. A Predicate is pruned only when its dependency is absent from the serialized subset; dependent
Predicate roots that use matches_predicate are then pruned transitively. Target-unlowerable operators over retained
dependencies remain serialized. An admitted immutable Reference retains immutable: true. A predicated direct
Association retains its
exact current Predicate locator from the same qualified CompilationInput relationship catalog used for rendering.
Exact byte fixpoints cover representative Validation, Predicate, and Ordering snapshots. This proves snapshot bytes
rather than semantic validity or generated Rails behavior; target admission remains a separate stage.
The implemented local lifecycle starts a dedicated process that reads the same logically locked graph. The
production Render adapter starts the same entry point in a One-Off. The implemented
CaptureProjectGraph (repository-only)
verifies the expected graph_version and captures every current semantic Project-owned row in one
REPEATABLE READ transaction. It loads every declared Active Record association one hop from those records, then
rewrites every association target to one canonical Ruby instance per database row. This finite operation closes
cycles and through associations without recursively preloading paths. Ordinary repeated association traversal is
therefore query-free after the transaction closes.
The capture marks every returned record strict-loading and read-only, recursively freezes every attribute value and
loaded collection-target array, then freezes the record. Association readers remain usable. The qualified compiler
input separately retains an immutable canonical-graph snapshot so use-time matching rejects association-target
replacement as well as attribute drift. Rendering still runs under SqlGuard, and renderers may not call
relation-producing methods such as where, count, or reload. The Head record, AnalysisRun rows,
Compilation rows, and internal subject-identity registry are explicitly excluded from renderer traversal. Dispatch
and other host-operational rows remain outside the captured Project graph.
The association lists are a code-owned capture contract rooted at Project, not another data model. A reflection-census test requires each new Project or Foundation Plan association to be classified explicitly. A strict-loading failure on a declared traversal is a capture defect; a renderer query or excluded traversal is a renderer defect. Sequential renderers will receive those same rich Active Record objects. A later process executor may fork after capture so children inherit copy-on-write views without parsing the Plan snapshot or materializing a second graph.
The capture uses the Rails 8.1.3.1 Active Record Preloader only as a batching primitive, then canonicalizes its results itself. Its query count is fixed for a given graph composition rather than its row count; the focused test pins an exact representative count as a Rails-upgrade and per-row-regression tripwire.
Capture preserves each association's loaded order; it does not invent one universal semantic order. A renderer must preserve a model-declared order when that order carries authored meaning and must apply an explicit stable order at the point of use for an otherwise unordered collection. PostgreSQL return order is never an output contract.
A separate App Schema argument would add another synchronization edge without giving the Compiler more information.
Pipeline
- Verify the current graph version, accepted AnalysisRun, target profile, and compilation snapshot.
- Completely preload one coherent Project Active Record graph.
- Enumerate deterministic tasks whose renderers can inspect that complete graph.
- Render one complete file value per task without touching the filesystem.
- Combine renderer output with the pinned Foundation Rails Core, reserve parent- and tool-owned paths, and validate complete path ownership.
- Materialize the resolved source files once in a clean workspace, run only declared isolated tool steps, and collect their declared outputs.
- Write the parent-owned provenance files once in their reserved paths, then compute the final manifest and package the local artifact.
- On every terminal path, persist the ordered diagnostics and Compilation outcome. Store the exact artifact on success, and clear the active Project pointer only while that Compilation still has authority.
A separately authorized publication may consume the retained artifact only after the Compilation succeeds. That later lifecycle owns repository provisioning, Git ref mutation, and remote recovery in a durable Publication rather than extending the Compilation through a publishing phase. If a remote mutation has an unknown outcome, recovery performs only read-only reconciliation; it does not automatically reissue the mutation. The durable orchestration is implemented and locally tested with fake remotes.
The parent-owned provenance files in step 7 are the exact Foundation Plan snapshot and a pre-publication projection of the Compilation Record. The projection may include fixed inputs, coverage, and ordered Compiler diagnostics. It may also include completed generation, tool, and verification outcomes. It cannot include the repository commit, manifest digest, final publication status, or other values that would make the artifact describe its own identity recursively. The final manifest digest belongs on the durable Compilation. Repository commit identity and final publication status belong on a separate durable Publication. Parent-owned files obey the same path and collision rules as renderer and Core files, and their paths are reserved before any materialization begins.
Compilation should not choose between materially different valid outcomes. It may derive table names, indexes implied by uniqueness, fixed template content, and other details whose alternative would not change the user's meaning or material consequences.
Rendering boundary
Each renderer task owns one final relative path and returns one complete value:
class GeneratedFile < Data.define(:path, :contents, :mode, :owner, :source_subject_uuids)
end
A model renderer can enumerate one task per Entity, while routes or application-layout renderers enumerate one task for the whole Project. Every task can inspect the complete captured graph, so a Reference may consistently affect models, migrations, controllers, forms, displays, sample data, and tests without renderers communicating or patching one another's output.
Focused Rails-target query objects may calculate naming, cardinality, or other mechanical consequences shared by
several renderers. They are derived values, not another authored graph or an emission-complete AnalysisRun API.
The implemented Association-method input builder projects every local Association key and Reference realization
from either one six-query snapshot or the already sealed Compiler Project graph, without later SQL. It wraps the
target-independent Association graph rather than adding Rails realization facts to that semantic graph.
Compiler context supplies the selected target, target profile, Compiler release, Core release, and other fixed
inputs that affect output. Generated-file provenance uses Project-scoped subject UUIDs rather than internal row IDs.
The Application object has no authored subject UUID, so the five identity-only Core replacements deliberately carry
an empty subject-provenance set; their Project generation and exact Foundation Plan snapshot remain
Compilation-level provenance. The schema replacement instead carries every Entity, Field, and Reference UUID that
contributes to its Domain tables and constraints. The application-global ApplicationController replacement
carries empty subject provenance for an ordinary application and the Account Entity UUID when a realized Account
adds its request-identity interface. The conditional ErrorsController replacement carries that same Account Entity
UUID for its public-navigation helper.
RenderingContext always carries one verified Core package. Normal Compilation also carries its exact qualified
CompilationInput::Result. That result is fixed evidence from the same captured Project generation rather than a
second copy of Project state. Direct renderer tests may omit it only when the captured Project has no authored
Validations; direct rendering cannot silently omit a Validation graph.
Qualified input
CompilationInput performs that same-generation qualification once per AnalysisRun or Compilation. It retains
exact Project, graph-version, target, and profile provenance; one frozen ordered Domain-candidate catalog; one
same-generation Relationship catalog; exact Predicate, Ordering, Policy, Policy-operation-census, and Scaffold
projections; the collision-qualified legacy Web projections; optional Account lowering, registration projection,
and Account self result; the exact ApplicationAppearance result; browser ApplicationNavigation; a separate
ios_navigation; the downstream selected-or-declined iOS result; and one deterministically ordered diagnostic set.
Construction recomputes browser navigation from the admitted Web Scaffold and Account inputs. Ordinary input
recomputes iOS navigation from the public-only Scaffold subset without Account self or profile; only private native
qualification shares its Account-bearing navigation with iOS. Every projection must carry the same Project
generation and, where source evidence matters, the exact submitted-document digest and pointers.
At use time, matching rederives the complete ordinary, Account-qualified, or Policy-qualified result, including the ordinary input-owned Account lowering and any private Policy-gate evidence. The capture deep-seals every compared attribute value and snapshots every canonical association edge. Cross-capture correspondence compares the explicit Project semantic attributes and the complete captured association closure one-to-one while excluding service-ownership and lifecycle-only Project state. Reconstructing an input or a private rendering context therefore cannot remove unsupported-client diagnostics or coherently alter Web and native output.
Task planning
The model, create-table migration, post-table foreign-key migration, schema, and Web paths read the qualified catalogs
through RenderingContext, so task discovery and rendering share the same admitted values without repeating
whole-graph selection. Direct contexts remain strict. The registry discovers every named direct renderer class, while
its default selection excludes renderer families that require an explicit release gate. The default set includes the
application-global ApplicationController and conditional ErrorsController renderers. The latter replaces Core's
error controller only for a realized Account, supplying a current_account helper that returns nil without
authentication or database access. The application task plan combines that set with the released Web families, the
conditional Appearance family, the two application-global dependency renderers, the two complete Account families, one
Account-self family, and five Policy-gate renderers. The dependency pair is always selected together even though each
task owns one final path. Merely loading an experimental renderer does not enroll it in application output. iOS
rendering uses a separate explicitly selected task family because its context and Core package differ from the Rails
phase.
The complete Account families do not introduce renderer dependencies. Their Ruby, route, template, and mailer references resolve when the generated application boots, while the migration filename controls Rails migration order. Renderer execution order cannot change their bytes or runtime ordering, so a task DAG would add no semantic information. The aggregate Account emitters remain only as differential test oracles; production composition uses the selected one-path tasks.
The active ApplicationManifest coordinator owns that same-generation composition seam. It builds one immutable
ApplicationTaskPlan from a sealed Project, its exact valid CompilationInput::Result, a RenderingContext carrying
that same result object, and a caller-supplied lowercase SHA-256 for the complete Plan snapshot. Planning constructs
and retains the exact dependency plan and CorePathRegistry. It preflights the complete selected scalar, Web,
Account, Appearance, Policy, and dependency task census, exact duplicate final paths, required Core replacement
owners, and any selected task that claims an immutable Core path. The same checks include the selected
ApplicationController, conditional ErrorsController, Gemfile, and Gemfile.lock tasks. One executor then
renders every Rails file from that qualified generation without repeating dependency selection or projection.
CoreComposer reuses the planned registry and composes Rails Core exactly once. The Plan digest passes unchanged to
an immutable iOS rendering context instead of being derived from the smaller native projection. That context binds the
exact CompilationInput::Result, its exact successful iOS projection object, and the Plan snapshot SHA-256. A
declined iOS result selects no tasks and returns the Rails manifest without loading iOS Core. A selected result
retains one pinned iPhone Core and preflights its complete seam census and replacement modes. It renders exactly seven
one-path tasks through SequentialExecutor and passes that same Core object to final composition. Task planning also
requires the exact expected paths and owners. IosCoreComposer repeats seam, owner, and mode validation as
defense-in-depth before both trees enter one final OutputManifest.
The normal executor reads the one qualified Domain catalog while discovering tasks and rendering each Domain file. It does not repeat whole-graph Domain selection inside individual renderers.
Rails lowering by subject
Each subject's Rails lowering has its own target page, indexed by the Rails target profile: generated names, Domain storage, Fields, References and Associations, scopes and ordering, data integrity, authentication, authorization, Scaffolds, state transitions, the application shell, UI and Appearance, and native clients. Rails Core composition covers the Core pin, path ownership, dependency projection, and application documents. Generated Rails tests covers the emitted RSpec suite and its ownership.
Output and packaging
Each renderer returns GeneratedFile values; the parent validates, composes, and packages them.
contents is an immutable byte string. The generic file value preserves arbitrary bytes, including binary assets,
while text renderers remain responsible for producing valid UTF-8. The manifest hashes the exact bytes. mode is
the regular-file mode 0644 or 0755. The initial Compiler does not generate symlinks.
The parent collects every GeneratedFile before it writes anything. It rejects invalid paths and duplicate path
claims, then combines generated values with one pinned Foundation Rails Core. Core paths are already owned by Core.
A small explicit map may transfer intentional full-file replacements such as Gemfile or
config/routes.rb to one named renderer; every other collision fails. A selected, generated Home index also omits
the fixed welcome controller, view, and Home-only locale paths. The complete Core archive digest pins its contents,
so a second per-file digest
registry is unnecessary for Core in general. Dependency projection is a narrow compatibility exception: the maximal
universe records and verifies the exact Core Gemfile and Gemfile.lock digests because every replacement pair is
projected from those inputs. The application-wide dependency plan adds the State Machine contributor only when
behavior admission succeeds; storage-only machines do not add AASM.
The implemented output kernel validates an intentionally narrow portable path grammar, canonicalizes file bytes and provenance UUID sets, and builds a path-sorted manifest with exact SHA-256 digests. It rejects exact, case-folded, file/directory-prefix, and inconsistently cased directory claims before materialization. The Core composition layer loads the bundled archive as values and applies the same policy to Core plus renderer output without writing either input to disk.
The implemented
ArtifactEnvelope (repository-only) turns that complete
manifest into exact UTF-8 JSON bytes suitable for a dependency-free client. Its top-level keys are format,
provenance, manifest_sha256, and files. Files retain manifest order and contain path, exact-byte SHA-256,
numeric mode (420 for 0644 or 493 for 0755), owner, ordered subject UUIDs, and strict Base64 contents. The
manifest digest is SHA-256 over compact JSON with one files key and the ordered metadata array. Each metadata
object orders path, sha256, mode, owner, and source_subject_uuids; it uses decimal integers, no
insignificant whitespace, and unescaped solidus characters. The artifact digest covers the complete envelope bytes
and remains outside the body, avoiding recursive identity. The current in-database prototype boundary caps an
envelope at 128 MiB. That ceiling is deliberately generous relative to the 1 MiB submitted-Plan boundary because
the envelope contains Base64-encoded generated files, the exact submitted source, and the reviewed GapSet.
The implemented decoder admits retained envelope bytes only when their external artifact digest, manifest digest,
and file count agree; the JSON is the exact canonical encoding above; and the provenance, ordered metadata,
per-file digests, modes, paths, and strict Base64 contents reconstruct one valid OutputManifest. Its returned
value has no public construction or copy path, so a later transport cannot substitute files after decoding and
still satisfy the boundary by type alone. Decoding does not attest the provenance against a Compilation row or
authorize publication; those checks belong to the persisted publication lifecycle.
The envelope's provenance.core object identifies the root Rails Core. When the artifact includes the separately
pinned iPhone Core, ios/FOUNDATION_PROVENANCE.json identifies that mounted source. The manifest metadata and
artifact digest cover those exact provenance bytes; clients that need the nested pin must inspect the downloaded
artifact rather than the root-Core metadata object.
CompileCapturedProject (repository-only) is the
pure orchestration seam. Given one sealed Project plus explicit lifecycle provenance, exact submitted source, and
the frozen GapSet bytes, it serializes the admitted graph, builds one CompilationInput, rederives target planning
as an integrity check against the GapSet, installs those values in one RenderingContext, and calls
ApplicationManifest with the snapshot digest. It returns the graph snapshot, complete Rails and optional iPhone and
Android manifest, and envelope without persistence or filesystem writes. It copies the GapSet and submitted source as
parent-owned files and cannot amend their contents. It does not claim, cancel, publish, or
otherwise implement a durable Compilation. The Compilation ID, AnalysisRun ID and release, and Head source digest
are shape-checked caller assertions; this pure seam does not attest their relationship to the Project. The
SQL-allowed lifecycle wrapper now binds those values before calling it. Because the SQL guard is process-wide, this
seam must run only in the dedicated process described below, never inside Puma or a shared job worker.
Qualification, Rails application-task planning, selected Rails task rendering, iOS task planning, and selected iOS
task rendering occupy separate sequential SqlGuard activations. Account runtime and web-flow output runs in the
same selected-task phase as the other Rails output. A query during either planning phase has no task identity. A
query or other rendering failure in either executor carries the exact one-path task identity. Core loading,
value-only composition, and packaging run between or after those guards.
SqlGuard is process-wide and
non-reentrant: wrapping CompileCapturedProject or ApplicationManifest in one broader guard would nest an
existing activation and raise SqlGuard::AlreadyActiveError.
Each concrete renderer declares one immutable lowercase ASCII identifier, from which its stable renderer:...
owner is derived. It inherits only the task-enumeration and rendering contract from Renderers::Base; shared
formatting and Rails-target logic remain modules or query objects rather than deeper renderer inheritance.
RendererTask retains the named direct renderer class, a copied stable task key, and one prevalidated final path.
Its execution boundary requires exactly one GeneratedFile with that declared path and owner.
The implemented stateless registry asks Zeitwerk to eager-load the renderer namespace on every discovery, rejects
live out-of-namespace, indirect, and duplicate-identifier renderer classes, and sorts direct subclasses by binary
class name. Anonymous, removed, and replaced class objects that are no longer bound to constants are not registered
renderers and are ignored, so discovery does not depend on garbage-collection timing. A renderer may mark itself for
explicit application selection; discovery still reports and validates it, but the default task set omits it. The
registry likewise validates an explicitly selected renderer set and each renderer's task Array, rejects duplicate
classes, foreign tasks, and duplicate renderer/key identities, and returns a deterministic frozen task sequence.
It does not cache classes or tasks. ApplicationTaskPlan rejects exact duplicate final paths and selected claims on
immutable Core paths before rendering. IosApplicationTaskPlan separately rejects any selected task set other than
the complete seven expected iOS Core replacements and preflights the retained package's seams and modes.
Both task sets are independent, so their current one-path tasks have no ordering edges and need no task DAG.
OutputManifest retains the complete final collision policy for every composition boundary.
The implemented Materializer accepts one complete OutputManifest and one caller-owned empty directory. It
creates every directory implied by the manifest's file paths and every file itself, opens files exclusively without
following leaf symbolic links, writes binary contents, applies exact 0755 directory modes and each file's declared
0644 or 0755 mode, and then verifies the complete path, digest, and mode set. The manifest carries files, not
empty directory entries, so empty directories are neither representable nor materialized. The Materializer refuses
missing, non-directory, symbolic-link, and nonempty destinations instead of clearing or merging them. A failed write
may leave a partial tree; the future coordinator owns a private disposable workspace and discards that workspace on
failure.
Rendering isolation
The implemented first executor is sequential. It asks the registry for tasks and renders them in the registry's
order, then constructs one OutputManifest only after every task returns. A later process executor may fork only
after Rails constants and the complete graph are loaded. Renderers perform no SQL, graph mutation, filesystem
writes, or reads of another renderer's output. Connection disconnection before fork remains hygiene.
The sequential executor installs one process-wide evented sql.active_record subscriber around both task
enumeration and rendering, then removes it before ordinary Compilation persistence resumes. Raising from the
subscriber's start callback prevents synchronous reads and writes before their adapter block executes: Rails
starts the notification before yielding, and Active Record executes adapter work inside
that yield. Cached, schema, and transaction events receive no exemption. The subscriber also
rejects buffered publish_event delivery, but Rails can publish that event after asynchronous SQL
finishes. The guard therefore refuses to arm unless ActiveRecord.async_query_executor is
nil, as it is in the current application. Nested or concurrent guards in one process fail instead of
misattributing process-global events. Renderers that create threads must join them before returning. A future fork
executor installs a fresh guard in each child after forking.
This process-wide enforcement is valid only in the implemented dedicated bin/compile process, a production Render
One-Off running that entry point, or a dedicated forked child. It must not run inside Puma, the Solid Queue
supervising dispatcher, or any process with unrelated
database-using threads: their queries would correctly trip the guard and be prevented. Keeping the subscription
process-wide is intentional because a renderer must not evade the rule by starting a joined thread.
The guard records the active task identity, issuing thread name, and call-site backtrace without retaining SQL, binds, the connection, or the notification payload. It latches the first violation for its end-of-phase check but raises each later event with that event's own facts. A renderer therefore cannot hide a query by rescuing the immediate exception: the deferred violation still becomes a task-specific failure.
The executor converts a task-rendering StandardError into an immutable RendererFailure containing the stable
failure category, task identity, exception class, valid UTF-8 message, and original backtrace. Core Exception
accessors keep hostile overrides from defeating capture. A valid overridden #message wins; core Exception#to_s
and the core class name are the ordered fallbacks. Core Object and Module accessors likewise preserve the real
exception class name despite hostile class or name overrides. The custom Marshal path reconstructs deeply
frozen strings and retains no task, renderer class, Project, context, or original exception. RendererFailed is
the local parent-side exception and deliberately has no original exception cause. Registry discovery errors occur
before any task identity exists and remain distinct. Process-control exceptions are not converted. A later process
executor sends only GeneratedFile or RendererFailure values from child to parent.
RendererTask intentionally retains its live renderer class. Tasks are enumerated before a process executor forks
and are inherited by its children; they are not a portable queue or spawn payload. GeneratedFile values and plain
renderer-failure details are the process return boundary. Supporting a spawn- or queue-based executor would require
a separate class-resolution transport contract rather than accidental marshalling of a live Class.
Any task-rendering StandardError fails the complete rendering phase and the current Compilation with the stable
failure category compiler.renderer_failed. Process-control exceptions terminate without being relabeled. The
parent persists no partial artifact. Exception details remain operational; the public diagnostic does not mistake
a Compiler defect for invalid Plan input or a declared support gap.
The zero-SQL tests prove both that successful rendering returns the expected manifest and that an independent observer saw no query event. They separately exercise a real prevented write, a cached read, direct and buffered notification delivery, swallowed and joined-thread violations, call-site retention, async-configuration rejection, task discovery, and ordinary and swallowed task-rendering violations. A renderer that raised before querying is not evidence of a successful query-free render.
An artifact path is 1–1,024 ASCII bytes separated only by /. Each 1–255-byte component matches
[A-Za-z0-9._][A-Za-z0-9._-]*, so a hyphen cannot lead a component. Dot segments, trailing dots,
case-insensitive .git, and Windows device stems such as CON or LPT1.txt are invalid. These deliberately
conservative rules cover every current Core path and make collisions independent of the host filesystem.
Every final artifact path is declared before its owning tool runs. After each tool step, the parent rejects any created, modified, or deleted path outside that declaration. Tool caches, logs, and scratch files are directed outside the artifact root or removed before the check; they are not silently omitted from the manifest. The final manifest covers every path in the artifact.
This bounded admission is not complete target validation. Application keys too long for generated database or
service identifiers, unresolved Render service naming or uniqueness constraints, and identifier limits for future
References, indexes, and constraints remain open. The current application coordinator owns one RenderingContext
and composes against that exact context's Core package; accepting an independent Core input would permit a
mixed-revision manifest. Production dispatches the whole bounded pipeline through bin/compile in a Render One-Off.
A general isolated renderer-process executor, general tool-step runner, durable dispatch attempts, and automatic
launch reconciliation remain future work.
Omissions and the GapSet
Ignore-and-list is the implemented candidate default (the owner revision of 2026-08-21
on #405 (repository-only) and
#413 (repository-only)). Malformed bytes and schema violations block before
import. Semantic contradictions in admitted meaning and broken supported references still prevent a valid AnalysisRun.
A target limitation does not block when the affected subject and its dependents can be omitted or conventionally
partially generated. Import prunes each service gap at its existing per-path diagnostic and returns the skips as
warnings. Analysis then combines those skips with target-planning omissions and freezes one self-contained
firstdraft.foundation-gaps/2 document for the Head SHA-256, graph generation, analyzer/Compiler release pair, and
selected target. Compilation admits the imported subset and succeeds under the ordinary status vocabulary, with no
partial-success status or gap-specific request field. Meaning that imported but has no lowering in the current release
— unsupported Account registration shapes, Policy roots outside the bounded public record lowering, unsupported
Predicates and Orderings, unsupported Validations, non-public or otherwise inadmissible Scaffold definitions, Field
shapes outside the ten scalar kinds and exact required enum subset, ignored Field modifiers, and unsupported native
clients — is omitted from the generated output and listed before Compile. Application Appearance is generated for
the Rails shell, while every app receives name-derived bookmark icons. Selected native shells honor the theme and
colors but retain stock launcher icons, so either emitted platform retains that narrowed partial gap. Every generated
application copies the canonical GapSet bytes unchanged to .firstdraft/gaps.json and retains the exact submitted
source at .firstdraft/submitted-foundation-plan.json. Each JSON Pointer therefore remains resolvable.
The support inventory (repository-only) records which probed subject kinds the
Rails target generates.
The coherent incomplete Foundation decision (repository-only) applies the same rule to generated names and every other target limitation. Never emit invalid or ambiguous source. Use a semantics-preserving projection when one exists; otherwise omit every collision participant and its dependents and list the consequence. Some existing naming analyzers still report blocking diagnostics instead. That is a known implementation exception to remove, not a target-admission category to extend.
A semantics-preserving pre-alpha projection need not be production-safe or minimal. It may retain the ordinary Rails scaffold route set, permissive CRUD, or an unguarded request path that a developer would tighten next, provided the artifact lists the unmet Plan meaning and claims only what was realized.
The Compiler is the deterministic part of the current product sketch. Foundation Plan authoring may involve judgment. Compilation should validate and execute the choices already present in one coherent Project graph.
Supported behaviors cannot be emitted as unrelated snippets. A fixture that combines features should settle that combination's semantic question, Plan choice, forced consequence, support gap, and proof.
The Compiler should return structured, location-aware diagnostics (repository-only). A contradictory definition is an error because it lacks one coherent instruction. Incomplete target realization is an ordinary gap when the Compiler can emit a conventional bootable starting point.
A recognized but unsupported subject now produces coherent output with a disclosed gap: the implemented
ignore-and-list default skips it, Analysis freezes the complete support delta, and Compilation emits the supported
subset plus that exact .firstdraft/gaps.json. The authored subject is never silently reinterpreted: the artifact's
exact submitted Plan source and reviewed GapSet disclose exactly what was skipped.
Dispatch and the local lifecycle
Compilations::Start prepares an optimistic graph capture and deterministic Plan serialization before entering a
short Project-locked transaction. Its separate persistence seam lets another Project-owned lifecycle add durable
state inside that same write transaction without nesting the capture's REPEATABLE READ snapshot. The transaction
rechecks the current Head's strong ETag, graph version, target, current valid AnalysisRun, analyzer and Compiler
releases, and absence of another active Compilation. The AnalysisRun and normal Compilation share the same
qualification, so a Project with an unsupported Validation still reaches the required current valid run: the
blocked Validation is listed in the frozen GapSet and the admitted declarations compile. A focused lifecycle test
imports a complete candidate with a Movie title length Validation, analyzes it, starts and executes the Compilation,
decodes and materializes the artifact, and finds the
emitted declaration in app/models/movie.rb. A successful start stores the canonical admitted-graph snapshot, a
byte-for-byte copy of the submitted source stored by the Head, and the reviewed AnalysisRun/GapSet identity,
sets project.active_compilation_id, runs the caller's optional persistence step, and enqueues the dispatcher
atomically.
The initial production adapter deliberately retains that single-table boundary. Attempts and remote launch
identities remain deferred until measured recovery needs justify them.
The Solid Queue dispatcher deliberately shares the primary database transaction and runs no Compiler code itself.
Development and test use the local adapter. It starts argv-only bin/compile in a process group, polls durable
cancellation, and supervises that child for its lifetime. Production instead makes one authenticated Render API
request to start bundle exec ruby bin/compile COMPILATION_ID as a Standard One-Off Job. The worker-provided
RENDER_SERVICE_ID selects the worker base service, so the One-Off receives that service's latest successful build
and environment snapshot. bin/compile removes the inherited Render API key before loading Rails because execution
does not need dispatch authority. The short dispatcher returns after dispatch instead of occupying a compilation
worker thread for the Compilation's lifetime.
A Render 4xx response or missing local configuration proves that the dispatcher did not receive an accepted job. That path safely fails only a still-queued Compilation and clears only its matching Project pointer. A timeout, transport failure, 5xx response, or unexpected response status can occur after Render accepted the request. That ambiguous path raises to the queue operator but leaves the Compilation queued and preserves its active pointer. The possible One-Off can still claim and promote through the ordinary database fences. The initial operator recovery is explicit Cancel; automatic dispatch reconciliation and remote cancellation remain future work.
The local adapter still sends TERM and bounded KILL when cancellation wins, then reaps the process. If local
supervision unwinds unexpectedly, it delegates reaping without killing a child that may already own and complete
the Compilation. A hard child or One-Off exit after claim leaves the row running because no Attempt identity can
prove which dispatcher owned it. Cleanup is not the authority mechanism.
The child first requires the retained Compiler release to equal the current CompileCapturedProject release. A
superseded release fails the still-queued Compilation before claim with
compilation.compiler_release_superseded and tells the client to start a new Compilation from the current
Foundation Plan. Other input-provenance mismatches retain compilation.capture_failed. The child then claims
queued as running under Project-then-Compilation locks, captures the exact graph generation, reserializes it,
and compares both bytes and digest with the immutable graph snapshot. It verifies the submitted-source digest and
calls CompileCapturedProject only after those checks. Promotion decodes the complete artifact and repeats the
active pointer, graph, AnalysisRun, releases, target, Head, status, provenance, exact GapSet file, and exact
submitted-source fences under the same lock order before storing artifact bytes and clearing the pointer.
Cancellation wins any race that reaches
those locks first. A cancelled snapshot mismatch exits quietly; an uncancelled mismatch becomes a safe structured
failure. Supported failure and cancellation transitions go through their Project-then-Compilation service
boundaries rather than direct record updates. When a Compilation belongs to a GitHub Publication, those services
lock that Publication next and terminalize both records atomically. The CI lifecycle smoke crosses the real
OS-process boundary and persists the complete Rails and iOS artifact.
The User-scoped HTTP slice requires an active bearer token and conditionally starts that exact lifecycle from the
current strong Head ETag. It polls metadata-only current state and immutable provenance without loading either
retained byte payload, and invokes Cancel through the same Project-first lock boundary. A succeeded artifact GET sends
the exact retained bytes with the CLI's vendor media type, byte count, strong digest ETag, and no-store, no-transform. After a linked Compilation succeeds, the
GitHub integration owner (repository-only) owns Publication coordination.
Program boundary
The authoring service, analysis workers, and Compiler will share this Rails application, release, migrations, and Postgres database. Render One-Off Jobs provide independently scalable compilation compute without requiring a separate Compiler service or duplicated graph.
Generated applications still require a separate dependency environment. Devise, Rodauth, and other emitted gems run through the generated application's bundle in a scrubbed child process. Sharing the First Draft codebase must not cause generated dependencies or environment variables to resolve through the host application's bundle.
The old compiler-rails repository is a pinned SQLite migration oracle, not a production gem dependency. Invoke
it only as an isolated child process when a porting parity test needs the old behavior.
The initial artifact publication target is a GitHub repository. The hosted flow deliberately compiles before any repository mutation. Only after the Compilation succeeds and retains its exact artifact may a separately authorized Publication provision the destination through a GitHub App and publish that artifact. Repository creation is a visible product action, not renderer or Compilation work.
The Compiler does not need to conduct product discovery, approve the Plan, host the application, merge into an owner-modified repository, or keep later changes representable in First Draft's vocabulary.
Compilation Record
Every accepted compile request should produce a Compilation Record. The Project may therefore contain several records for one accepted graph generation. A future attempt-aware lifecycle may create more than one Compilation Attempt inside that record after ambiguous dispatch or infrastructure retry. The complete record should include:
- its own UUIDv7 identity and overall status;
- Project identity and graph version, plus Foundation Plan snapshot format and content hash;
- target profile, Compiler, Core release and archive digest, and toolchain identities;
- the outcome for each Plan subject;
- structured diagnostics for every partial or omitted result;
- meaningful output identity;
- generation-step outcomes and Compiler qualification identity; and
- source and dependency provenance.
Candidate overall statuses are succeeded and failed; per the owner revision on #405 a Compilation with
capability skips is an ordinary success. Its AnalysisRun-owned GapSet distinguishes service_support_gap from
target_support_gap and uses per-entry statuses skipped_at_import, partially_generated, and not_generated.
Each self-contained entry carries a stable code, kind, status, reason, observable consequence, and source pointer
or readable path where available. A service entry for a larger pruned unit points to the complete omitted scope and
uses cause for the exact triggering path, so a nested unsupported property cannot hide omission of its whole owner.
This Compilation Record describes what happened. It does not add decisions or become another input artifact. Generated repositories include the one canonical machine-readable GapSet plus the exact submitted source it addresses. Clients may render a human view on demand; a second persisted prose projection would create another owner and drift surface. The durable Compilation adds final artifact fields that cannot be embedded in the artifact they identify. A separate Publication adds remote repository, tree, commit, and publication-lifecycle fields.
An illustrative record could look like this:
{
"id": "019f9433-b6f7-78ad-a641-870aa6ca8f2d",
"status": "succeeded",
"project": {
"id": "019f941f-43f7-7b1d-b1a7-ae052bffecad",
"graph_version": 5
},
"foundation_plan": {
"format": "<Plan format>",
"sha256": "plan-content-hash"
},
"compiler": {
"id": "compiler-rails",
"release": "<Compiler release>"
},
"target": {
"id": "rails",
"profile": "<target profile>"
},
"core": {
"release": "<Core release>",
"sha256": "core-archive-hash"
},
"coverage": [
{
"subject": {
"kind": "field",
"readable_path": "message.original_content",
"subject_uuid": "019f9425-5412-7928-91f6-32163f22cfe2"
},
"status": "generated",
"artifacts": ["app/models/message.rb", "db/migrate/..._create_messages.rb"]
},
{
"subject": {
"kind": "reference",
"readable_path": "like.target",
"subject_uuid": "019f9425-5412-7104-b765-494c61edf559"
},
"status": "not_generated",
"reason": "unsupported_by_compiler_release",
"diagnostic": "compile.reference.realization_unsupported"
}
],
"meaningful_output_sha256": "normalized-output-hash",
"artifact": {
"sha256": "artifact-envelope-hash",
"manifest_sha256": "manifest-hash",
"file_count": 198
},
"generation": {
"status": "completed"
}
}
Angle-bracketed values stand for the release identities that the machine reference (repository-only) names. This is an illustrative public projection, not a current response schema. Operational attempt and log details may remain internal. Timestamps, complete toolchain inventory, provenance entries, and coverage granularity remain to be designed. The implemented bounded GitHub Publication projection is a separate response shape. The stateless GitHub transport (repository-only) already defines exact root-commit and non-force ref-publication behavior beneath that lifecycle.
Reproducibility
For a fixed Project generation, snapshot, profile, Compiler, Core, and toolchain, repeated runs should produce the same meaningful source and behavior. Exact bytes are useful for owned templates but may be too strong when external tools emit harmless volatile metadata.
A practical Compiler release test can combine:
- normalized output hashes;
- golden-tree diffs;
- dependency and toolchain identities; and
- representative generated-application behavior checks.
The initial production compilation path should not run the emitted application's full test suite. The Compiler release earns confidence through its fixture and behavior suite before deployment; each user job records the exact qualified release and deterministic artifact identity.
Network resolution, mutable tags, hidden environment state, and inherited process configuration should not
silently alter the result. If the Compiler is a Ruby process, inherited BUNDLE_GEMFILE and RUBYOPT deserve an
explicit isolation test because the prototype experienced that class of coupling.
Why compile early
Generation is a sufficiency test for the design. A model can look internally consistent and still lack enough information to create migrations, boot the app, render a form, enforce denial paths, or explain the result to the next agent.
Golden source comparisons and behavior tests in the Compiler's release suite can expose different omissions without running a generated application's tests during every user compilation.
Code and checks
script/compiler_generated_repository_ci qualifies one complete generated repository downstream of this pipeline. It
imports the reviewed Oscar Party Plan, requires its exact checked-in GapSet, compiles and materializes the retained
artifact, and initializes its parentless main commit through the same Git artifact builder used by Publication. Its
receipt records the artifact envelope, manifest, pinned Core, submitted and compiled Plan, GapSet composite identity,
and generated commit and tree before invoking the emitted repository's complete bin/ci with CI=1. The qualifier
does not repair or format generated source. After the emitted command finishes, it requires every manifest file's
bytes and mode to remain exact and the checkout to remain clean. The command is an opt-in qualification gate for the
reconciled release identities. Qualification does not authorize merge, publication, deployment, or release.
The materialization and Policy record-gate qualification scripts require the Rails target profile's Ruby 4.0.5 and
Node.js 24.18.0, access to the locked gem and npm sources, and a disposable PostgreSQL role permitted to create and
drop databases. The compiler group's CI job installs both target runtimes and passes their installation prefixes
to both scripts; no other group's job installs them. Locally, each script resolves the target installations through
asdf unless COMPILER_TARGET_RUBY_PREFIX or COMPILER_TARGET_NODE_PREFIX is explicit. The scripts put those
concrete installations first in the child PATH. The materialization smoke additionally asserts the generated
app's running RUBY_VERSION and rejects a Node version mismatch before installing packages. Both scripts install
into fresh temporary bundle and npm cache paths rather than accepting host dependency trees as evidence.
Each generated-app smoke in the compiler group compiles its Plan once and materializes the artifact once. Repeated
output is owned by focused tests: captured-project
compilation (repository-only) compiles twice and
pins digests, manifest characterization (repository-only) pins five representative manifests, and the
Materializer test (repository-only) compares two trees. A
smoke first migrates an empty database with the emitted db/schema.rb withheld, and the Rails schema dump must equal
the emitted bytes. It then loads that schema into a fresh database. Development seeds, browser specs, eager-loaded
boot checks, and the emitted RSpec suite run once, against that schema-loaded database. Running them on the migrated
database would not test it: the first RSpec or rails/test_help process finds no schema_sha1 in
ar_internal_metadata, so maintain_test_schema! reloads the schema first. The materialization smoke still checks
both databases, because each one has a different physical column order.
The evidence index tracks the Rails Compiler row.
Alternatives and open questions
Each alternative below names its disposition and revisit trigger.
- Compile inside the web request or Solid Queue worker process — declined. Development and test supervise a dedicated local process for the bounded lifecycle. Production dispatches a Standard Render One-Off to isolate expensive generation and generated-app commands while reusing the application release and database.
- Require a separate HTTP service or Lambda — superseded for the first experiment. It would recreate the graph handoff that the monolith direction removes. Revisit only if measured Render behavior cannot meet the workload.
- Compare exact output bytes only — declined. External tools may add harmless volatile data. Pair normalized source comparison with runtime proof. Revisit exact bytes for Compiler-owned templates.
- Let the Compiler repair or complete the Plan — declined. That would move judgment beyond review and create an undisclosed effective Plan. Revisit only if the product intentionally adds another visible review step.
Deferred: attempt-aware dispatch
Deferred until measured recovery needs automatic retry. The initial production adapter keeps the single-table boundary described under Dispatch and the local lifecycle.
If measured recovery requires automatic retry, a later Render compile-start flow would record durable dispatch
intent, create the first pending_dispatch
Compilation Attempt, and enqueue a short Solid Queue job. The dispatcher would reserve only a pending_dispatch
attempt and move it to dispatching before asking Render to start bin/compile COMPILATION_ATTEMPT_ID. A confirmed
response would make the attempt launched; an ambiguous response would make it launch_unknown. Solid Queue would
not perform expensive compilation.
A row-locked retry would mark the prior launch_unknown attempt superseded and create a new pending_dispatch
attempt rather than replay the same launch identity. The One-Off would claim execution only by moving the current
launched attempt to running. A successful run would promote its exact retained artifact and end as succeeded;
attempts could instead end as failed, cancelled, or superseded. A separate Publication could observe that
terminal Compilation afterward. The Compiler remains an ordinary callable object; bin/compile is a thin Rails
entry point, and a Rake task may be only a local convenience. Durable compilation state belongs in Compilation rows
and explicit services rather than an in-memory Interactor or Organizer context.
In the future attempt-aware dispatch design, Render launch arguments may contain a CompilationAttempt identifier
because it is a non-secret fencing identity, not a bearer capability. The One-Off's deployed environment and database
access would establish trust. A row-locked execution claim would accept only the current launched attempt;
running, superseded, cancelled, and terminal attempts would reject replay. Current Compilation authority ends after
promotion stores the exact artifact. A publication worker uses its own durable identity and fence rather than
inheriting future Compilation Attempt authority.
A future attempt-aware Compilation Record would project diagnostics only from its authoritative attempt. Diagnostics from superseded or otherwise non-authoritative attempts would remain operational attempt detail rather than being merged into the product result.
Open questions
These remain open; none blocks the bounded release.
- How is meaningful output normalized without hiding a material dependency or generated-source change?
- Which dependency inputs may use a cache, lockfile, or network, and how are they pinned and recorded?
- How do Compiler and target-profile releases declare compatible Foundation Plan formats?
- Which invalid requests advance far enough to receive a durable Compilation Record?
- How should end-user cancellation and operator recovery be exposed around the implemented fenced ref reads and retained-Head replay without weakening the no-mutation-replay boundary?
- What replaceable preview can precede transfer of the owner-controlled GitHub repository without confusing temporary First Draft control with repository ownership?
Decisions
Decision 0004 (repository-only) records the separate-program rationale, and decision 0019 (repository-only) records the move to one Project graph and same-release One-Off compute.