Dbt ingest

--- name: dbt_ingest description: Maps dbt schema.yml / properties.yml models and sources into ktx semantic-layer overlays and column notes. Covers sources: vs models:, column data_tests (notnull, unique, acceptedvalues, relationships), and how bundle-time writes complement manifest backfill from git sync. Load when the WorkUnit's skillNames includes dbt_ingest or when raw files are dbt-YAML under models/ / sources/. callers: [memory_agent] --- # dbt → ktx (bundle ingest) Use this skill for uploaded dbt projects (dbt_project.yml at stage root, models/**, sources/**, schema.yml). There is no fetch() in v1:scheduled dbt parse / manifest.json pulls are out of scope; host-provided dbt sync may still backfill structured test metadata into _schema on the next sync. ## Mapping (models / sources → SL) | dbt | ktx | Notes | |-----|--------|--------| | models: entry with columns: | Overlay on the manifest table with the same name (after discover_data / entity_details) | One SL source per physical table; model name may differ from DB name:resolve using read_raw_file + warehouse context. | | sources: → tables: | Same as models; use identifier when present instead of logical name. | Schema + name must match how the connection views tables. | | Column description | column_overrides[].descriptions.user on the overlay | Do not overwrite dbt description keys from sync. | | data_tests: not_null / unique | Short hint in the column descriptions or notes: “dbt: not null”, “dbt: unique” | Full structured metadata is included in the manifest via sync; the skill ensures that the bundle-time SL text remains useful for the agent. | | accepted_values | Add a brief line in the column description: allowed values (truncate long lists) | Also mention enum-like use in discover_data / filters. | | relationships | Add or confirm joins: on the overlay only when to resolves to a real table via read_raw_file + discover_data / entity_details | If the ref cannot be resolved, capture the intent in a wiki page instead. | ## Physical schema grounding dbt YAML is documentation and test metadata; it does not grant permission to invent physical columns. Before writing any table-backed SL source, confirm the actual warehouse schema using discover_data, sl_discover, or entity_details, and use only confirmed column names in column_overrides:, computed-only columns:, grain:, joins:, segments:, and measures[].expr. For dbt context-source ingest, the dbt connection is usually not the warehouse connection. First, call sl_discover without connectionId, then write overlays to the connection that owns the matching manifest-backed source (for example, postgres-warehouse), not to the dbt connection (for example, dbt-main). If no matching manifest-backed source is visible on any warehouse connection, do not call sl_write_source; record emit_unmapped_fallback and keep the fact wiki-only. If a models: entry has no columns: block, or the available raw files do not confirm the physical column names, do not synthesize a full standalone source. Instead, write a wiki note or a description-only overlay for the resolved manifest table. If a business metric is described but its referenced column is not confirmed in the warehouse schema, omit the measure and document the unresolved intent in the wiki. Include rawPaths on every wiki_write, sl_write_source, and sl_edit_source call, using only the dbt YAML files that directly support the action. After every sl_write_source, call sl_validate. A validation error stating that a declared column or measure reference is missing from the physical table is a critical error: re-read the warehouse-backed source and rewrite it with confirmed names, or remove the invalid SL fields. ## Identifier Verification Protocol Before writing a wiki page or SL source on any topic: 1. discover_data({query: "<topic>"}) - see what wikis, SL sources, and raw tables already exist. Prefer updating existing pages over creating new ones. Before emitting any schema.table or schema.table.column into a wiki body, SL source, tables: frontmatter, sl_refs, or `emit_