AI-Enabled Data Migration

Circini Limited · Data Engineering

Data migration is usually the longest stretch of any data-platform program. The slowest, most repetitive part of that work is turning a mapping document into transformation logic you can actually trust, and it's a step where small errors are costly.

We wanted to know whether generative AI could shorten that step without cutting corners: could it take a mapping document and produce complete, reliable dbt models, under strict governance and with a human expert checking every result?

The short answer is yes. Below are two use cases we validated, the four-step flow that keeps the output consistent, and the controls that keep it trustworthy.

Key takeaways

• AI can build complete dbt models straight from a mapping document and add fields or updates to existing models.
• A fixed flow - Input → Enhance → Validate → Generate - keeps results consistent.
• A human SME validates and tunes every model; nothing ships unreviewed.
• All terminology is masked, work stays inside approved cloud boundaries, and testing is security-first, under official AI policy.
• Output quality tracks directly with mapping-document precision and structured prompting.

What is AI-enabled data migration?

It's the use of generative AI to write and update transformation code (dbt models and SQL) that moves data from source to target. The mapping document is the input, and an SME checks the output. This isn't about replacing engineers; it's about cutting out the repetitive part of migration so engineers can focus where judgment actually matters.

One governed flow, applied to every script

Both use cases run through the same four steps:

1. Input - the mapping document (plus existing SQL, for updates).
2. Enhance - descriptive, structured prompting frames the task and the expected logic.
3. Validate - an SME reviews the proposed logic in a human Q&A loop.
4. Generate - complete, verified dbt / SQL is returned.

The validation step is the trust anchor. Structured prompting and good inputs get you a strong draft; the SME makes it production-ready.

Use case 1: Initial load script generation

The AI builds a complete dbt model using only the mapping document as its source of truth. We used this for areas like Additional Info and Contracts: mapping document in, reviewed SQL out.

Use case 2: Incremental additions

This one handles change. We give the AI the existing SQL plus the mapping document, and it returns an updated model with the new fields already added, no full rebuild needed. One example: adding a Supplier Performance Reports field to a model already in use.

What makes the output reliable?

Three lessons stood out:

• Prompt engineering matters. Structured prompts significantly improve output quality and logic.
• Input quality is everything. AI performance directly reflects the precision of the mapping documents. Vague inputs produce vague models.
• Human validation is non-negotiable. SME oversight remains essential for verifying generated SQL models and tuning them.

Security and governance, by design

Because this involves enterprise data, the guardrails came first, not last:

• Policy alignment - all work was conducted under the official company AI policy.
• Data privacy - every term was masked; no sensitive data was used.
• Security first - testing happened only within approved boundaries.

In practice that means anonymising all terminology to prevent data leakage, keeping data strictly within approved cloud environments, and applying a security-first approach to every code-generation task.

Why it matters

• Development efficiency - accelerated model building from scratch and faster incremental updates.
• Governance - full alignment with company AI policy and data masking.
• Scalability - a proven flow for generating verified dbt models at scale.

Frequently asked questions