Data Engineering Services: Buy Contracts, Not Pipelines

Ask a data team how much they have built and you will get a pipeline count. Ask how many teams can get a trustworthy number out of it without asking that team for help, and the room goes quiet.

That gap is the whole problem with how data engineering services get bought. Pipelines are the unit of work. They are a terrible unit of value, because a pipeline nobody trusts is worse than no pipeline at all: somebody is already making decisions on it.

What are data engineering services?

Data engineering services are outside help designing and building the systems that move, store, shape, and serve data so other teams and systems can use it without rebuilding it each time.

The work covers five things, and they are usually sold as a bundle when they should be sequenced.

• Ingestion. Getting data out of source systems reliably and repeatably.
• Modeling. Turning it into shapes that answer real questions.
• Serving. Making it available to people, applications, and models.
• Governance. Provenance, quality, access, and definitions.
• Operations. Monitoring, alerting, and the boring work that decides whether anyone trusts it.

Most engagements over-invest in the first two and under-invest in the last three, which is exactly backwards for whether the result gets used.

Buy data products, not pipelines

The change that fixes most of this is a change in what you contract for. Instead of a pipeline, buy a data product: a named dataset with an owner, a documented definition, a freshness and quality guarantee, and a stable interface other teams read from.

The difference is not cosmetic. It changes who is accountable when the number is wrong.

• A pipeline has an author. When it breaks, somebody investigates.
• A data product has an owner. When it breaks, somebody is accountable for the consumers.

The data contract test

For any dataset that matters, six questions. If you cannot answer all six, it is not a product yet, whatever the platform cost.

1. Who owns it? A person, not a team inbox.
2. What does each field mean? Written down, in the same place as the data.
3. How fresh is it, guaranteed? A number, not “nightly, usually.”
4. What happens when it breaks? Who gets told, and how consumers know not to trust it.
5. Who is allowed to see it? Enforced, not documented.
6. How does a change get made without breaking consumers? Versioning, or you have a hostage situation.

Question six is the one that separates a platform from a liability. We have walked into estates with a well-built warehouse where nobody would change a column, because forty unknown consumers were reading the tables directly. That is not a data problem. It is a missing interface.

The anti-metrics

Four numbers that go up while things get worse. If your reporting leans on these, the program is measuring effort.

The caveat: none of these are useless, they are just inputs. Pipeline count is a fine capacity measure for a team lead. It is a bad progress measure for an executive, and it is the one that shows up in steering decks.

What “AI-ready” actually requires

Every data program now gets an AI justification attached, and most of them are honest about the ambition and vague about the requirement. The requirement is specific: a model needs data whose provenance, definition, and quality are documented at the moment it is used, because that is what makes the output explainable later.

That is not a Cabin opinion. NIST’s AI Risk Management Framework 1.0, released January 26, 2023, organizes AI risk work into four functions, Govern, Map, Measure, and Manage, and the Map and Measure functions are where the documented context and the tracked, evidenced characteristics of a system live. You cannot map or measure a system whose inputs are undocumented.

In regulated sectors the same requirement arrives as a data standard rather than a framework. ASTP/ONC’s HTI-1 final rule, issued December 13, 2023, set USCDI version 3 as the baseline set of data classes for certified health IT with a January 1, 2026 effective date. The lesson generalizes: the industries furthest along all converged on the same answer, which is that the record has to be structured and defined before anything clever sits on top of it.

This is why we get the data usable first in every engagement where a model is eventually going to read it, and why machine learning work that starts with the model rather than the record produces demos instead of systems.

What to build first

One data product, for one team that will actually depend on it, covering the entity your business argues about most. Customer, account, policy, patient, or order.

Build it with the contract attached: owner, definitions, freshness guarantee, access rules, versioned interface, monitoring. Resist the urge to make it general. The second and third products are dramatically cheaper because the pattern exists, and a first product scoped for every possible consumer never ships.

Where the sources are hostile, which is usually, that first build is as much data integration work as data engineering, and sometimes it means pushing back into the source system through legacy application modernization. Both are normal. What is not normal, and what we argue against, is a two-year platform build with no consumer attached to it.

For the industry-specific versions of this same sequencing argument, see digital transformation in banking and insurance digital transformation.

Frequently asked questions

What is the difference between data engineering services and data platform consulting?

In practice, scope. Platform consulting tends to mean selecting and standing up the technology, and data engineering means building the products that run on it. The failure mode is buying the first without a named consumer for the second, which produces a well-architected platform with three dashboards on it.

Do we need a warehouse, a lakehouse, or both?

It is the wrong first question, and vendors love it because it is the one they can answer. Decide the first data product and its consumers, and the storage choice usually narrows to something obvious. Teams that pick the architecture first spend the next year retrofitting requirements onto it.

How long until something is usable?

One governed data product for one consuming team is a one-quarter build for a focused team, including the contract and monitoring. If the first usable output is more than two quarters out, the scope is a platform program wearing a data product label, and it should be cut down.

Should we hire in or use outside help?

Use outside help to build the first product and establish the pattern, then own it. The pattern is the transferable part: the contract shape, the review, the monitoring, the way a change gets versioned. If a partner’s model depends on writing every subsequent pipeline for you, the cost curve never bends.

Where to start

Pick the entity your leadership meetings argue about, find the team that would depend on a trustworthy version of it, and build exactly that with a contract attached. One product, one owner, one consumer, in production. Then count consumers instead of pipelines.