The Modern Data Stack Decoded

Meaning Before Data: Ontology, Entity & Schema

Before a single byte moves through a pipeline, someone must answer a more fundamental question: what does this data mean? The concepts of ontology, entity, and schema are the language of that answer. They define the universe of things your data describes and the rules governing how it is represented.

1. Ontology

An ontology is a formal, machine-readable representation of the concepts within a domain and the relationships between them. More than a dictionary or a taxonomy, an ontology encodes logic — it tells a system not just what a “Customer” is, but that a Customer can place Orders, that Orders contain Products, and that Products belong to Categories.

In data architecture, ontologies underpin knowledge graphs, semantic search, and AI reasoning. Gartner’s March 2026 Data & Analytics Summit declared ontologies — combined with semantic layers — a “non-negotiable foundation” for AI-ready enterprise data by 2030. The discipline of context engineering, which emerged prominently in 2025–2026, treats ontologies as the structured-context layer that allows LLMs to reason reliably over enterprise data.

2. Entity

An entity is a distinct, identifiable object in the real world that your data describes. Entities are the nouns of your data model — Customer, Product, Transaction, Supplier, Employee. In the context of AI and natural language processing, Named Entity Recognition (NER) automatically identifies and classifies these objects within unstructured text.

Entity resolution — the process of determining whether two records across different systems refer to the same real-world entity — is one of the most persistent and valuable problems in data engineering. A customer who appears as “John Smith” in your CRM and “J. Smith” in your billing system represents an entity resolution challenge that, left unresolved, corrupts every downstream analysis.

8. Schema

A schema is the blueprint of a dataset — it defines the structure: which fields exist, what data types they hold, and what constraints apply. Schemas are the contract between data producers and consumers. A schema change upstream, such as renaming a column or altering a data type, is one of the most common triggers for downstream pipeline failures.

Modern data teams distinguish between schema-on-write (enforced at ingestion, traditional databases) and schema-on-read (applied at query time, common in data lakes). Schema registries — tools like Confluent Schema Registry for Kafka streams — allow schemas to be versioned and governed across teams.

-- A schema definition in SQL (schema-on-write)
CREATE TABLE orders (
  order_id      UUID          PRIMARY KEY,
  customer_id   UUID          NOT NULL REFERENCES customers(id),
  placed_at     TIMESTAMPTZ   NOT NULL,
  total_amount  DECIMAL(12,2)  CHECK (total_amount >= 0),
  status        TEXT          CHECK (status IN ('pending','fulfilled','cancelled'))
);

The Three-Layer Architecture: Physical, Logical & Semantic

Every mature data architecture separates concerns across at least three distinct layers. This separation ensures that business users can interact with clean, meaningful data without needing to understand the raw, physical storage underneath. Understanding where each layer begins and ends is fundamental to designing scalable, governable systems.

4. Physical Layer

The physical layer is where data literally resides — on disk, in memory, across network-attached storage, or distributed across cloud object stores. It encompasses raw files (Parquet, ORC, CSV), database table files, indexes, and partitions. Decisions at this layer — compression algorithms, file formats, partitioning strategies, and storage tiers — directly determine query performance and storage cost.

Physical layer concerns include columnar storage formats (Parquet and ORC dramatically reduce I/O for analytical workloads), data compaction (merging small files into optimal sizes for Spark or Trino processing), and partition pruning (organising data by date or region so queries skip irrelevant partitions).

6. Logical Layer

The logical layer describes how data is conceptually organised — independently of how it is physically stored. It defines tables, views, relationships, and transformations. This is the domain of data modelling tools and transformation frameworks like dbt, where analysts define business logic through SQL models that are compiled into physical queries.

A well-designed logical layer acts as a translation zone: it consumes messy, normalised source data and produces clean, denormalised, analysis-ready datasets. Common logical patterns include the Kimball star schema (fact and dimension tables optimised for BI tools) and the Data Vault (hub-link-satellite model optimised for auditability and change tracking).

5. Semantic Layer

The semantic layer is perhaps the most discussed — and most frequently misunderstood — concept in modern data architecture. It sits above the logical layer and translates technical data structures into business-meaningful concepts. A semantic layer does not just expose a table called orders; it defines what Revenue means, how Churn Rate is calculated, and which dimension relationships are valid for a given metric.

“By 2030, universal semantic layers will be treated as critical infrastructure, on the same level as data platforms and cybersecurity.”

— Rita Sallam, Distinguished VP Analyst, Gartner Data & Analytics Summit, March 2026

By 2025, three architectural patterns had crystallised for the semantic layer: BI-native (semantics embedded in a single BI tool like Looker or Power BI), platform-native (semantics built into Snowflake, Databricks, or BigQuery), and universal/headless (tool-agnostic layers like Cube or AtScale that serve metrics to any consumer, including AI agents). The Open Semantic Interchange specification, co-published by dbt Labs, Snowflake, Databricks, and Salesforce in 2025, signals industry convergence toward open, cross-vendor standards.

Metadata & Data Modelling: The Architecture of Understanding

3. Metadata

Metadata is, literally, data about data. It describes the origin, structure, meaning, usage, and lineage of a dataset. There are three primary types, each serving a different audience:

A transformative shift now underway is the move from passive to active metadata. Rather than serving as static documentation in a data catalogue, active metadata dynamically triggers pipeline decisions, propagates governance policies, freshens alerts, and informs AI recommendations. This positions metadata as a real-time control plane for the entire data platform — not a document left to grow stale.

9. Data Modelling

Data modelling is the practice of creating an abstract representation of your data and the relationships between its elements. It bridges business requirements and technical implementation, ensuring that the data structure actually reflects how the organisation thinks and operates.

Data modelling happens at multiple levels: conceptual (what are the key entities and high-level relationships?), logical (what tables, attributes, and relationships are needed?), and physical (how are these implemented for the specific database engine?). The rise of analytics engineering — led by tools like dbt — has made data modelling a collaborative, code-based, version-controlled discipline rather than a one-off diagram exercise.

Data Virtualisation & Vector Databases: A New Access Paradigm

7. Data Virtualisation

Data virtualisation enables users to query and analyse data from multiple sources without physically moving, copying, or consolidating it. Rather than building yet another pipeline to centralise data, virtualisation creates a virtual data layer that presents disparate sources — cloud warehouses, databases, APIs, files — as a single, unified view.

The architectural advantage is significant: virtualisation eliminates data duplication, reduces latency for certain access patterns, and dramatically simplifies governance (because data stays in its source system). The semantic layer frequently leverages virtualisation as its data access mechanism — connecting to Snowflake, BigQuery, Postgres, and a data lake simultaneously and presenting a single, consistent view to the BI tool or AI agent above.

10. Vector Database

A vector database is purpose-built to store and query high-dimensional vector embeddings — the numerical representations generated by machine learning models (particularly large language models). Unlike traditional databases that search by exact match or range, vector databases perform similarity search: finding the records whose embeddings are mathematically closest to a query embedding in high-dimensional space.

Vector databases are the backbone of Retrieval-Augmented Generation (RAG), the technique that allows LLMs to answer questions using your organisation’s private data without fine-tuning. A document, product description, or support ticket is converted into a vector embedding, stored in the vector database, and retrieved based on semantic similarity to a natural-language query. Leading platforms include Pinecone, Weaviate, Qdrant, pgvector (Postgres extension), and Databricks’ native vector search.

Data Pipelines & Orchestration: Engineering in Motion

11. Data Pipeline

A data pipeline is the automated sequence of processes that moves data from source systems through transformation steps to a final destination where it can be consumed. Pipelines are the circulatory system of a data platform — without them, data sits inert and unreachable. Every stage of a pipeline represents an opportunity to add quality, governance, and business value.

Modern pipelines take several forms, each suited to different latency and volume requirements:

A critical discipline in pipeline engineering is designing for failure — every pipeline will fail eventually. Idempotency (running a pipeline twice produces the same result), retry logic, dead-letter queues, and schema compatibility checks are not optional extras; they are the hallmarks of production-grade pipeline engineering.

12. Orchestration

Orchestration is the discipline of coordinating, scheduling, and monitoring the execution of pipeline tasks. While a pipeline defines what needs to happen, an orchestrator defines when, in what order, and what to do if it fails. Most orchestration frameworks represent pipelines as Directed Acyclic Graphs (DAGs), where nodes are tasks and edges define dependencies.

A best practice for 2026 is a unified orchestration framework that manages both real-time and batch workloads within a single visibility layer. This ensures consistent monitoring, SLA enforcement, and error handling across all pipeline types — ending the fragmentation of having separate tools for streaming and batch jobs with no unified operational view.

The Data Trust Triad: Quality, Observability & Lineage

You can have the most elegant architecture in the world — perfect ontologies, a beautifully designed semantic layer, and flawlessly orchestrated pipelines — and still have data that executives do not trust. Data quality, observability, and lineage are the operational disciplines that close the gap between having data and trusting it. They are, collectively, the immune system of your data platform.

13. Data Quality

Data quality is the degree to which data is fit for its intended purpose. It is not a binary condition — data is not simply “good” or “bad” — but a multi-dimensional assessment across standardised dimensions:

Data quality checks are increasingly embedded directly into pipelines — using tools like dbt tests, Great Expectations, or Soda CL — rather than being a separate, post-hoc audit activity. The most forward-looking teams in 2025–2026 are deploying AI-generated data quality tests: an AI agent samples tables, understands business context, and generates hundreds of test cases automatically, removing the bottleneck of manually writing tests for every column in every table.

14. Observability

Data observability is the ability to understand the health, status, and behaviour of your data system from its external outputs — without having to examine every internal component directly. Borrowed from software engineering (where it complements monitoring and logging), data observability applies the same “understand your system from its outputs” philosophy to data pipelines and datasets.

The five pillars of data observability, which should be continuously measured and alarmed upon, are:

Effective observability distinguishes between detecting a problem and diagnosing it. Detecting anomalies is the easy part — every orchestrator can send a failure alert. The real value is in root-cause analysis: automatically correlating signals across quality metrics, pipeline execution logs, schema change history, and user behaviour to pinpoint exactly where and why something went wrong, reducing mean-time-to-resolution from hours to minutes.

15. Data Lineage

Data lineage is the documented map of a data asset’s journey: where it originated, every transformation it underwent, and every system or report that consumes it. It answers the two most urgent questions in data incident management: “Where did this number come from?” and “If I change this table, what will break?”

Lineage operates at two levels. Technical lineage traces table-to-table dependencies through pipelines, transformations, and queries. Business lineage maps data to business processes — showing a non-technical stakeholder that their “Monthly Revenue” dashboard pulls from three different source systems, via four transformation steps, governed by two different data owners.

-- Lineage chain example (simplified)
-- Source: Stripe → raw_payments → stg_payments → fct_revenue → exec_dashboard

Source System:  stripe.payment_intents          [physical layer]
    ↓  (ingestion via Fivetran)
Raw table:      raw.stripe_payments             [physical layer]
    ↓  (dbt staging model)
Staging:        stg.payments                    [logical layer]
    ↓  (dbt mart model + currency conversion)
Mart:           fct.revenue                     [logical layer]
    ↓  (semantic layer metric definition)
Metric:         Monthly Recurring Revenue (MRR)  [semantic layer]
    ↓  (consumed by BI tool)
Dashboard:      Executive Revenue Report         [consumption]

Modern lineage is captured automatically — from orchestration layer execution logs (which pipeline reads from which dataset), database query logs (which queries reference which tables), and transformation tools (dbt’s native lineage graph). The key requirement for enterprise-scale lineage is column-level granularity: knowing not just that Table A feeds Table B, but that the net_revenue column in Table B is derived from the amount column in Table A minus the refund_amount column in Table C.

How the 15 Concepts Form a Coherent Architecture

These 15 concepts are not independent silos — they compose a layered, interdependent system where each element depends on and enables the others. Here is how they connect in a mature, AI-ready data organisation:

What This Means in Practice: A Maturity Roadmap

Understanding these 15 concepts is necessary but insufficient. The following maturity progression gives data leaders a practical framework for sequencing investments: