Traditional BI models are built around structured data and predefined schemas, but valuable business context increasingly exists in documents, text, logs and semi-structured sources. This blog explores why schema-first BI can lose that context and how modern data pipelines can use intelligent ingestion, feature extraction and semantic enrichment to create richer, more connected business intelligence.
Business intelligence has traditionally been built around a simple assumption: if data can be structured into rows and columns, it can be analysed.
That assumption shaped an entire generation of BI architecture.
Data was extracted from operational systems, transformed into predefined tables, loaded into a warehouse and organised into dimensional models. Once the schema was established, dashboards and reports could be built on top of it. The architecture worked because the most important business information was expected to arrive in predictable formats: transactions, customer records, financial figures, product data and other system-generated fields.
The problem is that this is no longer where all of the important information exists.
Modern organisations generate enormous volumes of information through contracts, emails, PDFs, presentations, technical reports, customer conversations, support tickets, forms, images, web content, application logs, JSON payloads and other sources that do not conform neatly to a relational schema.
The issue for BI is therefore not simply that organisations have more data.
It is that much of the context required to understand the structured data already sitting inside the warehouse exists somewhere else.
A sales figure tells you what was purchased. A customer email may explain why the account is at risk. A contract may define the commercial terms behind the transaction. A service report may explain why delivery costs increased. A technical document may contain the reason a product failed. An operational log may reveal the sequence of events that preceded an incident.
Traditional BI can report the number.
Modern BI needs to understand the context behind it.
That requires a fundamental change in how data pipelines are designed.
Organisations such as Merit Data & Technology are increasingly approaching this problem from the data layer upwards, combining intelligent document processing, data engineering and semantic enrichment to make information from non-traditional sources usable alongside structured enterprise data. The significance of this approach is not the extraction technology itself. It is the architectural principle behind it: data needs to be understood before it can be reliably modelled for intelligence.
Traditional BI architectures tend to begin with the target schema. The organisation decides which entities matter, defines fields and relationships, establishes data types, maps source systems to those fields and then builds transformation pipelines to populate the resulting model.
For structured operational data, this approach remains effective. An ERP transaction containing a customer ID, invoice number, date, currency and value already has a defined structure and relatively explicit meaning. The source system has performed much of the semantic work before the data reaches the BI pipeline.
The problem begins when the same modelling approach is applied to information whose meaning depends on context, hierarchy or relationships between multiple pieces of information.
This is where premature dimensional flattening becomes a fundamental architectural problem.
Premature dimensional flattening occurs when complex source information is forced directly into a rigid analytical structure before the relationships and context within that information have been properly understood. A multi-page contract, for example, may contain parties, obligations, pricing conditions, renewal terms, exceptions and dependencies spread across different clauses. A diagnostic or service report may connect an asset to a failure condition, a sequence of observations, a maintenance intervention and a subsequent outcome. A semi-structured JSON payload may contain nested objects whose meaning depends on their position within a larger hierarchy.
When these sources are mapped directly into a predefined star schema, the pipeline has to flatten information that is inherently relational, hierarchical or contextual.
The result is often a lossy representation.
Nested dependencies can be reduced to individual fields. Causal chains can become disconnected attributes. Temporal conditions can be separated from the events they qualify. Relationships that existed naturally within the source can be compressed into arbitrary string columns or fragmented across multiple tables without preserving the context that made those relationships meaningful in the first place.
Consider a service report describing a machine failure. The analytical value is not simply that the document contains the terms "bearing failure" and "maintenance". The useful information lies in the relationship between the machine, the affected component, the operating condition, the failure event, the maintenance action, the replacement part and the subsequent operational outcome.
Flattening those elements too early can leave downstream analytics with individual values but not the structure connecting them. The pipeline may successfully populate a database, but the resulting schema can contain less usable information than the original source.
The same problem exists in commercial and operational data. A contract clause may state that a pricing condition applies only after a particular volume threshold is reached and that the condition changes at renewal. Representing those details as isolated text fields does not preserve the dependency between the threshold, pricing rule and renewal event. The information has technically been captured, but its business meaning has been weakened.
This is why the problem with schema-first design is not simply that schemas are rigid. It is that the schema is being imposed before the information has been understood.
A modern BI pipeline should first establish what the source represents, which entities, attributes and events it contains, how those elements relate to one another, what temporal or contextual conditions apply and where relationships span multiple sources. Only after that semantic structure has been established should the information be transformed into structures optimised for a particular analytical purpose.
The sequence therefore changes from:
Source → schema → transformation → warehouse → BI
to:
Source → ingestion → understanding → feature extraction → semantic enrichment → modelling → BI
This does not mean abandoning dimensional modelling. Star schemas, relational models and analytical warehouses remain valuable for consumption and performance. The architectural change is that they should represent information that has already been understood and enriched, rather than acting as the first place where meaning is established.
This is also where intelligent document processing becomes an important part of the BI architecture rather than a separate document automation capability. The role of technologies such as those used by Merit is not simply to extract text or populate predefined fields. It is to interpret information across complex documents and formats, identify the entities, attributes and relationships within them and create structured representations that can subsequently be enriched and connected with operational data.
The schema therefore becomes an output of the data engineering process, rather than the starting assumption that determines what information survives it.
The objective is not to make every source conform to the same structure. It is to preserve and establish the meaning within those sources before deciding how that meaning should be represented for analytics.
In conventional BI, ingestion is often treated as a transport problem.
Move the data from source A to storage B.
For structured systems, this approach remains useful. But when the source contains documents, text, images, logs or evolving payloads, ingestion becomes the first stage of intelligence.
A modern ingestion layer must preserve the original source while also identifying what is contained within it.
That means recognising document types, detecting layouts, extracting text, identifying tables, interpreting metadata, identifying entities and capturing relationships between pieces of information. For semi-structured sources, it means understanding changing payload structures, nested objects and inconsistent field representations rather than assuming that every source will continue to conform to yesterday's schema.
The raw source should therefore remain available as evidence, while extracted information becomes progressively more structured through subsequent processing.
This distinction matters.
If ingestion immediately converts a complex document into a handful of predefined database fields, everything that does not fit those fields can disappear. The resulting dataset may look clean, but it can be less informative than the original source.
A modern BI pipeline should therefore treat ingestion as the beginning of a layered data architecture rather than as the point at which information is forced into an analytical schema.
A Medallion-style ingestion pattern provides a useful architectural model for this. At the Bronze layer, raw source artefacts should be ingested and preserved immutably in their original form. This includes not only documents and structured payloads, but also multi-modal content such as PDFs, images, tables, JSON, XML, logs and other source formats. Where spatial layout carries meaning, the ingestion process should preserve the associated layout metadata alongside the source rather than reducing the content immediately to plain text.
This creates an immutable source record that can be revisited as extraction methods, semantic models or analytical requirements evolve.
The Silver layer can then perform the deeper processing required to make that source analytically usable. Semantic parsing, entity extraction, relationship identification, terminology normalisation and other enrichment processes can transform raw artefacts into structured information while maintaining a link back to the originating source. Provenance is therefore preserved through the transformation process rather than being lost when information is extracted.
Only after these stages should information be prepared for dimensional aggregation and downstream analytical models.
The resulting architecture is therefore better represented as:
Bronze: Immutable source artefacts → Silver: Semantic parsing, entity and relationship extraction → Gold: Analytical models and dimensional aggregation → BI and analytics
The distinction between these layers is important. Bronze preserves what the organisation actually received. Silver establishes what that information means and how its constituent entities and relationships connect. Gold then shapes that enriched information into structures optimised for reporting, analytics and business intelligence.
Merit's approach to intelligent document processing fits naturally into this model because documents are treated as sources of business information rather than simply files that need to be converted into machine-readable text. This is particularly important when documents contain tables, spatial relationships, narrative context, inconsistent terminology or information whose meaning depends on its relationship to other entities and events.
By preserving the original source and its provenance before dimensional aggregation, organisations retain the ability to trace analytical information back to the evidence from which it was derived. The objective is therefore not to make every source look identical at ingestion. It is to progressively increase the analytical usability of that information while preserving the context and provenance required to understand it.
One of the biggest misconceptions about unstructured data is that its value comes primarily from the words it contains.
In reality, much of its analytical value lies in the relationships those words describe.
Consider a maintenance report. The useful information is not simply that the report contains the phrase "bearing failure". The analytical value comes from understanding that a particular machine experienced a particular failure, that the failure occurred at a specific point in time, that a particular component was involved, that a maintenance intervention was performed, that a replacement part was used and that production downtime subsequently increased.
These are not isolated data points. They form a connected set of entities, events and relationships.
This is where Knowledge Graph Entity Extraction becomes important. Rather than treating an unstructured report as a block of text or reducing it to a collection of keywords, the extraction process can decompose the information into typed entity nodes and the relationships connecting them.
A machine can be represented as one entity, a bearing as another, the failure as an event, and the maintenance intervention as a subsequent event. Directed relationships can then connect those entities according to what the source actually describes.
These relationships can be represented through ontological triples, such as:
Machine A → experienced → Bearing Failure
Bearing Failure → affected → Component B
Bearing Failure → occurred at → Timestamp T
Maintenance Intervention → addressed → Bearing Failure
Maintenance Intervention → used → Replacement Part C
The significance is that the analytical system no longer has to treat the document as the primary unit of search. It can operate on the entities and relationships extracted from it.
Instead of asking whether a report contains the words "bearing failure", a BI platform can traverse the resulting knowledge graph to identify machines associated with a particular failure mode, trace the maintenance actions that followed those failures, identify the components involved and examine the operational outcomes associated with them.
The same principle applies to commercial information. A contract may specify a pricing arrangement, a volume commitment, a renewal condition and an exception clause. These elements can be extracted as typed entities and connected through relationships that preserve the conditions under which they apply. The resulting representation can then be connected to customers, transactions, products and other operational records.
This is fundamentally different from flattening extracted information into arbitrary fields or storing the original text for basic keyword search. A text search can find where a term appears. A knowledge graph can represent what an entity is, what it is connected to, what happened to it and how that relationship relates to other entities and events.
The BI pipeline therefore needs to move beyond extraction towards a representation of entities, relationships and business meaning.
The objective is not simply to turn a PDF into a table. It is to transform the information contained within that PDF into connected, structured knowledge that can participate in the organisation's wider data architecture.
For example, a document processing layer can identify a component mentioned in an engineering report, resolve it against a known equipment record and connect the reported failure to a specific maintenance event. That relationship can then be made available to downstream analytical models alongside structured equipment, production and maintenance data.
This distinction is central to the way Merit approaches data and intelligent document processing. Extraction is only the first step. The greater objective is to create structured and connected information that can be incorporated into the wider enterprise data environment, preserving the relationships that make the original information analytically meaningful.
Semi-structured data is often treated as an easier version of unstructured data because it already contains machine-readable elements.
That assumption can be misleading.
JSON, XML, application events, API payloads, log files and other semi-structured formats can contain substantial business value, but their structures are often dynamic. Fields can be added, removed, nested differently or represented inconsistently between systems and versions.
This creates a problem that modern BI pipelines need to address at two different levels: syntactic change and semantic change.
At the syntactic level, modern pipelines need to be capable of absorbing changes in source structure without requiring the entire downstream architecture to be redesigned. Dynamic schema evolution and reconciliation mechanisms, such as those available through technologies including Delta Lake and Apache Iceberg, can allow pipelines to accommodate new fields, changing nested structures and evolving payload definitions.
For example, an upstream API may introduce a new nested object within an existing JSON payload. A resilient ingestion layer should be able to recognise and preserve that structural change rather than treating it as a pipeline failure or silently discarding the new information.
But syntactic compatibility does not establish business meaning.
A pipeline may successfully ingest a new field called status, for example, without knowing whether the value "closed" means that a customer issue has been resolved, that a workflow has reached its final state or that a particular operational process has been terminated. Similarly, an upstream system might replace a numerical status code with a textual enumeration without changing the underlying business concept.
This is where semantic contract binding becomes necessary.
Semantic validation boundaries establish what a field, entity or event is expected to mean, independently of the precise syntax used by the source system. Rather than allowing structural changes to flow directly into downstream metrics, the pipeline can evaluate whether the changed representation still maps to an established business concept.
This creates an important separation:
Syntactic evolution asks: Can the pipeline understand the new structure?
Semantic contract binding asks: Does the new structure still represent the same business meaning?
Both are required.
Without syntactic flexibility, relatively minor upstream changes can break ingestion pipelines. Without semantic validation, those same changes can pass through successfully while quietly altering the meaning of downstream data.
Consider an operational API that introduces a new status enumeration or changes the structure of an existing status object. A schema-evolution engine may absorb the structural change automatically. A semantic validation boundary should then determine how the new representation maps to the organisation's established business definition of that status.
Only once that mapping has been validated should the value be allowed to influence downstream KPI definitions and analytical models.
This distinction becomes increasingly important when information from multiple operational systems is brought together. Data integration is not achieved simply because two sources can be loaded into the same warehouse. Their representations need to be reconciled at the semantic level so that apparently similar fields and events do not produce inconsistent business interpretations.
Modern BI pipelines therefore need to separate schema evolution from semantic governance. The ingestion architecture should be flexible enough to absorb changes in source syntax, while the semantic layer should provide the boundaries that protect established business definitions.
The result is an architecture that can evolve without allowing upstream syntax changes to silently corrupt downstream intelligence.
The objective is not to prevent source systems from changing. It is to ensure that when they do change, the BI environment can absorb the structural variation while preserving the meaning of the metrics and business concepts that depend on that data.
This changes the role of the semantic layer in modern BI.
A semantic layer should not merely provide friendly names for database fields. It should provide a consistent representation of business entities, concepts, relationships and metrics across heterogeneous sources.
To achieve that, the pipeline needs to establish canonical representations.
A customer may appear as a legal entity name in a contract, an account number in an ERP system, a company name in an email and an organisation ID in a CRM. These are not necessarily different customers. They may be different representations of the same business entity.
Similarly, a product can have different names across engineering systems, sales systems, supplier documentation and service records.
Without semantic enrichment, BI platforms may treat these representations as separate data points.
With semantic enrichment, they can become connected observations about the same underlying entity.
This is why entity resolution, terminology normalisation, relationship extraction and metadata enrichment should happen before the final analytical model is established.
The pipeline needs to understand the business before the BI layer attempts to measure it.
This principle also changes how semantic layers should be designed. Rather than building the semantic model exclusively around the fields that happen to exist in a warehouse, organisations can build it around the entities, relationships and business concepts discovered across their wider information estate.
The warehouse then becomes one representation of that model, rather than the sole definition of it.
This is the architectural shift at the centre of modern BI.
In a schema-first pipeline, the sequence is generally:
Source → schema → transformation → warehouse → BI.
For unstructured and semi-structured data, a more effective model is:
Source → ingestion → feature extraction → semantic enrichment → canonical entities and relationships → analytical schema → BI.
Feature extraction is the point at which raw information becomes analytically meaningful.
For a customer interaction, extracted features might include sentiment, topic, intent, product mentioned, complaint category, escalation indicator and referenced account.
For a technical document, they might include equipment, component, failure mode, operating condition, maintenance action and incident date.
For a contract, they might include parties, obligations, dates, pricing conditions, renewal terms, jurisdictions and exceptions.
These features do not necessarily belong directly in the final BI schema.
They first need to be interpreted and related.
Once that semantic structure exists, the organisation can decide how it should be represented for reporting, analytics and downstream applications.
This creates a much more durable architecture because the analytical model is no longer tightly coupled to the exact format of the source document.
It also creates an important separation between extraction and modelling.
Extraction asks: What information exists in this source?
Semantic enrichment asks: What does that information mean and what does it relate to?
Modelling asks: How should that information be represented for a particular analytical purpose?
Keeping those stages distinct gives organisations greater flexibility. The same enriched information can potentially support different analytical models, applications or AI workflows without requiring the organisation to repeatedly interpret the original source.
Building for unstructured and semi-structured data first does not mean replacing relational databases, warehouses or structured operational data.
Structured data remains essential.
Financial transactions, inventory levels, customer records, production volumes and operational KPIs provide the quantitative foundation for BI.
The problem is that structured data often represents the outcome of a process without capturing the complete context behind that outcome.
The modern architecture therefore needs to connect both.
A sales dashboard may show declining revenue from an account. Structured CRM and ERP data can quantify the decline. Unstructured customer correspondence can reveal dissatisfaction with service. Contract data can show that a renewal is approaching. Support records can reveal recurring incidents.
Together, these sources produce a materially different picture.
The value comes from connecting them.
Modern BI is therefore less about choosing between structured and unstructured data and more about creating an architecture in which both can contribute to the same analytical model.
This is where data engineering becomes more than an exercise in integration. The challenge is to create a common information layer in which structured records and information extracted from documents, text and other sources can be interpreted in relation to one another.
The result is not an unstructured BI environment.
It is a richer structured environment built from a much broader definition of enterprise information.
This changes what organisations should expect from their BI pipelines.
The pipeline is no longer simply responsible for moving information into a warehouse.
It needs to establish a chain of understanding.
At ingestion, the system should preserve the original source and its metadata.
During extraction, it should identify relevant entities, attributes, events and signals.
During enrichment, it should resolve terminology, establish relationships, add contextual metadata and connect information across sources.
During normalisation, it should map heterogeneous representations to canonical business concepts.
Only after these stages should information be shaped into the structures required by the analytical environment.
This approach creates a more durable separation between source systems and analytical consumption.
A supplier can change its invoice format without forcing the BI team to redesign the entire business model. An operational application can introduce new fields without necessarily breaking the downstream semantic representation. A new document type can be incorporated into the information pipeline without requiring every existing dashboard to understand its original structure.
The architecture becomes capable of absorbing change.
This is particularly important as organisations add AI to their BI environments. AI systems require broader context than conventional dashboards because their outputs can depend on relationships and information that are not represented in a single structured table.
If that context is never extracted and connected, no downstream AI system can reliably recover it.
The architectural argument ultimately comes back to what BI is supposed to achieve.
The purpose of business intelligence is not to create more dashboards.
It is to help organisations understand what is happening, why it is happening and what they should do about it.
That requires context.
Consider an organisation monitoring customer churn. Traditional BI might identify accounts with declining usage or reduced spend. A modern pipeline can combine those signals with support conversations, contract terms, service incidents and customer feedback to establish a much richer picture of churn risk.
Consider manufacturing. Production data can show that downtime has increased. Maintenance reports, machine logs, engineering documentation and spare-part records can help establish which components, conditions and maintenance patterns are associated with that increase.
Consider professional services. Financial systems can show engagement profitability. Contracts, time records, project correspondence and delivery documentation can explain where scope changes, commercial exceptions or operational issues are affecting margins.
In each case, structured data provides the measurement.
Unstructured and semi-structured data provides much of the context.
The BI architecture needs both.
For Merit, this convergence of data engineering and intelligent information processing is important because the value of an extracted data point is ultimately determined by what it can be connected to. A customer complaint has greater analytical value when it can be linked to the customer, product, transaction, service history and commercial relationship to which it relates.
The same principle applies across industries.
Context turns extracted information into usable intelligence.
When organisations continue to design BI pipelines around structured data first, unstructured information is often pushed to the edge of the architecture.
It becomes a separate search system, a document repository or an AI experiment.
This creates another form of fragmentation.
The warehouse contains the numbers. The document repository contains the evidence. The CRM contains the customer record. The service platform contains operational history. The AI system may contain extracted insights.
Users are then expected to mentally connect them.
That is not a modern data architecture.
It is a collection of disconnected information systems.
The result is often slower analysis, duplicated extraction work, inconsistent definitions and incomplete decision-making. More importantly, organisations may invest heavily in analytics while continuing to leave some of their most valuable business context outside the analytical environment.
The answer is not necessarily to put every document into a warehouse.
It is to ensure that the information contained within those documents can participate in the same semantic architecture as structured data.
This is also why document intelligence should not be treated as an isolated automation project. If extracted information never makes its way into the organisation's broader data model, the organisation has automated extraction without solving the underlying information problem.
Merit Data & Technology approaches modern BI from the principle that data needs to be understood before it can be reliably operationalised.
Its Intelligent Document Processing capabilities are designed to process information across heterogeneous documents and formats, extracting entities, attributes and relationships that can be incorporated into downstream data environments.
This is particularly important where business information exists across multiple representations and cannot be reliably captured through conventional field-based extraction alone.
The objective is not simply document digitisation.
It is the creation of structured, enriched information that can participate in enterprise data models.
Merit's approach combines intelligent extraction with data engineering and semantic enrichment so that information from documents and other non-traditional sources can be connected to structured operational data. This enables organisations to move from isolated document processing towards a broader information architecture in which business entities, events and relationships can be used consistently across analytics and downstream applications.
KIAA, Merit's Know It All Agent, extends this approach into governed orchestration, helping coordinate data processing and workflow execution across complex information environments.
This matters because modern BI increasingly sits at the intersection of data engineering, intelligent document processing, semantic understanding and AI.
The organisations that can connect these layers will have a fundamentally richer analytical foundation than those that continue to treat documents and other unstructured sources as secondary data.
The most important change in modern BI architecture is not another dashboard technology or another warehouse platform.
It is a change in where data modelling begins.
For decades, organisations have started with the assumption that data must first fit a schema before it can become useful for analytics.
That assumption is becoming increasingly difficult to defend.
Business information does not arrive in rows and columns. It arrives through transactions, conversations, documents, applications, machines, APIs, reports, images and operational events.
The architecture needs to understand those sources before deciding how they should be modelled.
That means ingestion must become intelligent. Feature extraction must precede schema design. Semantic enrichment must establish meaning and relationships. Canonical entities must connect information across systems. Only then should the analytical schema and semantic model determine how the resulting intelligence is exposed to BI users.
The warehouse is still important.
The dashboard is still important.
The metric is still important.
But none of them can compensate for information that was discarded before the analytical process even began.
Modern BI will therefore belong to organisations that stop asking, "How do we fit this data into our schema?"
The better question is:
"What does this data tell us, what does it relate to, and how should we represent that understanding so the business can use it?"
That is the architectural shift from schema-first BI to intelligence-first data pipelines.
And it is becoming essential for turning the full breadth of enterprise information into reliable business intelligence.
