How to Drive Digital Engagement with a Digital Data Architecture

by Rick Skriletz – 

Customers are the lifeblood of every business. Providing services and experiences customers desire, using the channels they prefer, and keeping up with their rapidly changing tastes and expectations is a challenge for every business. Digital technologies accustom customers to new behaviors and expectations. Customer experiences translate into engagement expectations for employees and partners as well.

Successful digital engagement requires understanding and adapting to the new behaviors and expectations of digital users quickly. Their digital experiences require enterprise data, analytics, and applications to adapt to real-time digital processing – an adaptation that requires a next-generation, enterprise digital data architecture.

The Digital Architecture Framework

To understand how a digital data architecture is different, it is necessary to understand how digital engagement is changing the requirements for using and structuring data.  

Digital engagement is driven by events and platforms. Whether via a web page, mobile app, a digital assistant using natural language processing, or an automated device, interactions are discreet events that need to be processed. Whichever channel is used, an action is chosen that will require a response. There are two important factors a digital architecture needs to consider.

The first is that digital engagement uses platforms. Mobile apps are developed for iOS and Android. Digital assistants are developed using Alexa, Cortana, Google Assistant, and Siri. These platforms have their own standards, SDKs, and development standards that must be adhered to. They also offer features to take advantage of, like maps, auto-dialing phone numbers, language vocabulary, and more to enhance the customer experience. These platforms also have ongoing changes and requirements to keep digital engagement applications current.

The second is that customers and other digitally engaged users expect all channels to have the same information about them and to operate in a consistent manner, whichever channel they use. This has implications for a data architecture:

• The data must be separate from individual digital platforms so that information about a customer is available to all of them.
• Data will need to be collected from a digital platform so that a complete history of a customer’s interactions is available wherever needed.
• Interactions will be real-time, driven by the digital event initiated by a customer.
• Interactions are small, bite-sized units of data rather than the transaction or report data needs of an application or data mart.
• As platforms evolve, new digital interactions will be created, which means a digital data architecture must evolve easily and quickly as well.

Digital engagement requires a bridge between real-time digital events and legacy application systems of record and existing data warehouses, data marts, and other systems of insight. A digital architecture framework is required:

Architecture for Digital Engagement

For data, an event-responsive, digital data architecture means physical data structures are easy to adapt and change. It must implement new digital platform interactions quickly. Most importantly, it needs to perform well in a real-time digital environment and provide the data captured to application systems of record and others needing it.

Requirements for an Event-Responsive, Digital Data Architecture

SQL-based data architecture practices are not suited to supporting digital engagement. Real-time digital operations are too dynamic and users’ digital platform behaviors too variable to utilize data architecture practices driven by transactions, reports, dashboards and queries.

Here are some important requirements for event-responsive data:

• Real-time, event-responsive data is dynamic. Driven by events, event-responsive data is a dynamic and, to support new APIs, frequently changing data store. That differentiates it from application databases, data warehouses, and data marts that are architected first to support defined transaction, reporting, dashboard, and query use cases.
• Event-responsive data must be architected to operate in, and across, clouds. Digital platforms operate in the cloud but not necessarily all in the same cloud. This means hosting the digital platforms it supports.
• Event-responsive data interactions are via APIs. So, another aspect of event-responsive data is that it operates using cloud-ready APIs and containers to respond to data events and process data streams.
• Event-responsive data is designed around real-time API interactions. API data content is the basis for the data architecture of an event-responsive data store. Relational tables are not as suited to the demands of event-responsive data as data objects, like JSON.
• Event-responsive data objects provide additional flexibility. Data objects can add new data elements, lineage, metadata, history of data changes, and more into their content. This flexibility enables audits, compliance, and real-time responses needed for success.
• Event-responsive data aggregates dynamically. An important aspect of a dynamic, real-time data store is that events can change aggregated values such as the total number of items in a shopping cart or the value of a portfolio. Automated processes that calculate and store aggregates asynchronously as a result of events are required.
• Event-responsive data informs processing systems. As digital interactions become the primary method of engaging customers, event-responsive data captures data important to non-event driven systems of record and systems of insight. This data needs to be delivered to them for further operational business processing.

Data architecture has been the design backbone of applications, data warehouses, data marts, data lakes, and analytics. Common aspects of data architecture include “how it is stored, arranged, integrated, and put to use”¹. This is not sufficient for an event-responsive, digital data architecture.

Architecting a data solution for an evolving, API-responsive, digital data store is different than a logical and physical data design driven by transactions, reports, dashboards and queries, all of which have an inherent structure to them that guides the architecture.

Architecting a Digital Data Architecture through Governance 

Transformation of data architecture is needed to support digital engagement. As stated above, the digital architecture framework provides the ability to architect digital engagement, event-responsive data, and data delivery functionality to enterprise, rather than siloed operational, standards.

This governance practice transforms data architecture from logically and physically structuring data for particular uses, like an application or data mart, to specifying and governing enterprise data before it is built and put to use.

Proactive, before-the fact governance-driven data architecture applies new principles:

• Use the same names and object structures for data objects and data elements wherever they are used for digital engagement, event-responsive data, and data delivery.
• Incorporate operational metadata into the data objects to support lineage, audits, and more.
• Incorporate history of changes to a data object in the object as well.
• Incorporate security, privacy, and compliance policies into data objects so they are bound together, inseparable, and independent of processes that may fail in real-time.

Governance-driven data architecture establishes standards for data, its interactions, and its use throughout the enterprise. Rather than focusing on how data is stored, arranged, integrated, and put to use, this approach provides consistency in data terminology, taxonomy, security, metadata, and more throughout the enterprise.

Guidelines for Implementing a Digital Data Architecture

Unlike a data warehouse or data mart, an event-responsive, digital data architecture evolves through an ongoing sequence of projects. This is possible without needing a data architecture to start its development for a few reasons:

• Containers and APIs, using Docker and Kubernetes, have emerged as the standard for digital and cloud-based processing.
• APIs for data interactions decouples data structures from data use, which frees the user of event-responsive data from navigating data structures.
• An API service can be thought of as a microservice that provides or processes data that allows a digital event to perform its specific activity.
• Using data objects instead of data tables for event-responsive data provides flexibility so adding new data objects and data elements has little impact on previously created API services.

Here are the guidelines for how to implement a digital architecture and event-responsive data project by project:

• Focus on delivering projects quickly. Digital engagement is volatile because digital platforms change regularly, as do users’ tastes and behaviors. Projects implement new digital interactions in response to this and need to be delivered quickly.
• Design the user experience. Digital platforms require a well-designed user experience to engage users. The user experience also identifies the data each event needs.
• Define the APIs and their data content. The data an event creates or uses is an API interaction with the event-responsive data repository. This is the basis for designing data objects and their data content.
• Govern the data content of the APIs. Because the APIs are an outcome of the user experience design, data objects, relationships between them, and data elements that are new to the event-responsive data repository can go through governance before they are physically in place or used. Governance establishes for the enterprise:
  • Definitions of data objects and data elements.
  • Rules to enforce taxonomy / data classifications and hierarchies.
  • Approval of any automated processes used to process data presented by a digital event (this is another form of rule).
  • Approval of data for use by the API, including data privacy, data masking, and other regulatory and compliance concerns.
  • Specification of data history, lineage, metadata, and operational processing data to be captured and retained in data objects.
  • Security requirements, such as data encryption, tokenization, and key management.
• Populate the event-responsive data repository. When data needs to be added to the event-responsive data repository, such as customer data for a new digital engagement platform, only add data objects and data elements that support designed APIs and have been fully approved by this before-the-fact governance process.
• Deliver new or changed data elements to existing processing systems. Once digital events begin to bring new data objects or changed data into the event-responsive data repository, these need to be communicated to existing processing systems:
  • Providing these as a data stream using a pub-sub pattern supports near real-time as well as batch processing systems.
  • The content of the data stream should be the new or changed data objects to enforce a consistent view of enterprise data from origination through delivery.

Then repeat this process for each subsequent project. Projects can also be concurrent because the only dependencies are capacity for designing APIs, governing data content, and loading the event-responsive data repository with any data required for a project to function.

Fortunately, technologies exist that make this method of delivering projects with a digital data architecture possible, but that is a topic for another time.

Works Cited

1. Wikipedia "Data architecture" https://en.wikipedia.org/wiki/Data_architecture