by Rick Skriletz –
Customers are the lifeblood of every business. Providing services and experiences customers desire, using the channels they prefer, and keeping up with their rapidly changing tastes and expectations is a challenge for every business. Digital technologies accustom customers to new behaviors and expectations. Customer experiences translate into engagement expectations for employees and partners as well.
Successful digital engagement requires understanding and adapting to the new behaviors and expectations of digital users quickly. Their digital experiences require enterprise data, analytics, and applications to adapt to real-time digital processing – an adaptation that requires a next-generation, enterprise digital data architecture.
The Digital Architecture Framework
To understand how a digital data architecture is different, it is necessary to understand how digital engagement is changing the requirements for using and structuring data.
Digital engagement is driven by events and platforms. Whether via a web page, mobile app, a digital assistant using natural language processing, or an automated device, interactions are discreet events that need to be processed. Whichever channel is used, an action is chosen that will require a response. There are two important factors a digital architecture needs to consider.
The first is that digital engagement uses platforms. Mobile apps are developed for iOS and Android. Digital assistants are developed using Alexa, Cortana, Google Assistant, and Siri. These platforms have their own standards, SDKs, and development standards that must be adhered to. They also offer features to take advantage of, like maps, auto-dialing phone numbers, language vocabulary, and more to enhance the customer experience. These platforms also have ongoing changes and requirements to keep digital engagement applications current.
The second is that customers and other digitally engaged users expect all channels to have the same information about them and to operate in a consistent manner, whichever channel they use. This has implications for a data architecture:
- The data must be separate from individual digital platforms so that information about a customer is available to all of them.
- Data will need to be collected from a digital platform so that a complete history of a customer’s interactions is available wherever needed.
- Interactions will be real-time, driven by the digital event initiated by a customer.
- Interactions are small, bite-sized units of data rather than the transaction or report data needs of an application or data mart.
- As platforms evolve, new digital interactions will be created, which means a digital data architecture must evolve easily and quickly as well.
Digital engagement requires a bridge between real-time digital events and legacy application systems of record and existing data warehouses, data marts, and other systems of insight. A digital architecture framework is required:
Architecture for Digital Engagement
For most companies, enterprise architecture focuses on processing systems. This digital architecture framework focuses on real-time digital operations for digital engagement, event-responsive data, and data delivery functionality architected for enterprise data use while integrating with, and minimally impacting, legacy processing systems.
For data, an event-responsive, digital data architecture means physical data structures are easy to adapt and change. It must implement new digital platform interactions quickly. Most importantly, it needs to perform well in a real-time digital environment and provide the data captured to application systems of record and others needing it.
Requirements for an Event-Responsive, Digital Data Architecture
SQL-based data architecture practices are not suited to supporting digital engagement. Real-time digital operations are too dynamic and users’ digital platform behaviors too variable to utilize data architecture practices driven by transactions, reports, dashboards and queries.
Here are some important requirements for event-responsive data:
- Real-time, event-responsive data is dynamic. Driven by events, event-responsive data is a dynamic and, to support new APIs, frequently changing data store. That differentiates it from application databases, data warehouses, and data marts that are architected first to support defined transaction, reporting, dashboard, and query use cases.
- Event-responsive data must be architected to operate in, and across, clouds. Digital platforms operate in the cloud but not necessarily all in the same cloud. This means hosting the digital platforms it supports.
- Event-responsive data interactions are via APIs. So, another aspect of event-responsive data is that it operates using cloud-ready APIs and containers to respond to data events and process data streams.
- Event-responsive data is designed around real-time API interactions. API data content is the basis for the data architecture of an event-responsive data store. Relational tables are not as suited to the demands of event-responsive data as data objects, like JSON.
- Event-responsive data objects provide additional flexibility. Data objects can add new data elements, lineage, metadata, history of data changes, and more into their content. This flexibility enables audits, compliance, and real-time responses needed for success.
- Event-responsive data aggregates dynamically. An important aspect of a dynamic, real-time data store is that events can change aggregated values such as the total number of items in a shopping cart or the value of a portfolio. Automated processes that calculate and store aggregates asynchronously as a result of events are required.
- Event-responsive data informs processing systems. As digital interactions become the primary method of engaging customers, event-responsive data captures data important to non-event driven systems of record and systems of insight. This data needs to be delivered to them for further operational business processing.
A strategy to make legacy processing systems digital: Today, legacy processing systems and event-driven systems need to coexist to support business operations. There is simply too much essential functionality performed by legacy applications. The future will be increasingly digital so replacing, over time, legacy functionality with digital engagement counterparts is an important strategic practice. This deprecates legacy applications and replaces them, piece by piece, with event-responsive data and automated analytics serving business functions through digital engagement platforms.
Data architecture has been the design backbone of applications, data warehouses, data marts, data lakes, and analytics. Common aspects of data architecture include “how it is stored, arranged, integrated, and put to use”1. This is not sufficient for an event-responsive, digital data architecture.
Architecting a data solution for an evolving, API-responsive, digital data store is different than a logical and physical data design driven by transactions, reports, dashboards and queries, all of which have an inherent structure to them that guides the architecture.
Architecting a Digital Data Architecture through Governance
Transformation of data architecture is needed to support digital engagement. As stated above, the digital architecture framework provides the ability to architect digital engagement, event-responsive data, and data delivery functionality to enterprise, rather than siloed operational, standards.
The opportunity for enterprise data management is to build new event-responsive data capabilities that support multi-cloud operations, interact with digital engagement platforms and data delivery services solely through APIs, and govern its data content to enterprise standards from its inception.
The opportunity for enterprise data delivery is to build new data stream-based capabilities that support event-responsive data and existing processing systems with the same governance of data content to the same enterprise data standards as event-responsive data.
This governance practice transforms data architecture from logically and physically structuring data for particular uses, like an application or data mart, to specifying and governing enterprise data before it is built and put to use.
Proactive, before-the fact governance-driven data architecture applies new principles:
- Use the same names and object structures for data objects and data elements wherever they are used for digital engagement, event-responsive data, and data delivery.
- Incorporate operational metadata into the data objects to support lineage, audits, and more.
- Incorporate history of changes to a data object in the object as well.
- Incorporate security, privacy, and compliance policies into data objects so they are bound together, inseparable, and independent of processes that may fail in real-time.
Architecting an event-responsive data repository through governance provides an opportunity to establish data content that embodies the highest level of standards for data definitions, consistency, correctness, rules, metadata, security, and more. Governance-driven means these are established before data is physically created or used. Governance enables consistent standards for data used from that point forward.
Governance-driven data architecture establishes standards for data, its interactions, and its use throughout the enterprise. Rather than focusing on how data is stored, arranged, integrated, and put to use, this approach provides consistency in data terminology, taxonomy, security, metadata, and more throughout the enterprise.
Guidelines for Implementing a Digital Data Architecture
Unlike a data warehouse or data mart, an event-responsive, digital data architecture evolves through an ongoing sequence of projects. This is possible without needing a data architecture to start its development for a few reasons:
- Containers and APIs, using Docker and Kubernetes, have emerged as the standard for digital and cloud-based processing.
- APIs for data interactions decouples data structures from data use, which frees the user of event-responsive data from navigating data structures.
- An API service can be thought of as a microservice that provides or processes data that allows a digital event to perform its specific activity.
- Using data objects instead of data tables for event-responsive data provides flexibility so adding new data objects and data elements has little impact on previously created API services.
Here are the guidelines for how to implement a digital architecture and event-responsive data project by project:
- Focus on delivering projects quickly. Digital engagement is volatile because digital platforms change regularly, as do users’ tastes and behaviors. Projects implement new digital interactions in response to this and need to be delivered quickly.
- Design the user experience. Digital platforms require a well-designed user experience to engage users. The user experience also identifies the data each event needs.
- Define the APIs and their data content. The data an event creates or uses is an API interaction with the event-responsive data repository. This is the basis for designing data objects and their data content.
- Govern the data content of the APIs. Because the APIs are an outcome of the user experience design, data objects, relationships between them, and data elements that are new to the event-responsive data repository can go through governance before they are physically in place or used. Governance establishes for the enterprise:
- Definitions of data objects and data elements.
- Rules to enforce taxonomy / data classifications and hierarchies.
- Approval of any automated processes used to process data presented by a digital event (this is another form of rule).
- Approval of data for use by the API, including data privacy, data masking, and other regulatory and compliance concerns.
- Specification of data history, lineage, metadata, and operational processing data to be captured and retained in data objects.
- Security requirements, such as data encryption, tokenization, and key management.
- Populate the event-responsive data repository. When data needs to be added to the event-responsive data repository, such as customer data for a new digital engagement platform, only add data objects and data elements that support designed APIs and have been fully approved by this before-the-fact governance process.
- Deliver new or changed data elements to existing processing systems. Once digital events begin to bring new data objects or changed data into the event-responsive data repository, these need to be communicated to existing processing systems:
- Providing these as a data stream using a pub-sub pattern supports near real-time as well as batch processing systems.
- The content of the data stream should be the new or changed data objects to enforce a consistent view of enterprise data from origination through delivery.
Then repeat this process for each subsequent project. Projects can also be concurrent because the only dependencies are capacity for designing APIs, governing data content, and loading the event-responsive data repository with any data required for a project to function.
This approach changes, for most companies, their data governance, architecture, and delivery practices. It will change project funding and development practices as well.
Fortunately, technologies exist that make this method of delivering projects with a digital data architecture possible, but that is a topic for another time.
1. Wikipedia "Data architecture" https://en.wikipedia.org/wiki/Data_architecture