How using a blockchain database solved our scalability problem

This blog aims to illustrate pain points we've run into at various stages of development. We hope to help engineers and product folks to overcome similar obstacles by explaining how we overcome our own. 

In this first installment, Osano's head of IT discusses a common problem at any company, whether an early-stage company or a more mature stage: scalability. 

The problem 

Today, regulations like the EU's General Data Protection Regulation (GDPR) and California's Consumer Privacy Act (CCPA, but soon to be replaced by the California Privacy Rights Act) forced most companies, depending on size, to start capturing an end user's consent before collecting and storing their personal data.  

If we were to implement a consent-capturing service at Osano, how would we prove to our customers that a specific end user granted their consent? How could we capture every update to that consent so we could verify it throughout that customer's lifecycle?

Compounding that problem, we also faced performance concerns about building such a service. Osano's Consent Management Platform (CMP) is necessarily one of the first scripts to load on a customer's application. We also expected millions of consents to be stored every day, even every hour, on behalf of our customers' end users. So we knew it would have to be scalable, responsive and secure.

We immediately started looking towards blockchain technology to provide cryptographic proof that no one could tamper with a consent record we had stored. This approach would allow us to store an end user's consent, along with any later modifications to that consent, and later recall that information for both our customers and their end-users in an entirely verifiable way.

The solution

Amazon Web Services' Quantum Ledger Database (QLDB) service offers a ledger database along with a blockchain verifiable transaction log in a scalable managed service. A ledger database stores the records and doesn't permit their removal. The history of a particular record is logged, and any changes throughout can be recalled. Plus, using blockchain technology, this record transaction log can be cryptographically verified as to the authenticity and integrity of the history of that record.

QLDB has allowed our team to focus on building new privacy and compliance features and capabilities in our platform without building, maintaining and scaling a service to store consent records securely with a verifiable lineage. This service is essential to Osano's ongoing operations, and we've continued to extend its use with new capabilities we've recently added that also require a data lineage verification function.