OpenEMITValue_Proposition

"Write Once, Stream Anywhere"

A simple, secure, fast and portable
OpenAPI Standard and Integration Framework
for events, ditributed logs, object stores, FaaS and blockchain (DLT).

// Coded in Rust

---

The Problem

Problem 1: Lack of Interoperability/Portability

Almost all streaming/ledger/storage technology have a proprietary, non-standardized API. (SQL DBs being the notable exception.)

This leads to:

• Costly and lengthy rewrites to migrate applications across clouds, stacks, and APIs
• "Lock-in" to a particular offering/vendor and license fee structure
• "Analysis Paralysis" around selection and long-term commitment
• Wealth of choices including Kafka, Pulsar, Redpanda, AWS S3, Cassandra, Redis, FoundationDB, Couchbase, Riak, RockDB, CockRoachDB, YugabyteDB, AWS Lambda, Google Cloud Functions, IBM Cloud Functions, Azure Functions, OpenFaaS, OpenWhisk, Ethereum, Cardano, Komodo, HyperLedger, Raft consensus, and over 100 more…
  
  

  Note: Any database that can support a Hashmap (Key/Value), any messaging that can transport a Key/Value tuple, any distributed database that does both already, any network messaging, any ditributed ledger technology, any Serverless and any Pub/Sub service can be used to construct a log/ledger.

  Issues of security, encryption, safe storage, consensus, consistency, resilience, recovery, distribution, and availability are solved to varying degrees by all these products, and the SLAs they provide with respects to these concerns can be harvested for "Best-of-Breed" or "Cost/Value" stacks.

  Core services like Pub/Sub can be optimized for secure, fast and deterministic invocation speed in Rust. This is the OpenLedger Value Proposition.

Problem 2: Immutability

Ledgers are immutable by design. So how do we deal with a privacy order? How do you deal with Personally Identifiable Information (PII) in the ledger?

Problem 3: Complexity

Most technology in this arena has complex APIs, and require "high value" developers and architects.

---

The Solution

We can use combinations of existing technologies to produce Log/Ledgers with Pub/Sub Topics of varying broadcast scopes and encrypted immutability.

Ledgers require:

• Simple and Safe Storage and Retrieval of Entries
• Reliable Transmission of Entries across various Scopes (Process, Host, Cluster…)
• Support for Pub/Sub (implicit invocation)
• Blazing Performance
• Security-Driven Development

More importantly, logs/ledgers do not require different processing on a stream of data based upon the type of foundational storage, communication, consensus, or pub/sub offerings employed. This concept is paramount.

Write Once, Stream Anywhere.

Standardize with a Domain-Specific API

"Embarassingly Simple Model"

1. Ledgers exist within a (broadcast) Scope
2. Scopes to choose from are Process, Host, Cluster, Datacenter, Cloud, Enterprise, or Federation.
3. Ledgers host Topics (and SubTopics). Ledger Entries are published to one or more Topic(s), where Agents "filter and forward" to Subscribers.

Note: Entries are an Open Format. (Developer Definable)

Event-Driven Support

If you choose Events as your entry format (and there is good cause to do so), a little more API can go a long way to maturing you operations.

• Event
  • Timestamp{} // Cron Key and causal consistency support
  • SeqID{} // Order Key for integrity and optimal ordering criteria
  • enum LifeCycle { Submitted, Committed, Published }
  • Topics[] // set of Topics selected for Publication
  • MetaData{} // aka Head, Header, Manifest,…
  • PrivateData{} // Stored in Secrets Repo with anonymity capability
  • EventData{} // the "event"
• my_event = Event::new(meta_data, private_data, event_data)

Note: By separation of concerns (three data sections), we can store all Privacy Data that may need to be esponged from the system record in a separate repository (LDAP, Secrets, etc.). This allows the capability of anonomyzing the person or company from a ledger perspective. You can leave the ledger in tact, and replace the instance of the entity Privacy Repository with "John Doe 007" who worked for Company 9".

Standardized anonymity for all ledgers!

MetaData contains Process/Stream level and performance data used by OpenLedger and available to the developer should they care.

PrivateData gets stored in a Secure "Entity" Model termed the Privacy Repository (many offerings).

EventData is just that, a schema/structure of developer choice to be processed by one or more subscribers. You may choose to adopt schema and versioning management tooling of your choice. (aka Schema Registries).