Common Database

# Common Database aka: #### Data of the Commons (DotC) #### Distributed Common Data Network (DCDN) #### Distributed Open Collaborative Platform (DOCP)   - A missing data sharing structure to improve the common good worldwide. - A huge common database to store all words and knowledge on a resilient, fast, scalable, open-source and distributed p2p system. - Be the owner of your data, and decide who has access to what about you. Stop feeding the business of data corporations and companies selling what they know about you. - An optimal source of types of stuff, verbs-actions, agents public data, location names, units and other kinds of data for any common good platform or software. - A network to store and share any type of information or data with a quality auto-regulation system performed by all the peers, avoiding external censorships. - An extensible modular multi-tool to search for or contribute to common useful information, to join or create teams and projects, to communicate with people, to donate, exchange, buy, sell or rent resources of any kind, and account them as a real common [REA](/rea-chain.md) stream of economic events. - The rise of artificial common inteligence distributed platform, built by the people for the people.   ## Index: - Introduction - What? - Why? - How?: - Governance - Technically - Layers: - The backend layer - The frontend layer - Human Teams - Milestones   ## Intro: See the presentation slides at https://commondb.net/slides.html The IntegralDevs Team, whose goals are explained [here](https://gitlab.com/integraldevs/integral-developing), is developing a multi-device App with a simple frontend to browse, build and manage a resilient p2p scalable combination of distributed databases to store all Common Data to share between platforms and between agents in a conscious and informed way. This App is built totally modular, very simple when starting but extensible with plug-ins added by the projects you join, giving tools and options as features of the basic app, depending on your interests and the data of your profile. This software wants to became a super-improved version (distributed, fast and highly scalable) of the Open Collaborative Platform ([OCP](https://wiki.p2pfoundation.net/Open_Collaborative_Platform)), totally refactored with the best open-source technologies in the NoSQL databases field and the fastest programming languages to allow the infinite growing of the data size, and being capable of delivering fast queries (high availability). The first basic core function of the App is to provide a secure and private digital identity to agents. This agent identification is a process with various levels: it starts by confirming an email address and/or a phone and creating a cryptographic digital signature (like PGP), but more means of identification can be added to allow the use of certain features (e.g. to start a complete 'health' profile a physical biologic sample analysis of the person can be provided, to define some biologic facts and actual state of the agent's health). Once properly identified the person or agent, the App frontend can be used to build a profile (with public and private parts) and browse the network for things like: - finding and joining other projects, - buy products or project's shares, - make a donation in a crowdfunding campaign, - contribute to the network by adding useful data, knowledge, code or resources, - communicate with other agents or groups, - share your files with friends or with everybody, - build and map your online project to offer your products or services, Different projects you join can then include more tools and features available in the app, like for example the use or management of tasks, resources, workflows, processes or plans in a project, or legal working invoicing or contracting features, or governance tools for debates, polls and voting, and many others. This app can then be an optimal future for the already mentioned OCP, CoopFunding and FairCoop ecosystem tools such as UseFaircoin, GetFaircoin and FairMarket websites, to start with, but also for many others to serve the Common Good in general. With a proper development of the teams involved and the infinite possibilities of the tools modules, it can became one day a more resilient and multi-version Wikipedia (or Wikidata) for knowledge, a better open-source alternative to Telegram or WhatsApp for communication, a common file sharing system, a private but resilent vault for your personal heath data, a common AI and knowledge source which cannot be privatized nor transformed into a capitalistic profit business.   ## What? Each app instance is internally a node of various types of p2p databases at the same time (a polyglot system), but integrated as a single and easy to use interface. Each agent node can choose which data wants to store locally, depending on the device being used (drive space, memory) and the bandwidth, and depending on her/his interests. The app helps the network by suggesting stuff (data types) that is needing more nodes in order to be enough fast and resilient for the received demand of data by the whole network. This software creates a mesh of nodes that can also store personal and private data (individuals or groups) with various layers of encryption [*](#note1). The tool can be used to improve many sides of people's life, by adding functionalities in the basic app as plugins, to manage personal or collective data about economic facts, about health, education, comunication, transport, manufacture, tasks, etc   ## Why? The world needs resilient structures to share data as much as possible, to rise human consciousness, improving human awareness, knowledge and loving capacity, to avoid political censorships, to avoid feeding for free the data business of capitalistic corporations, and also to avoid many problems related the repetition of similar data in many databases and the disconnection of data silos when their API’s didn’t talk to each other perfectly in order to update data from various sources. To create a common good worldwide we can create all together a Common DB of common data, like the words of all languages: many verbs or arts (which can produce facts), many names given to nature parts, beings, places, inventions, units, types of stuff, concepts... There’s an infinite amount of data which can be shared as common if we want to improve as humans. The actual solution to this is the nightmare of API’s updating and replicating data from platform to platform, which is also very needed, and we'll include common vocabularies and protocols (like ValueFlows, ActivityPub, RDF, etc.). The development we are presenting here focuses instead in the building of a well-distributed database that can became the main source of many common data for many other platforms. Of course, we will implement the used vocabs and protocols in the API in order to import or migrate data to or from this common database.   ## How? ### Governance: To sustain and improve Quality, help the rise of the Common Good worldwide, facilitate the software for projects working for ethical Values and human rights, and avoid misuses or abuses of the system, not only is required plenty of open participative ways of decition (the assemblies of the main cooperative projects using the platform) but also is needed that the way the code is produced facilitates the good uses and difficults the bad uses. The politics to improve the ethical Values must be inserted in the code in order for the tool to be fair on its own. This data network is designed to become one day self-governed (like the www), based on the assumption that there are more people of goodwill than people with bad intentions. The nodes themselves choose the data they are interested to host, use and support, so the malicious or fake data should have less network support (fewer data resiliency) and the useful data became popular and resilient. One node or agent, even if it has a lot of resources (money, space, speed, bandwidth, etc.) should not have the voice of many nodes. All nodes have the same rights and political power in the network (one voice), regardless of their capabilities. Money should not be the cause of having more visibility or power, only the people's support give popularity and resilience to the data. The first period during the development of the core tool (before the network is ready to be self-governed), in order to protect the Values and common good goals, is needed a strong team of humans with clear healthy goals and resilient internal harmony, more or less protected from 'bad' influences, trolls and corporative capitalistic pressures willing to bias the project to their interests or willing to boycott the development of this common good meta-tool (and keep their status-quo). The IntegralDevs team, whose goals are detailed [here](https://gitlab.com/integraldevs/integral-developing), can try to hold that role of keeping the initial values on the political key development decisions, defending the healthy common good goals originally stated, from the core of the tool (the developers’ side). The tool, thanks to its encryption and authentication system can be used one day safely as a voting tool or a distributed decision-making system to improve governance and gather peoples feedback, not only for the governance of the whole database network but also for the governance of every project or assembly managed there.   ### Technically: Whether Blockchain technology could be optimal to store non-mutable data like economic transactions, it is not for an agile management of other types of data, for that purpose we reckon is better a distributed NoSQL database, like a graph db for relations, a key-value store db for words, types and short texts, a key-doc store for long texts or code, and a p2p file-sharing system for files.. The proposed App, in its first version, should allow to use the best db technology for the type of data being saved: - some of the economic events can be stored in a flexible blockchain (like [FairChains](https://www.fairkom.eu/en/fairchains), [Holochain](https://holochain.org) or [Chromia](https://chromia.com)) used as a "[REA chain](/rea-chain.md)", once there is consensus between the involved parts in an exchange. - other economic events like transactions of the many cryptocurrencies around (starting by FairCoin) are already stored by their blockchains - the words of the languages and their translations (if exist), the types of resources, types of skills or arts, the names of locations and areas, the units and the unit ratios, and many other types of data can be better stored in a key-value distributed JSON datastore, like the core Ddb of the app. - the images, the music, the videos and all heavy data is better stored and shared like in a p2p common filesystem (or torrent style). - the long texts, the code libs, templates, scripts, etc, can be stored better in a key-doc Ddb To achieve this we need to join forces in harmonic dev teams and some key ingredients: - a super scalable multi DB system that is able to grow indefinitely by adding people’s nodes ([in a p2p style](http://wiki.p2pfoundation.net/P2P_Networks)), - an integral data model that makes easy to manage the grow of the types of data and the relation between them ([proposed here](/integral-data.md)), with... - a set of widely consensus Common Resource Type Trees [**](#note2), where representative agents of the different sectors of activity take care of organizing and curate the needed and useful resource type branches and sub-types to use in that sector (alimentation, communication, etc.) which is an organic, always mutable process and this data partitioning in sectors and keywords allows us to develop... - an engine to analyse the network, update distributed indexes and balance the load by asking for more nodes helping in critical data sectors and all this back-end needs to be built with... - a fast enough functional language like Erlang-Elixir, JavaScript, but also Rust and others, and all this needs to be easily managed from... - an easy-to-use Graphical User Interface, that simplifies the ‘brow-serving’ and the configuration options. And that allows... - a myriad of modules to add functionalities, pages and options to the basic App, in order to do a lot of things, using the best open-source code available worldwide. The data you browse or use from the Ddb, cached in your device, is automatically served to other nodes by default, except private data or private browsing data, which defaults to no sharing but the agent can choose trusted peers to spread his encrypted private data only to their devices. <a name="note1"></a>(*) The encryption of the agent’s personal data has different levels, like layers, depending on the relation between the browsing agent and the data creator agent. The app should offer from easy sets of privacy config options up to a more granular configuration of the privacy settings. For example the personal information about Health can be shared only with family, friends or few chosen organizations nodes, but in case of emergency any accredited doctor can gain access to it. Some layers of encryption for the parts of the agent’s data: - a first layer to hide info to non-logged users - a second layer that only friends of the agent can access through (can be n levels), - a layer of technical data that only the app itself manages, - a set of layers for strictly personal data: can be defined a special encryption layer for medical data with different rules than a layer with profile data, education data, economic data, etc. <a name="note2"></a>(**) The most of the Types of things can be common (and perhaps some particular subtypes used by a single project don’t need to be common).   ## Layers: ### The backend layer: This meta-project is based on a concept of full flexibility and modularity totally Agent-centred, therefore what we call 'the back-end' is basically the database because nearly all the logic and code is also inserted as db records. To render a page the system gathers some objects from the db, not only the requested information but also the HTML template, js, css or any code snippet or lib needed to show the data. This way, to improve some code or fix a bug, it's only needed to edit a db record and all uses of it will change its behaviour globally. So, the code required to run any logic is gathered from the db, and then the only function of the back-end layer is the databases management and the interpretation of the code coming from the ddb. The network will have **nodes** and **super-nodes**, being the super-nodes just like regular nodes but offering 24/7 service and dedicated space, memory and bandwidth (so using the actual servers around as helpers of the network). The code for the super-nodes can be initially different from the one for regular nodes (an extension of it), but the goal is to put all logic of extensions as db records, and a regular node can became a super-node just upgrading its local app with the functionalities available if the node joins the super-nodes group in the network (a moderated process). The main parts of the backend code are: - **connectors** from the distributed databases used, depending on the type of data, to the frontend GUI. - **encryption mechanisms** to ensure: - privacy of personal data in various layers, and - to guarantee a private login authentification in the system. - **interpreters of the code** received as db query results: at least the basic languages used to run the basic functionalities of the app should be interpreted at the core level (rust, erlang, python, etc) - **engine to regulate the load balance** between nodes, improve resilence and availability of the data (depending on its popularity): - should warn nodes (in the frontend) about required help hosting crucial popular data and - let them choose sectors of data to host from their interests. - **unified indexing engine** joining the indexes of every used database system together, to distribute a self-building index of all the stuff in the ddb's in order to deliver: - fast searches of text data (full-text search) and - faster retrieval of precisely referenced items (by key or uid) - **distributed caching system** multi-level, splitting the cache in levels: - from most common resources (highly available) to more specialized resources (less distributed) The logic of this engines can be recorded as db 'rows' (and run them in the node once retrieved from the db) in order to update them globally when any bug fix or improvement is achieved after launch. If this is too hard to achieve from start, we can start with that engines being part of the hardcoded initial code and in another release make that engines easily updatable as db records. The goal is to make the basis generic enough to avoid needing updates of the app and rely as much as possible on '**code from the db**' techniques (like did the Maid-SAFE network).   ### The frontend layer: The GUI can be developed mainly as a multiplatform app, for mobiles and computers. We can develop also a web version if we find good ways to identify users safely enough and trust at least some browsers. The main parts of the frontend are: - authentication of the user agent: the proposal is a GPG alike auth system, and are needed: - ways to recover access if an agent loses her/his credentials, with a progressive KYC process, at the pace of the user, in order to have alternative access recovery ways (alternate emails, phone number, address, etc). - also requires ways to cancel active accesses to personal data if the access is stolen or compromised in some way - profile building and privacy settings: the UI to add data of the agent (individual or collective) and: - define what visibility and sharing settings affect each part of the profile data. - if there are not yet trusted peers to spread the profile data, the UI should allow to set some, starting to define the first social relations in order for the db system to work distributed and have your data there even if you are not online. - human relations and communication: the user relations are the main source of permissions and resilence of the user's data for the whole system - joining process of the user agent with projects of her/his interest and communication means with the coordinators of such projects - communication means for any group, both for internal private conversations and for open communications seen by everyone - private personal communication with other peers, offering options to delete, expire or edit messages - economic facts: this part of the UI has some sub-parts: - a generic system to represent 'exchanges': donations and received gifts, buys and sells, acquired shares, payments and bills, etc. - the currency wallets or accounts of the agent's coins or shares (it can manage internal real wallets or just show defined accounts activity by pulling-pushing data from-to the proper source) - the UIs to use the active payment gateways of the projects you interact with - a multi totals and resources system where the agent can track and make accountings of any type of resource or stock - a generic system to represent 'tasks': - from personal todos to assigned tasks when participating in projects processes or workflows, - remunerated or exchangeable work (normally related an exchange), - search for tasks related your skills and search people with certain skills (if they stated that info as publically known) - this two economic GUIs are mostly readonly: to create new economic facts is needed special app-tools that are available or not depending on the user relations, the main source of permissions. For example: - if the user agent is an active participant in a project like FairMarket or similar, then the UI tools to create a shop and define products becomes available. - if the user agent wants to join a cooperative, buying shares, the payment gateways options defined by the project became usable by the user (and manageable by the project's coordinators). - if the user demonstrates ownership of certain accounts (e.g. cryptocurrencies), the account managing options should appear in the economic UI (or links to the account manager tool if is a fiat or a not integrated yet currency) - if the user participates in projects with activated tasks and workflows, the needed tools to define the related tasks commitments and events appear available for regular users (or to define plans and process setups if the user is a coordinator of the project) - browserver: general search discovery of knowledge and network resources - the searches work using agent's relations too, searching first in known sources and after sending queries to any part of the network, but not floading all nodes with search requests. - The knowledge already present in the system (one day all words) and a common set of data 'sectors' or common resource types trees*, allows us to send queries for index data or real data only to the nodes known to be hosting that type of keyword data. - navigation of web pages stored in the db - sharing settings, to define how to share the cached browser data from the navigation - the UI should allow to contribute to the network by adding knowledge, words and definitions, like in a wikipedia style but decentralised - navigation of normal external web pages from the internet (served by servers) can be added if needed (maybe useful to crawl data into the common db) but there are plenty of existent browsers for this. Many more functionalities can be added by means of adding plugins to the app, available (or auto-installed) depending on the relations of the agent with different projects and her/his interests or working fields.   ## Human Teams: This project requires at least two initial sub-teams, to start in parallel but under the same phylosophy of the IntegralDevs main project: - The code developers team and - the comunication-coordination team (or facilitators). The coordination team should help from start to: - comunicate the project worldwide - interface between the core devs and the software users or potential users - manage a crowdfunding campaign in phases - engage patrons, donors and partners of the common good scene - find and try to involve healthy and skilled people for building the first needed Teams: - Security team (testing and improving privacy, stability, etc) - Backend team (improving data CRUD modules for the various ddb's used) - Frontend team (improving usability and graphic design as modular skins) - Communication team (improving the project's social connections, translations, etc) - Common Taxonomy team and sub-teams (improving-curating the basic main branches needed but mainly improving coordination of 'sectors of activity' in sub-teams to build consensus about common types or data useful in their sectors).   ## Milestones: ### Milestone #1: The first big milestone in the roadmap will be a first usable beta version of the minimum layer functionalities to add app plugins or components to start, at least: - authenticating users and allow building of simple agent profiles with location data, so... - mapping agents that wanted to share their location, organised in map layers to reflect types of agents, etc - filling up the db with useful common data and words (verbs, types, agents, units, unit-ratios, location and region names, definitions of concepts, etc) - joining projects (with moderated access or autojoin link), - exchanging resources (money to start with) for project's shares when they are needed for membership, that means... - having active payment gateways (defined by the projects), - tracking of exchange transfers, commitments and events, - showing transfers status and allowing to edit payment status if user is coordinator of the project and the gateway is manual, - some payment gateways can be fully automated, like cryptocurrencies and, up to some degree, also the credit-cards. - communicate with the project coordinators (or candidates and members if the user is a coordinator), at least via chat - settings of the users data privacy, visibility and sharing scopes - settings to allow your node to host data of your interests, of your friends and keen organisations, etc (depending on the used device capabilities) This first version can allow then the migration of: - the whole actual OCP membership system, with the custom forms, custom payment gateways of custom shares, etc - the UseFaircoin and other simple mapping solutions for projects, not only in the common good ecosystem, but also mapping company providers, institutions, or any type of organization (in map layers). Technically this first beta needs the node's backend functionality to: - authenticate the user cryptographically - connect and use shared data from the main core JSON distributed DB (for light mutable objects data, like profiles, words, etc) - connect and use shared data from the GraphDB or other 'relations' storage system (agent-to-agent, agent-to-resource, verb-to-verb, verb-to-name, etc) - connect and use the shared files system, to retrieve at least various common code libs and to save also the agent's logo or avatar images in a distributed way - decrypt query responses with the required keys depending on the user agent identity and relations (permissions) and the levels of privacy applied to the data when recorded - restrict the disk space, memory and bandwith usage depending on the used device capabilities - load balance of partitioned data amongst peers by type and popularity: very used data types should be in many nodes to improve availability and db response speed - send notifications to users (at least via email)   ### Milestone #2: In the second iteration or core code release we can include (once the first beta is tested and fixed enough to become 'stable') app plugins to: - allow projects to define other assets 'for sale' apart from membership shares, like cryptocurrencies directly or products of any kind - allow projects to define custom crowdfunding campaigns, with custom payment gateways, self-management tools, etc - communicate with other single users in a safe encrypted way using the app - communicate in group's private and public chats, (allowing the organisation and debates of every sectorial group, broad or small, in order for them to consensuate common sets of resource types, skill types, units and other stuff used in that sector of activity). This second milestone can allow then the migration of: - the GetFaircoin website functionality and other simple exchanges solutions that mainly sell cryptocurrencies - the CoopFunding website functionality - the whole FairMarket and other online multi-shop systems can be ported to this platform - the communication tools like Signal, Telegram or Whatsapp can start to be less used to coordinate people - the social networks behaviours can be imitated and ported in this opensource sytem to stop using corporate solutions that make business with our data   ### Milestone #3: In a third iteration or release we can include plugins to: - allow agents to store their health data facts and history, individually and via health facilitators and health organisations, with special access rules adapted to health data. - allow projects to create survey polls and allow agents to vote them or submit opinions, and other tools to improve governance of any common good project - allow projectcs to define plans and tasks of productive workflows or transformation steps in manufacture, input resources and outcome products or services, reusable process recipes, etc - ...