owned this note
owned this note
Published
Linked with GitHub
# Couchbase Transactions Design Document
Copyright (c) 2020 Couchbase, Inc.
Transactions add the ability to update multiple documents together in an ACID-compliant manner.
They are implemented with a protocol that is relatively simple from a high-level, though the details of robust error handling presents many complexities.
This document outlines exactly how these transactions work at a level of detail sufficient to be able to create an implementation in a new target language.
It is intended to be a living document, one that will evolve over time in a suitably versioned format.
# Additional documents
This is not a good document for building a broad understanding of how transactions work, this is intended to be an implementation guide and a point of reference.
The [Couchbase Transactions Index document](https://docs.google.com/document/d/1aIH7M9FXsBozh0mS_kaZ8fj3-wNWF5aH3TSWY6RMTNw) (limited to Couchbase employees) links to all Couchbase transactions materiel, including higher-level presentations that are more suitable for gaining a quick understanding.
The [Couchbase Transactions API](https://hackmd.io/cHIcXWZSQOi23qpJJAa8OQ) defines the API.
The [Couchbase Transactions Stages](https://hackmd.io/Eaf20XhtRhi8aGEn_xIH8A) defines the 'meat' of the algorithm, in implementation-level detail.
Due to hackmd document size limitations, additional documents [Couchbase Transactions ATR Stages](https://hackmd.io/wE1KU1RKQtONGR8tXbpA1Q), [Couchbase Transactions Cleanup](/w5hsXGwYTRmv2IxBfAoORw), [Couchbase Transactions Query Integration](/eMPHhhd9SJqjO3s9dT9p7A) and [Couchbase Transactions Stages Protocol 1.0](https://hackmd.io/g3PWQrHHRiew0nGF6iFgiA) have been split out from it.
There is also a [repo](https://github.com/couchbaselabs/couchbase-transactions-specs) where all the specs are periodically published to. It is not guaranteed to be up-to-date and hackmd is the source of truth.
# The Algorithm (Abbreviated Edition)
This is a whirlwind tour of how transactions work. For details, see the rest of this document.
* Txns have a single unique transaction ID (UUID). This means there’s no ordering of txns, which would require an expensive single-point-of-failure to distribute global ordered ids.
* All writes are durable, and rely on the new durability semantics in 6.5.
* Mutations inside a txn are staged in the doc’s xattrs during the prepare phase, until the commit phase, so we never show dirty data. This provides our read committed isolation level (actually we’re a little higher, with MAV - see later).
* There’s multiple Active Transaction Record (ATR) docs - at least one per vbucket. Each active txn has an entry in one of these docs. ATRs give a) a single point of truth for each active txn’s commit status and b) a means to find failed txns.
* Each mutated doc in the txn gets a link to its ATR during the prepare phase.
* The commit phase flips the txn’s status to Committed (in the active txns record). Rollback is no longer possible. Then it moves all doc’s write_intents into the docs, and clears the doc's xattrs txn metadata. Finally it sets the ATR status to Completed.
* Transactional reads check for staged transactional metadata, and if it’s present, read the corresponding ATR to see if status is now Committed. This gives us something of an atomic commit, and provides our true isolation level of Monotonic Atomic View, or MAV.
* However, non-transactional reads (KV, N1QL) will show an eventually consistent commit.
* There is a distributed cleanup algo that tries to find and recover any failed transactions from, for instance, an app crash.
* Two txns trying to update same doc is detected using a mix of CAS & checking for staged write_intent. The discoverer is aborted.
* (Aborted txns are normal and expected. The app logic is supplied in a callback that will be called multiple times if required.)
# Dependencies
Couchbase Server 6.5+ is required for Durability, which is essential for ACID transactions.
Given that durability is needed, we also depend on SDK 3.x. There will not be a SDK 2.x version of transactions, applications will need to port to SDK 3.x.
Protocol version 1.0 requires Couchbase Server 6.5.
Protocol version 2.0 requires Couchbase Server 6.6.
Queries require Couchbase Server 7.0.
NTP on the cluster is strongly recommended for timestamp consistency.
# Durability
All writes in the standard flow are done with the durability provided by Couchbase Server 6.5. This ensures that once a write has been confirmed as written, it will remain durable.
This gives consistency guarantees that are considered essential for ACID transactions. So, transactions are only supported on Couchbase Server 6.5.
Also, transactions depends on small features added in 6.5, including “$document.revid” support.
The default durability level is MAJORITY, which will wait until each write is stored in-memory to a majority of replicas. This provides a good balance of safety and performance. Applications that require absolute resistance to all forms of node and hardware failure may consider upping the durability level to wait for writes to also be written to storage on all replicas. This can be configured using TransactionConfigBuilder or PerTransactionConfigBuilder.
# Metadata
## Document Metadata
This is the metadata that will be stored in a document’s xattrs to stage a mutation:
### Protocol 2 Version
```json
{
"xattrs": {
"txn": {
"id": {
// Each transaction has a single UUID
"txn": "6a919480-d9fd-4d0d-9074-2506b3a210d1",
// A transaction can have multiple attempts, each with its own UUID
"atmpt": "91a8e712-b407-497d-a0de-97835cf0c7cc",
// Added in ExtThreadSafety: each operation has a UUID
"op": "363d8521-e032-4d1a-b30f-6f196ddf8fc6"
},
"atr": {
// Each txn attempt gets an entry in an ATR
// The ID of the ATR document
"id": "atr-324-#37",
// The name of the bucket the ATR exists on
"bkt": "default",
// The name of the scope the ATR exists on
"scp": "_default",
// The name of the collection the ATR exists on
"coll": "_default"
},
"op": {
// The staged mutation type, "insert" | "replace" | "remove"
"type": "replace",
// The full post-transaction body of the document, as JSON
// This field is not-present for remove operations
"stgd": {
"val": 2
},
// Added in ExtBinarySupport, and used instead of the JSON "stgd"
// to store binary data. Only one of "stgd" and "bin" will be
// present.
"bin": "cb-content-base64-encoded:Zm9v",
// The CRC32 of the staged change, e.g. the result of the '${Mutation.value_crc32c}' MutateInMacro
"crc32": "SOMECRC32"
},
// Added in ExtBinarySupport - see "Evolving the metadata" section
"aux": {
// The staged user flags that the document will use post-transaction.
"uf": 50331648
},
"restore": {
// Used for backup/restore {BACKUP-FIELDS}.
// The document’s CAS, before any transaction changes
"CAS": "0x15a9edcf58d60000",
// The document’s expiry time, pre-transaction
"exptime": 0,
// The document’s revid, pre-transaction
"revid": "80"
},
// Next field is optional
"fc":{
// The forwards-compatibility map
// The syntax is too dense and concise to go into here, see later
"CL_E":[{"p":"2.0","b":"f"}]
}
}
}
}
```
#### Evolving the metadata
In some implementations it's not desirable to fetch the entire "txn" xattr, as processing it will deserialize the full object including the staged data. This only needs to be stored as a string/bytes in most cases, so the deserialization is wasted effort.
Our goals are to avoid exceeding the 16 Sub-Document spec limit, and to minimise the number of fetched fields for performance. While avoiding that deserialization issue.
So, the recommended approach is that each top-level field ("id", "atr" etc.) is fetched as a Sub-Document spec, as is each individual field in "op". And "$document" and the document's body must be fetched as a separate spec. At present this is 9 specs. If an implementation has a way around the deserialization issue above, it can feel free to optimise further e.g. by fetching the full "txn" xattr.
When extending the metadata in the future we will not add additional fields into "op", and will aim to minimise adding additional top-level fields. Perhaps one more generic one will be added, that will include all future new fields.
ExtBinarySupport added this new generic top-level field, "txn.aux".
### Protocol 1 Version
```json
{
"xattrs": {
"txn": {
"id": {
// Each transaction has a single UUID
"txn": "6a919480-d9fd-4d0d-9074-2506b3a210d1",
// A transaction can have multiple attempts, each with its own UUID
"atmpt": "91a8e712-b407-497d-a0de-97835cf0c7cc"
},
"atr": {
// Each txn attempt gets an entry in an ATR
// The ID of the ATR document
"id": "atr-324-#37",
// The name of the bucket the ATR exists on
"bkt": "default",
// The name of the scope.collection the ATR exists on
"coll": "_default._default"
},
"op": {
// The staged mutation type
"type": "replace",
// The full post-transaction body of the document, as JSON
"stgd": {
"val": 2
}
},
"restore": {
// Used for backup/restore {BACKUP-FIELDS}.
// The document’s CAS, before any transaction changes
"CAS": "0x15a9edcf58d60000",
// The document’s expiry time, pre-transaction
"exptime": 0,
// The document’s revid, pre-transaction
"revid": "80"
}
}
}
}
```
## Active Transaction Record Metadata
There are multiple Active Transaction Record (ATR) documents (1,024 at minimum). A transaction can have multiple attempts and each attempt gets an entry in one of the ATRs.
Each ATR stores this metadata in an xattr named `attempts`. Xattrs are used to reduce the chance that an ATR will be picked up by a query.
`attempts` is simply a map of attempt ID to the ATR entry for that attempt.
Not all of the fields below will exist at all times. In particular, the “ins”, “rem” and “rep” fields indicating the mutated documents are only written when the entry’s status is changed to COMMITTED or ABORTED. The rest of this document will specify when each field is written.
```json
"attempts": {
// Attempt ID
"ef1e25ce-fa7e-4bcc-83a3-0160b1321385": {
// Transaction ID - optional, only available if
// implementation supports [ExtTransactionId]
"tid": "febcecce-2386-4277-9e9c-66a7c367dbe4",
// How long until the transaction (not attempt) expires, in millis
"exp": 3000,
// The state - see Attempt States
"st": "ROLLED_BACK",
// The "${Mutation.CAS}" 'timestamp' of:
"tst": "0x00005f07b4250016", // the "PENDING" write
"tsc": "0x00008f09b4250016", // any "COMMIT" write
"tsco": "0x00008f09b4250016", // any "COMPLETED" write
"tsrs": "0x00008f09b4250016", // any "ABORT" write
"tsrc": "0x0000d909b4250016", // any "ROLLED_BACK" write
// An array of any documents inserted in this attempt
"ins": [],
// An array of any documents replaced in this attempt
"rep": [],
// An array of any documents removed in this attempt
"rem": [
{
// The bucket, scope, and collection of the document
"scp": "_default",
"col": "_default",
"bkt": "default",
// The document's ID
"id": "syncWriteInProgressErrorDuringCommit_0"
}
],
// This next field is optional
"fc":{
// The forwards-compatibility map
// The syntax is too dense and concise to go into here, see later
"CL_E":[{"p":"2.0","b":"f"}]
},
// Used to ensure cleanup and the main algo cannot conflict
"p": 0,
// Added in ExtStoreDurability: Durability level used for this transaction.
"d":"m"
},
// ...
}
```
# Operations
Operations are `ctx.get()`, `ctx.replace()`, `ctx.insert()`, `ctx.remove()`, `ctx.query()`, `ctx.commit()` or `ctx.rollback()` call, in their entirety.
# Stages
Operations can consist of multiple stages which is generally a KV read or write and include:
* The internal logic for a `ctx.replace()`, `ctx.insert()` or `ctx.remove()` call.
* A single ATR operation, such as changing its state to Committed, or creating an ATR entry.
* Unstaging or rolling back a single document.
The exact logic required in each stage is defined later in [Stages in Detail](#Stages-in-Detail).
# Error Handling
## Error Classes
All errors either raised by the transaction library, by the application’s lambda, or by the underlying SDK, will be classified into broad error classes. The implementation will only do conditional logic upon these error classes. E.g. there will usually be no special-case handling for a particular SDK exception (with some exceptions for exceptions raised internally by the transactions library which will be detailed elsewhere).
This allows two things:
* We can be certain that all implementations work the same way.
* This document does not have to make repeated references to "exceptions A, B and C", it can simply refer to an error class.
* The tests can inject these error classes at all possible points without having to worry about whether every possible exception has been covered.
The error classes are an internal concept and should not be exposed to users.
The error classes are:
### FAIL_OTHER
The majority of exceptions fall into this category. If an exception does not match one of the classes below, it belongs to this.
Generally, the behaviour will be:
Pre-commit: rollback this attempt and then fail the transaction with a TransactionFailed.
Post-commit or app-rollback: continue trying to commit/rollback.
### FAIL_TRANSIENT
These represent unambigous errors (e.g. they definitely did not create a mutation), that are likely to be transient. E.g. retrying the transaction or operation may succeed. They include write-write conflicts, and some server errors.
Note, transient doesn’t mean it can be resolved in the very near future. It could require the server to be recovered manually to a good state first, e.g. it could take hours. However, that is unknown to the transactions implementation.
Generally, the behaviour will be:
Pre-commit: perform RETRY logic.
Post-commit or app-rollback: continue trying to commit/rollback.
Exceptions classified as this are:
* UnambiguousTimeoutException (with SDK3’s BestEffortRetryStrategy, many SDK2 errors such as TemporaryFailure will now surface as UnambiguousTimeoutException instead.)
* For testing purposes, @Stability.Internal exception TestFailTransient
### FAIL_DOC_NOT_FOUND
The document was not found. This will usually be treated as a FAIL_OTHER, but it is operation-dependent.
### FAIL_DOC_ALREADY_EXISTS
The document already existed. This will usually be treated as a FAIL_OTHER, but it is operation-dependent.
### FAIL_PATH_NOT_FOUND
The document was found, but a required path inside it was not. The handling of this is very operation-dependent.
### FAIL_PATH_ALREADY_EXISTS
A path that was not expected to exist, did. The handling of this is very operation-dependent.
### FAIL_WRITE_WRITE_CONFLICT
Trying to write a document that’s already in a transaction.
### FAIL_CAS_MISMATCH
A CAS mismatch occurred. This will generally be treated as a FAIL_WRITE_WRITE_CONFLICT, though exceptions exist (e.g. for ambiguity resolution).
### FAIL_HARD
Ths class emulates a hard application crash. It is only raised by the tests, but it is a crucial element of testing that implementations can handle this scenario, so it is mandatory for implementations to handle.
The behaviour is:
Stop whatever operation is in progress.
No further mutations will be done.
Do not rollback or retry the transaction.
Do create a valid attempt in the final result’s attempt field, and a cleanup request.
Raise a TranscationFailed containing this result.
The idea is that the transaction is left in exactly the state it was in at the point of the error, and a valid attempts field is returned so that the tests can verify everything is as expected.
All of this behaviour can be handled with [`ErrorWrapper`](#ErrorWrapper).
### FAIL_AMBIGUOUS
This class represents mutation operations where it’s unclear if they succeeded or not.
It is one of the hardest errors to handle.
The class includes AmbiguousTimeoutException, plus any platform-specific exceptions that fit into this category (e.g. on Java RequestCanceledException).
### FAIL_EXPIRY
The transaction expired during an operation.
### FAIL_ATR_FULL
An operation writing to the ATR failed as it was too full.
## Mapping
This shows what exceptions map to what error classes, and whether those exceptions are from the SDK or internal to transactions.
|Exception|Type|Error Class|
|----|-----|---|
|DocumentAlreadyInTransaction|Internal|FAIL_WRITE_WRITE_CONFLICT|
|DocumentNotFoundException|SDK|FAIL_DOC_NOT_FOUND|
|DocumentExistsException|SDK|FAIL_DOC_ALREADY_EXISTS|
|PathExistsException|SDK|FAIL_PATH_ALREADY_EXISTS|
|PathNotFoundException|SDK|FAIL_PATH_NOT_FOUND|
|CasMismatchException|SDK|FAIL_CAS_MISMATCH|
|UnambiguousTimeoutException|SDK|FAIL_TRANSIENT|
|TestFailTransient|Internal|FAIL_TRANSIENT|
|DurabilityAmbiguousException|SDK|FAIL_AMBIGUOUS|
|AmbiguousTimeoutException|SDK|FAIL_AMBIGUOUS|
|TestFailAmbiguous|Internal|FAIL_AMBIGUOUS|
|TestFailHard|Internal|FAIL_HARD|
|AttemptExpired|Internal|FAIL_EXPIRY|
|ValueTooLargeException|SDK|FAIL_ATR_FULL|
|Anything else|-|FAIL_OTHER|
Note that other SDK-specific errors that represent an ambiguous situation should be classified as FAIL_AMBIGUOUS. For instance, RequestCanceledException in Java SDK.
## `ErrorWrapper` / `TransactionOperationFailed` / `Error`
One key method to tame the complexity of error handling is that of the `ErrorWrapper`, also referred to as an `Error` or `TransactionOperationFailed` (its final name) interchangeably in this document. `Error` provides a means to streamline error handling, in that each stage is only allowed to raise an `Error`, and no other exception, to higher layers. And a means for each stage to control how errors should be handled by the higher layers. This allows higher layers to deal in easy abstractions ("should this transaction be retried?"), while allowing individual stages to control the flow without having to do the nitty-gritty of retrying the transaction themselves. Effectively it's a message-passing mechanism.
These three rules must be followed by implementations (these rules continue to apply in ExtThreadSafety):
1. The only exception that can leave a stage is an ErrorWrapper. That is, `ctx.replace()` can only raise an `ErrorWrapper` to the lambda-caller. The `ctx.replace()` code free to throw other exceptions internally (as in rule 3), but they must be converted to an `ErrorWrapper` before leaving the boundaries of `ctx.replace()`.
* Note that this rule is being bent in places now, with some specific exceptions allowed - e.g. DocAlreadyExists from ctx.insert(), and many errors from ctx.query().
3. If any code internal to a stage raises an ErrorWrapper, all layers should pass that straight through to the core loop without modification. This allows an inner function that has the context on how to handle a particular error, to be certain that its instructions will be followed. It also makes error flow much easier to understand. As soon as an `ErrorWrapper` is spied in the code, it's known exactly what will happen to the transaction next.<br>
* Note that the core loop itself is allowed to ignore or modify the ErrorWrapper, to handle cases such as the operation wanting the transaction to retry, but something going wrong during the rollback.
3. If any code internal to a stage is not certain how an error should be handled and wants to delegate that to higher layer in the stage, it can throw a non-ErrorWrapper exception.
The parameters of an `ErrorWrapper` are simple and encapsulate all possible error-handling:
| Name | Default | Description |
| ---- | ------- | ----------- |
| ec | mandatory |The Error Class of the exception that caused this.<br><br>This field is now deprecated, since it's an internal decision-making detail and a `TransactionOperationFailed` is intended to represent behaviour. It can be removed from `TransactionOperationFailed` and validation of this field can be removed from the performer.
| rollback | true | Whether the attempt should be auto-rolled back. This will almost always be true, except on FAIL_HARD cases where we want to simulate a hard application crash for testing.|
| retry | false | Whether a new attempt will be made after rollback. It is an error to set both rollback and retry. |
| finalErrorToRaise | FAILED | An enum of TRANSACTION_SUCCESS, TRANSACTION_FAILED, TRANSACTION_EXPIRED, TRANSACTION_COMMIT_AMBIGUOUS and TRANSACTION_FAILED_POST_COMMIT.<br><br>If the Error propages all the way to the end of the Core Loop, this is the failure that will be returned to the application. The getCause() will be the cause of this Error. |
| cause | null | An External Error, to raise as the getCause() of any exception raised to the application|
ExtThreadSafety: only the `cause` field is now used by the algorithm itself. But the other fields continue to be sent so that FIT can verify the implementation is behaving correctly.
## ExtThreadSafety
ExtThreadSafety greatly reduces the importance of `TransactionOperationFailed`, replacing this message-passing mechanism with internal state. This avoids problems if the user does not propagate the error (such problems were mostly mitigated by tracking the list of errors, but ExtThreadSafety goes further). Now, `TransactionOperationFailed` is used for just two things:
* We have to return success or throw an error to satisfy the API, so it's a simple signal that something went wrong.
* The `cause` field of the error continues to be used as the final cause of the `TransactionFailed`.
Important: Anywhere in the spec, whenever a `TransactionOperationFailed`/`Error`/`ErrorWrapper` is created, we now set internal state from it before raising it. (There are a few specific exceptions to this general rule that will be called out at those points, with an ExtThreadSafety label.)
The algorithm, taking an `err: TransactionOperationFailed`:
* sb = COMMIT_NOT_ALLOWED
* If !err.rollback sb |= SHOULD_NOT_ROLLBACK
* If !err.retry sb |= SHOULD_NOT_RETRY
* Call [SetStateBits](#SetStateBits) passing `sb` as the BehaviourFlags, and err.finalErrorToRaise as FinalErrorToRaise
## Exceptions Raised by the Application Lambda
The lambda can raise an exception for several reasons:
* If one of the stages like `ctx.replace()` fails. This will always be an [`ErrorWrapper`](#ErrorWrapper) exception.
* Update: that rule is being relaxed. Under ExtQuery, to support SLA some query errors are not `TransactionOperationFailed`. We want the same in the future for ctx.insert() and DocAlreadyExists errors.
* The application logic has a bug.
* The application wants to rollback and fail the transaction.
Any error `err` that is not an `ErrorWrapper` will be converted into one. Classify `err` as error class `ec` then raise `Error(ec, cause=err)`. ExtThreadSafety: as usual, when creating an `Error`, set the state bits from it. This is crucial to handle query SLA errors correctly.
ExtSingleQuery: this error converting method now takes a `singleQueryTransactionMode: Boolean` parameter. If this is set, `err` is just propagated without conversion. This allows `cluster.query()` to continue raising e.g. `ParsingFailedException` rather than wrapping that in a `TransactionFailed.`
If the application has thrown @Stability.Internal exception `RetryTransaction`, then set `retry` to true on the `Error`. This is a currently undocumented and internal method to allow the application to cause the transaction to retry, which we may commit to supporting at a later point if we see a valid need. (It is present currently only because some tests use it.)
## External Errors
The only exceptions that may be raised 'externally', e.g. to the application, are:
* `TransactionExpired` - the transaction expired pre-commit. It is unambiguously not commited, none of its changes are visible.
* `TransactionFailed` - the transaction otherwise failed pre-commit. It is unambiguously not commited, none of its changes are visible.
* `TransactionCommitAmbiguous` - it is ambiguous whether the transaction committed. Handling this is application-dependent but generally hard; nonetheless, it is unavoidable and inevitable reality of dealing with unreliable networks. For example, consider an ordinary mutation being made over the network to any database. The mutation could succeed on the database-side, and then just before the result is returned to the client, the network connection drops. The client cannot receive the success result and will timeout - it is ambiguous to it whether the mutation succeeded or not. Similar, while the transactions protocol will try to resolve the ambiguity, ultimately it is time-bounded by the expiry timeout, and it has to give up at some point. The points where this error can be raised are very limited, it should be regarded as a real edge case.
As of ExtSDKIntegration, all exceptions now have an `Exception` suffix and derive from `CouchbaseException`.
For backwards-compatibility reasons, all exceptions have to derive from `TransactionFailed`.
Historical note: originally, in the 1.0 Java release, there were only `TransactionExpired` and `TransactionFailed`. Error handling, especially post-commit, was considered much more deeply in https://hackmd.io/FG0x_PDdTI-yJfhGc9x8Bw and the model revised. However, to not break 1.0-users who may only be catching `TransactionFailed`, we cannot deviate from deriving all external exceptions from that.
### Cause
These exceptions should contain some platform-specific feature similar to Java’s getCause(), allowing the originating cause of the failure to be raised. This is largely for diagnostic reasons, so that if the TransactionFailed is written to the log, it will also show something useful there on why it failed.
The originating causes include:
* ActiveTransactionRecordEntryNotFound
* ActiveTransactionRecordFull
* ActiveTransactionRecordNotFound
* DocumentAlreadyInTransaction
* PreviousOperationFailed
* AttemptExpired
Plus these from the underlying SDK:
* DocumentNotFoundException
* DocumentExistsException
* FeatureNotFoundException
ExtThreadSafety adds:
* CommitNotPermitted
* RollbackNotPermitted
* ConcurrentOperationsDetectedOnSameDocument
* TransactionAlreadyCommitted
* TransactionAlreadyAborted
When to use them will be defined throughout this document.
Other originating causes are allowed. If the lambda throws ApplicationCustomException(), the root cause will be this. This is the primary mechanism for the application to fail a transaction for a given reason, and it needs to be able to check if it happened.
## Post-commit Error Handling
Options were explored in (Transactions Error Handling Enhancement
)[https://hackmd.io/FG0x_PDdTI-yJfhGc9x8Bw].
We chose to implement `TransactionCommitAmbiguous` and "Solution A": post-commit failures will result in the library raising an internal `TRANSACTION_FAILED_POST_COMMIT` which will ultimately result in returning success to the application, but `TransactionsResult::unstagingComplete()` will be false.
There are some graduations to what constitutes a "post-commit failure". Most error classes will be regarded as grounds for immediate failure. This includes FAIL_TRANSIENT - by the time the SDK returns to us, it has already been retrying that failure (if non-ambiguous) for a default timeout of 2.5 seconds.
Even though under "Solution A" all post-commit failures are allowed to immediately terminate the commit, there is some retrying of specific error classes, detailed below. The line over whether to retry an operation or not is somewhat arbritrary, and reflects that the complex error handling logic had already been written during a time when the commit would be retried until expiry. Since we may end up implementing that again as an optional mode ("Solution C") the logic is being left.
# Rollback
The attempt is being rolled back. This can be by either app-rollback or auto-rollback (and in the latter case, the transaction may either go on to make another attempt, or not).
## Triggers
There are two ways for an application to initiate a rollback:
| Exit points | Result |
| ----------------------------- | --------------------------------------------------
| Throw FAIL_OTHER | TransactionFailed |
| Call ctx.rollback() | Success |
Making `ctx.rollback()` return success was perhaps an error, but one we're stuck with. The application has no built-in way of knowing if their transaction committed, or rolled back. On Java, they could use an AtomicBoolean outside of the lambda to record which path they went down. Throwing a custom exception, and checking for it in the TransactionFailed exception, is the preferred route.
We call `ctx.rollback()` an `app-rollback`, all other forms of rollback `auto-rollback`.
```java
try {
transactions.run((ctx) -> {
throw MyCustomReasonForRollback();
});
}
catch (TransactionFailed err) {
if (err.getCause() instanceof MyCustomReasonForRollback) {
// all is good
}
else {
// all is bad
}
}
```
In ExtSDKIntegration, ctx.rollback() is removed, which also removes the app-rollback form.
## Error Handling During Rollback
Discussed more here https://hackmd.io/FG0x_PDdTI-yJfhGc9x8Bw.
All forms of rollback (app-rollback, auto-rollback, app-initiated or not), will retry any errors, until the transaction expires.
Then return success or failure, as follows:
| Exit points | Rollback type | If RB succeeds | If RB expires |
| ----------------------------- | -------------------------------------------------- | -- | -- |
| ctx.rollback() (app-rollback) | App initiated app-rollback | Return success | Raise TransactionExpired |
| throw MyCustomError() | App initiated auto-rollback | Raise TransactionFailed | Raise TransactionExpired |
| ctx.get("does not exist") | Auto-rollback, will not retry | Raise whatever error the op requested | Raise TransactionExpired |
| Transient server error | Auto-rollback, will retry | Go on to retry | Raise TransactionExpired |
## Primary Operations
During app-rollback (`ctx.rollback()`), the primary operation is the rollback itself.
During auto-rollback, the primary operation is whatever triggered the rollback.
Any error returned to the application is generally from the primary operation.
The exception to this, is if something goes wrong during auto-rollback, when the primary operation indicated that the transaction should be retried. We are unable to retry due to the failed auto-rollback, which will have expired the transaction with current logic. So raise the original `Error` from the primary exception, but with retry=false.
Some examples:
If the transaction failed because a document could not be found (the primary operation), and then something goes wrong during the auto-rollback: the application cares mainly about the missing document. So it should get back TransactionFailed with a cause of DocumentNotFoundException, rather than TransactionExpired.
But if the transaction reaches the point of `ctx.rollback()`, and something goes wrong then, then obviously that gets raised.
# Attempt States
A transaction involves one or more attempts. Each attempt has a unique UUID, and the transaction has a unique UUID too. Each attempt gets an entry in an ATR, if the attempt reaches the SetATRPending state. Each attempt has a state:
| State | Description |
| ----- | ----------- |
| NOTHING_WRITTEN | The state before the ATR entry is created.<br><br>Historical note: The Java enum calls this NOT_STARTED, which is a poor name, as a read-only transaction can complete successfully with a single NOTHING_WRITTEN attempt. But this was realised after 1.0.0 was published. |
| PENDING | Set once the ATR is set to PENDING, indicating the staging process is ongoing. |
| COMMITTED | Set once the ATR is set to COMMITTED, indicating that it is in the process of being committed. The documents are in an unknown state (perhaps committed, perhaps not). The moment the ATR is set to this state, the attempt (and hence the transaction) are regarded as committed. This is a point of no return. |
| COMPLETED | Set once the commit is fully completed. All documents are committed, and the ATR is set to COMPLETED. ExtRemoveCompleted: the ATR entry is now removed here, instead. |
| ABORTED | Set once the ATR is set to ABORTED, indicating that it is in the process of being rolled back. This can be an app-rollback or an auto-rollback, no distinction is made. The documents are in an unknown state (perhaps rolled back, perhaps not). |
| ROLLED_BACK | Set once the attempt is fully rolled back. All documents are rolled back, and the ATR is set to ROLLED_BACK. ExtRemoveCompleted: the ATR entry is now removed here, instead. |
| UNKNOWN | Set once the attempt is fully rolled back. All documents are rolled back, and the ATR is set to ROLLED_BACK. ExtRemoveCompleted: the ATR entry is now removed here, instead. |
# Expiration
Transactions expire after a configurable amount of time. The default is 15 seconds.
Expiration is handled in two ways:
* In the main algorithm, it's simpy an in-process timer, implemented using whatever platform-specific tools are available. Since expiration times are expected to be multiple seconds, timing accuracy to < 50 millis is acceptable and high precision clocks are not required.
* The cleanup algorithm uses a different approach based around the CAS timestamp of the transaction's ATR's node. This is detailed in [the cleanup section](#Lost-Transactions-and-Cleanup).
## ExpiryOvertimeMode
Introduced in TXNJ-124, this mode implements the behaviour that, upon expiring, a transaction should make one last effort to "finish".
Finish means:
* If pre-commit, try to rollback.
* If post-commit, try to finish commit.
* If post-abort, try to finish rollback.
Essentially the algo enter "overtime". If there are any errors, then the algo will bailout and make no further attempt to interact with the cluster (leaving a lost transaction for the cleanup process). If there are no errors, then the transaction will finish. Either way, it will ultimately return success or failure depending on what state it was in. E.g. if the transaction expires post-commit, that will still return success. The exact behaviour required by each stage in ExpiryOvertimeMode is detailed throughout [Couchbase Transaction Stages](/Eaf20XhtRhi8aGEn_xIH8A).
ExpiryOvertimeMode strikes a balance between not overwhelming an already struggling cluster (this being a likely cause of an expired transaction), while trying to complete the transaction immediately.
However, it also adds a lot of complexity, and in new functionality it is no longer being added. It will eventually be removed throughout.
# Locks
Txns currently doesn’t take actual locks using the pessimistic locking feature of Couchbase, as these locks aren’t replicated. This may change in a future server release.
However, any document that's had a staged mutation written to it is effectively locked from modification by other transactions due to write-write conflict detection, until the document has either been committed or the txn expires.
# UUIDs
Each transaction, plus each attempt inside a transaction, gets a UUID.
These are type 4 UUIDs, e.g. pseudo-randomly generated 128-bit unique ID, following [RFC4122](https://www.ietf.org/rfc/rfc4122.txt). All UUIDs generated must be in this format.
UUID clashes are expected to be unlikely to the point of being ignorable, especially since they are not stored indefinitely by the protocol. Generally at most a few tens of thousands will be extant and comparable at any one time.
# TransactionAttemptContext
(Pre-ExtSDKIntegration this was `AttemptContext`).
The `TransactionAttemptContext` is the core driver of a single transaction attempt. It is provided to the application's lambda so it is the key way that the lambda specifies what happens in the transaction.
A transaction consists of at least one attempt. Each retry will trigger another attempt. A small amount of data is saved for each attempt to be returned in the final `TransactionResult`, for debugging, transparency, and testing.
The context contains some internal mutable state, though it is kept as minimal as possible without inflicting burden:
* `state` - reflecting the [Attempt States](#Attempt-States). This will almost always reflect the written ATR entry state, though in some exceptional cases generally involving FAIL_AMBIGUOUS, it won't (but always in harmless ways). It starts in NOT_STARTED.
* `ThreadSafeList[StagedMutation] stagedMutations` - any mutations written will also get saved here. While somewhat bloating memory requirements, it means the unstaging can be done without having to reread the document. It starts empty.
* ~~`Map[String, MutationToken] finalMutationTokens` - stores the mutation tokens after documents are successfully unstaged.~~
* Removing: this is in the Java implementation. Mutation tokens (used for AT_PLUS) are not stored and returned in other implementations. Partly as query does not return them.
* Optional[`atrId`] and Optional[`atrCollection`] - set when the location of the ATR entry is chosen, just before the first mutation. Throughout this document, use whatever platform idiom best matches an Optional monad. Start empty.
* `expiryOvertimeMode` - specified in [ExpiryOvertimeMode](#ExpiryOvertimeMode). Starts false.
* `ThreadSafeList[ErrorWrapper] errors` - stores errors from any operations so the transaction cannot erroneously proceed to committed if the app does not propagate errors. Removed in ExtThreadSafety.
* `UnstagingMode unstagingMode` - configures to be in ExtMemoryOptUnstaging or ExtTimeOptUnstaging mode
* EXT_QUERY: `ThreadSafeBoolean isDone` - true iff the attempt has started committing or rolling back, in which case no further operations are permitted. (This is a more robust version of the pre-EXT_QUERY check, which simply checked the `state`). Removed in ExtThreadSafety.
And some additional immutable state:
* `attemptId` - a [UUID](#UUIDs) for this particular attempt
ExtThreadSafety:
* Removes `isDone`.
* Removes `errors`.
* Adds a mutex lock, `lock`.
* A reentrant lock is not required.
* The suggested implementation in this spec assumes a lock that can be safely double-unlocked. E.g. if an actor isn't sure if it holds the lock or not (on an error being raised), it can unlock with its lock token and this will be a no-op if that lock token previously already unlocked. This significantly simplifies error handling as the lock does not have to be explicitly unlocked whenever an error occurs under the lock. If such a lock is not available, the implementation instead needs to perform that explicit unlocking.
* See [internal discussion](https://couchbase.slack.com/archives/C014WB8U2MQ/p1636133095102700) for the exact details of the custom lock used by Java and Go, which immediately notifies the next waiter on unlock.
* Adds a Go-style 'WaitGroup', `kvOps`.
* A WaitGroup allows a) an unlimited number of operations to be added to it, b) marking those operations are done, and c) waiting for those operations to be done.
* It is used so that at crucial points (commit, rollback, and switching to queryMode), we can wait for all KV operations to complete.
* Adds state bits (see immediately below).
* `stagedMutations` no longer needs to be a thread-safe list, as it is always written under lock.
* Makes `expiryOvertimeMode` a thread-safe variable.
* It would be a challenging spec change and a performance issue to always be locked when this needs to be written, plus expiry overtime mode is being phased out anyway as we aim towards making the transaction timeout a strict SLA. Using a thread-safe variable is the compromise.
## State bits
This is added in ExtThreadSafety.
Prior to this, transactional behaviour was partly driven through propagating `TransactionOperationFailed`, which contains information indicating whether to rollback, retry, which error to raise, and the cause to set on the final exception. State bits replace the first three with internal state.
Exactly how the state bits are stored is not hugely important, as it is purely internal state. A suggested implementation including bit layout is provided below for implementation convenience.
The suggested implementation requires that the implementation platform supports a thread-safe int32 that can detect concurrent modifications, necessary to safely merge in concurrent changes. Java uses an AtomicInteger, Go uses atomic.CompareAndSwapUint32.
If this is not available, it can be replaced with additional thread-safe locking. Usually the bits are updated and read inside the AttemptContext-wide lock anyway, though some exceptions exist - the main one being when an error is raised.
**BehaviourFlags**
The first (lowest) 4 bits are used for these fields, which replace `shouldRetry` and `shouldRollback` from `TransactionOperationFailed`, and `isDone` from ExtQuery. They are called the BehaviourFlags:
| Name | Bit | Description |
| --------------------- | -------- | -------- |
| CommitNotAllowed | 0x1 | If set, commit (either ctx.commit() or implicit commit) is not allowed. |
| AppRollbackNotAllowed | 0x2 | If set, ctx.rollback() is not allowed. Auto-rollback is still permitted. |
| ShouldNotRollback | 0x4 | If set, auto-rollback will not be done. |
| ShouldNotRetry | 0x8 | If set, the transaction will not retry. |
**FinalErrorToRaise**
The next 3 bits are used to store the finalErrorToRaise field from `TransactionOperationFailed`, and are called FinalErrorToRaise:
| Name | Numerical value |
| --------------------- | -------- |
| TRANSACTION_SUCCESS | 0 |
| TRANSACTION_FAILED | 1 |
| TRANSACTION_EXPIRED | 2 |
| TRANSACTION_COMMIT_AMBIGUOUS | 3 |
| TRANSACTION_FAILED_POST_COMMIT | 4 |
It is important that they are represented with those numbers (see [SetStateBits](#SetStateBits) below).
The state bits start off at 0. That is, we assume until an error occurs that we can commit, app-rollback is allowed, and rollback and retry are allowed (the last two will only happen if an error happens, of course).
### SetStateBits
The state bits are modified using a thread-safe merging algorithm that simply does a bitwise OR on the BehaviourFlags, but will only 'upgrade' the FinalErrorToRaise to a higher number. Higher numbers always either encode more information or come from a later protocol stage, so it's never possible to go from a higher failure to a lower.
The algorithm in proto-code:
```
STATE_BITS_MASK_FINAL_ERROR = 0b1110000
STATE_BITS_POSITION_FINAL_ERROR = 4
// newBehaviourFlags = bitmask of bits to set, e.g. CommitNotAllowed | AppRollbackNotAllowed
// newFinalError = FinalErrorToRaise as a number, e.g. TRANSACTION_COMMIT_AMBIGUOUS=3
void setStateBits(newBehaviourFlags, newFinalError = 0) {
oldValue = stateBits
// BehaviourFlags are merged with a simple bitwise OR
newValue = oldValue | newBehaviourFlags
// Only save the new ToRaise if it beats what's there now
if (newFinalError > ((oldValue & STATE_BITS_MASK_FINAL_ERROR) >> STATE_BITS_POSITION_FINAL_ERROR)) {
newValue = (newValue & STATE_BITS_MASK_BITS) | (newFinalError << STATE_BITS_POSITION_FINAL_ERROR)
}
// compareAndSet represents a thread-safe set of the new value that will fail if the current value of stateBits != oldValue
// e.g. it returns false if there has been a concurrent modification
while (!stateBits.compareAndSet(oldValue, newValue)) {
oldValue = stateBits;
newValue = oldValue | newBehaviourFlags
if (newFinalError > ((oldValue & STATE_BITS_MASK_FINAL_ERROR) >> STATE_BITS_POSITION_FINAL_ERROR)) {
newValue = (newValue & STATE_BITS_MASK_BITS) | (newFinalError << STATE_BITS_POSITION_FINAL_ERROR)
}
}
}
```
## UnstagingMode
|Name|Description|
|----|----|
|MEMORY_OPTIMIZED|For ExtMemoryOptUnstaging mode|
|TIME_OPTIMIZED|For ExtTimeOptUnstaging mode|
## StagedMutation
This is an internal-only data struct with no application visibility.
Stores these fields:
|Field|Type|Description|
|-----|----|-----------|
|doc|TransactionGetResult|This is the post-mutation doc, e.g. it contains the CAS of the mutation. In fact only a few fields are required from it - CAS, id, bucket/scope/collection/names - so feel free to store only these instead|
|content|byte[]|The transcoded JSON content that was sent to the cluster. It will be null in ExtMemoryOptUnstaging mode, e.g. if `unstagingMode` == `MEMORY_OPTIMIZED`|
|type|StagedMutationType|Enum is defined below|
|mr|MutateInResult|The result of the mutation|
|operationId|String|Added under ExtThreadSafety, each operation gets a UUID.|
|stagedUserFlags|int32|Added under ExtBinarySupport, each operation gets a user flags field, that will be set on the document at commit point.|
|currentUserFlags|int32|Added under ExtBinarySupport, each operation gets the document's current (pre-transaction) user flags. In some cases (INSERT), this will be the same as `stagedUserFlags`.|
`StagedMutationType` indicates what mutation is being performed, and consists of REPLACE, REMOVE and INSERT.
# The Core Loop
The core loop of a transaction is responsible for running the application's lambda one or more times, and handling any errors that result.
It must be possible to run multiple transactions concurrently, of course. This may be performed on a threading pool, a reactor scheduler, or whatever makes most make sense for the platform.
The core loop is reasonably simple, as the complexities of error handling (such as deciding when and how to rollback or retry) are decided by the individual [stages](/Eaf20XhtRhi8aGEn_xIH8A?edit). These decisions are raised in a thrown `ErrorWrapper`.
Before the loop begins:
* At the start of a transaction, the transaction gives itself a [UUID](#UUIDs) transactionId.
The loop takes parameters:
* `<anonymous function> code` - arbitrary code to run. Generally the user's lambda, but in ExtSingleQuery will be the tximplicit query instead.
* `Boolean singleQueryTransactionMode`
The loop:
* Create a new [AttemptContext](#AttemptContext). (A transaction contains one or more attempts.)
* Execute `code`.
* If `code` propagates an error `err`, follow [Exceptions Raised by the Application Lambda](#Exceptions-Raised-by-the-Application-Lambda), passing `singleQueryTransactionMode`.
* From now on, all errors must be an `TransactionOperationFailed` (this is no longer true for singleQueryTransactionMode).
* Perform an implicit commit if necessary, following the same logic as performed on ctx.commit():
* singleQueryTransactionMode: implicit commit is never performed.
* ExtThreadSafety: determine if necessary by checking the BehaviourFlags for !CommitNotAllowed (this is done outside the lock, which is ok as the state bits are a thread-safe atomic variable). This indicates either an error happened or commit/rollback have already happened.
* Else: use a lack of error from the lambda together with the isDone flag.
* Perform auto-rollback if necessary:
* Deciding whether to auto-rollback:
* singleQueryTransactionMode: auto-rollback is never performed.
* ExtThreadSafety: determine if necessary by checking the state bits, outside the lock. BehaviourFlags must be !ShouldNotRollback and FinalErrorToRaise must be !TRANSACTION_SUCCESS.
* Else: determine by if a `TransactionOperationFailed` was raised with rollback=true.
* !ExtThreadSafety: if `state` == NOT_STARTED and !queryMode, skip rollback. There is nothing to do (except in queryMode, where we don't know what state the transaction is in on the query side, so ROLLBACK for safety).
* ExtThreadSafety performs this same check instead inside the rollback logic, after waiting for KV ops to finish and entering the lock. Since one of those may be in process of writing the ATR entry but has not yet got to updating the state.
* Perform the auto-rollback, following [this logic](https://hackmd.io/Eaf20XhtRhi8aGEn_xIH8A?view#rollbackInternal), passing isAppRollback=false.
* If auto-rollback fails with error `err`:
* Propagate the original `TransactionOperationFailed` that provoked this rollback, with retry changed to false.
* Discussion: we do not propagate `err` because the user isn't interested in problems during auto-rollback, they care about what made rollback happen.
* ExtThreadSafety: at this point we have either committed or rolled back, and any new ops are blocked from happening by the BehaviourFlags. So we don't need to lock anymore.
* [AddCleanupRequest](#AddCleanupRequest), if the cleanup thread is configued to be running.
* Perform retry if necessary:
* Deciding whether to retry:
* ExtThreadSafety: determine if necessary by checking the state bits, outside the lock. BehaviourFlags must be !ShouldNotRetry and FinalErrorToRaise must be !TRANSACTION_SUCCESS.
* Else: determine by if a `TransactionOperationFailed` was raised with retry=true.
* If we are to retry:
* Check first if the transaction has expired using stage "HOOK_BEFORE_RETRY" (the stage was only added in ExtThreadSafety). If so, raise `Error(ec = FAIL_EXPIRY, rollback=false, raise = TRANSACTION_EXPIRED)`, setting state bits as normal. This avoids an unnecessary retry.
* Apply [OpRetryBackoff](#OpRetryBackoff), with randomized jitter. E.g. each attempt will wait exponentially longer before retrying, up to a limit.
* Go back to the start of the core loop, e.g. a new attempt.
* We are not going to retry and have reached the end of the core loop and hence the transaction. We may have a propagated `CouchbaseException err` or not (on success or if user has swallowed exceptions):
* Set `toRaise` to:
* ExtThreadSafety: `FinalErrorToRaise` stored in the state bits.
* Else: the `raise` field of the propagated `err`. If there is no such error, then TRANSACTION_SUCCESS.
* Then based on `toRaise`:
* `TRANSACTION_FAILED_POST_COMMIT` -> Failure post-commit may or may not be a failure to the application, as the cleanup process should complete the commit soon. It often depends on whether the application wants RYOW, e.g. AT_PLUS. So, success will be returned, but `TransactionResult.unstagingComplete()` will be false. The application can interpret this as it needs.
* `TRANSACTION_SUCCESS` ->
* If singleQueryTransactionMode && `err` is present: propagate `err`.
* Discussion: can get here due to SLA. E.g. a ParsingException from query won't set `toRaise`.
* Else return a successful `TransactionResult`, with `unstagingComplete()` true.
* `TRANSACTION_EXPIRED` -> Raise `TransactionExpired` to application, with a cause of `err.cause` (or null/empty if `err` is not present).
* `TRANSACTION_COMMIT_AMBIGUOUS` -> Raise `TransactionCommitAmbiguous` to application, with a cause of `err.cause` (or null/empty if `err` is not present).
* `Else` -> Raise `TransactionFailed` to application, with a cause of `err.cause` (or null/empty if `err` is not present).
# Stages in Detail
This has been moved to [Couchbase Transactions Stages](https://hackmd.io/Eaf20XhtRhi8aGEn_xIH8A) as it was becoming too large.
# Couchbase Transactions Cleanup
All cleanup discussion has been moved to [Couchbase Transactions Cleanup](https://hackmd.io/w5hsXGwYTRmv2IxBfAoORw) for size reasons.
# Interaction with Couchbase Data Platform
## Backup
The goal for 6.5 was not to to preserve atomicity, which will require a much more radical solution, but for live restore to not interfere with the current live transactions. Work is underway to preserve atomicity across backup in a future release.
In 6.5 and 6.6, and possibly higher (TBD):
* Restore will strip out transactional metadata, e.g. the "txn" xattr.
* Restore will restore the document with its pre-transaction CAS, revid and expiry. E.g. as though the document had not been involved in a transaction at all.
* Restore will not restore any documents starting "_txn:".
* Backup will not negotiate the IncludeDeletedUserXattrs DCP_OPEN flag added in 6.6, so it will not receive staged inserts as of protocol version 2. Only transactions will be using this feature (discussed and agreed on by SDK and memcached teams), and restore will not be restoring any transactional metadata anyway.
Backup in 6.5: [design document](https://docs.google.com/document/d/1Nsy3m_-SmdCCpTw8dGijqjbCkQ_QrQGe7yd9ViMZhhQ/edit#)
Backup in future: [PRD](https://docs.google.com/document/d/1eo2gfymU02ldtO861Z0lkIMEy8IOhQTc8Jncxf_HMvw/edit) and [implementation ideas](https://docs.google.com/document/d/1RxLKEK_NS2JGENMualtbdQVnvmM4PT4YNCQxb3-Fab0/edit#heading=h.6kyy1siveyv2)
## N1QL & GSI
As of 7.0, query is deeply integrated into transactions.
## Eventing
Key tickets: [MB-39942](https://issues.couchbase.com/browse/MB-39942)
## Sync Gateway
ExtMobileInterop improves handling of metadata-only changes to documents, which improves some quite niche Sync Gateway interop issues.
Key tickets:
* [CBD-3860](https://issues.couchbase.com/browse/CBD-3860)
* [CBD-3859](https://issues.couchbase.com/browse/CBD-3859)
* [CBG-921](https://issues.couchbase.com/browse/CBG-921) - SG ignores ATRs as of 2.8.0
## FTS
Integration is not currently on the roadmap.
FTS will have a Read Committed isolation level of transactions. And commits will be eventually consistent.
## Analytics
Integration is not currently on the roadmap.
Analytics will have a Read Committed isolation level of transactions. And commits will be eventually consistent.
Key tickets:
* [MB-34546](https://issues.couchbase.com/browse/MB-34546) - cbas ignores all transactional metadata docs as of 6.5.0.
## XDCR
Filters out all transactional metadata (both documents and staged changes) from 6.5.0.
[MB-47813](https://issues.couchbase.com/browse/MB-47813): XDCR used to not replicate docs with `txn` xattr at all. But following issues with this approach, it changed to replicate the docs, but stripping `txn`. This change was made in 7.1.0.
# Testing
There are multiple transactions implementations, so to make sure that all implementations are tested to the same standard, and also to test interop between them, transactions testing is handled in a centralised fashion.
All tests are written in a single language (Java), and run from a "driver" process.
The driver drives language-dependent "performer" processes. There is a performer for each of the implementation target language.
In fact there can be variants for each performer, e.g. there can be concurrently running a Java performer running version 1.0.0 of the transactions library, and a Java performer running 1.1.
There are tests in the driver that will perform interop testing between two performers. So it can test e.g. the C++ interacting with the Java performer, or two performers using different versions.
An implementation just needs to implement a performer, a relatively small piece of code, to fit into this testing infrastructure.
The code is available [here](https://github.com/couchbaselabs/transactions-fit-performer).
## Hooks
Transactions involves many interactions with the cluster, and much of the complexity of error handling stems from these interactions. It is essential that any possible fault can be injected at any possible point. To allow this, the protocol defines a large number of 'hooks' that are triggered generally before and after each interaction with the cluster.
Hooks are simply callbacks that the implementation calls at various points, and that the tests may override. Their return value is ignored, and they basically allow exceptions to be thrown (though in advanced cases they can also be used to execute arbritrary code, say for validation purposes).
The hooks themselves are defined throughout the [Couchbase Transaction Stages](/Eaf20XhtRhi8aGEn_xIH8A) document.
The exact mechanism whereby hooks are overridden by tests is left platform-specific. Some platforms support sophisticated mocking which would enable this. If not, then the implementation may use the approach used by the Java implementation, which is to provide a method similar to this on the `TransactionConfigBuilder`:
```
/**
* For internal testing. Applications should not require this.
*
* @param attemptContextFactory provide a factory that will create
* {@link com.couchbase.transactions.AttemptContextReactive} whenever one is
* required, allowing methods on this to be mocked out. Can be null, in which case
* the default factory will be used.
* @param cleanerFactory provide a factory that will create {@link CleanerFactory} whenever one is
* required, allowing methods on this to be mocked out. Can be null, in which case
* the default factory will be used.
* @param clientRecordFactory provide a factory that will create {@link ClientRecordFactory} whenever one is
* required, allowing methods on this to be mocked out. Can be null, in which case
* the default factory will be used.
*/
@Stability.Internal
public TransactionConfigBuilder testFactories(AttemptContextFactory attemptContextFactory,
CleanerFactory cleanerFactory,
ClientRecordFactory clientRecordFactory) {
```
# Protocol Versions
Everything under here has been moved to [Couchbase Transactions Protocol Extensions](https://hackmd.io/bJzZMt3ASfe8b7309lzK1A), due to hitting hackmd size limits.