Seed CX Market Data System - Version 1 ====================================== ## Version 1 Deprecation Notice :::danger This document describes **version 1** of Seed CX Market Data System. While this protocol version is currently deployed in the production environment, it is slated for deprecation and has been replaced with [version 2](https://www.seedcx.com/api/bin-md/). Please contact Seed CX operations for more information. ::: Introduction ============ Trading applications need information about the state of the market in order to effectively execute their strategies on an exchange. Seed CX provides a Market Data System which delivers Level 3 market data. The Seed CX system uses an instrument-centric, proprietary binary protocol to communicate with client applications. This document describes the protocol and serves as a comprehensive guide to consuming Seed CX market data. System Overview =============== Seed CX Market Data is distributed using a binary protocol, which is loosely modeled on industry-standard PITCH and ITCH protocols. It is a so-called *Level 3* or *market-by-order* system, where a complete view of the market (down to the level of anonymized individual orders) is made available for every instrument traded at Seed CX. Client applications wishing to consume market data need to be connected to Seed CX. This [can be done](#Getting-Connected) via VPN or cross-connect. As stated previously, market data in the Seed CX system is instrument-centric. Each instrument has it's own: * numeric identifier; * sequence numbers, and; * can be consumed independently from all other instruments. An instrument's market data comes in two flavors: full-depth market snapshots and incremental updates. These are delivered by two distinct services of Seed CX's market data gateway. ![Seed CX Market Data System Diagram](https://s3.amazonaws.com/basically_me_images/seed/seed-market-data-system.svg) The snapshot mechanism is used when joining the data stream and for recovery purposes. Snapshots are obtained by establishing a TCP connection to the *Market Snapshot Service* and sending an [Instrument Snapshot Request](#Instrument-Snapshot-Request). Such requests are meant to be infrequent and are rate-limited. Applications should obtain an initial snapshot and then apply subsequent incremental updates to it. Incremental updates are sent by the *Incremental Update Service* on two redundant channels using UDP multicast. These updates are essentially commands (such as [add order](#Add-Order-Message), [delete order](#Delete-Order-Message), [replace order](#Replace-Order-Message), etc.) to the application telling it how to update its in-memory order book. Each multicast channel may contain feeds for multiple instruments. Refer to the reference data for a given [environment](#Environments) to obtain the multicast group addresses and ports for desired instruments. Each UDP packet will only contain messages for a single instrument. Applications can inspect the `instrument id` contained within the [packet header](Packet-Header) and decide to either ignore or process a given UDP packet. The [Consuming Market Data](#Consuming-Market-Data) section describes this process in detail. ## Getting Connected [Contact Seed CX Operations](https://www.seedcx.com/#Contact) to establish cross-connect or VPN connectivity to the Seed CX network. Use reference data for a given [environment](#Environments) to obtain the list of available instruments and details on their corresponding multicast channels. In addition, reference data contains endpoints for obtaining full market snapshots and specifies the number of decimal places in the implied price. ## Environments Seed CX provides market data for two distinct environments. ### Certification The Certification environment is available for applications to prove out and certify their readiness to work with Seed CX systems. This includes mock trading via FIX protocol as well as consumption of binary market data. The Seed CX Certification environment is hosted in the Equinix CH1 data center located in Chicago, IL. The reference data URL is ==https://instruments.cert.seedcx.com==. *Note: While the reference data URLs are being activated, please contact Seed CX support to receive reference data in form of an XML file.* ### Production The Production environment exists for live trading and order execution. It is recommended that applications conduct trial runs in the [certification](#Certification) environment before executing live trades and consuming live market data. The Seed CX Production environment is hosted in the Equinix NY4 data center located in Secaucus, NJ. The reference data URL is ==https://instruments.seedcx.com==. *Note: While the reference data URLs are being activated, please contact Seed CX support to receive reference data in form of an XML file.* Consuming Market Data ===================== Each instrument has it's own market, unique identifier, sequence number, and data feed. While feeds for several instruments may reside on the same multicast channel, every feed may be processed independently using the algorithm outlined below. Each step is expanded in detail later in this section. > 1. Initialize > > a) Open UDP sockets and [join multicast groups](#Joining-the-Multicast-Group). > > b) Prepare data structures for desired instruments. > > 2. Perform feed recovery > > a) Begin caching incremental updates [read from multicast channels](#Reading-incremental-updates-from-multicast). > > b) Request instrument snapshot. > > c) [Reconcile](#Reconciliation) snapshot with cached incremental updates. > > - You're up to date. Go to step 3. > > 3. Build book from incremental updates > > a) [Read incremental update](#Reading-incremental-updates-from-multicast) from multicast channels. > > - [Bad sequence](#Sequence-numbers)? Go to step 2. > > b) Apply the update to the order book. > > c) Go to step 3. ## Detailed Steps ### Joining the Multicast Group A client application should obtain reference data for a given [environment](#Environments) and decide which instruments it wants to subscribe to. Each instrument's reference data contains information about the snapshot endpoint, as well as multicast group IP addresses which need to be joined in order to consume incremental updates. For redundancy, the same incremental market data updates are sent on two separate multicast channels: `A` and `B`. It is recommended that client applications join both multicast channels in order to minimize the probability of data loss. Below is an example reference data snippet showing the multicast groups for `BTC/USD` instrument. ```xml <instruments> <instrument> <instrument_id>1</instrument_id> <price_decimals>0</price_decimals> <instrument_code>COSP:BTC/USD</instrument_code> <instrument_name>Bitcoin USD Spot</instrument_name> <!-- snip --> <market_data> <feed> <type>Incremental</type> <name>A</name> <protocol>UDP/IP</protocol> <ip>239.0.1.1</ip> <port>1100</port> </feed> <feed> <type>Incremental</type> <name>B</name> <protocol>UDP/IP</protocol> <ip>239.0.1.2</ip> <port>1100</port> </feed> <feed> <type>Snapshot</type> <name>External</name> <protocol>TCP/IP</protocol> <ip>10.50.10.15</ip> <port>65001</port> </feed> </market_data> </instrument> <!-- snip --> </instruments> ``` ### Reading Incremental Updates from Multicast For increased redundancy, the same incremental market data updates are sent on both `A` and `B` multicast channels. Due to the nature of UDP multicast, messages from either feed may arrive out of order. Applications consuming instrument feeds need to inspect the [packet header](#Packet-Header) to see if the instrument's sequence number has already been processed. #### Sequence Numbers Each instrument is sequenced separately, which implies that applications need to keep track of the sequence per instrument. The [packet header](#Packet-Header) in the Seed CX binary protocol contains the sequence number of the *first* message in the packet. The sequence numbers of remaining messages are implied. For example: ``` Packet Header instrument id: 1 sequence number: 10 message count: 3 Message 0 implied sequence number: 10 (packet header + 0) Message 1 implied sequence number: 11 (packet header + 1) Message 2 implied sequence number: 12 (packet header + 2) ``` In general, applications should keep track of the *next expected* sequence number to detect potential data gaps. :::info next expected sequence number = packet sequence number + message count ::: If a sequence number matches the next expected sequence number, the packet is new and should be processed. If a sequence number is lower than the next expected sequence number, the packet has already been processed and should be discarded. If a packet is received with a sequence number higher than expected, it is an indication that data loss has occurred and the instrument's feed needs to be recovered. ### Snapshots and Feed Recovery Whether performing a late join, or recovering from a data gap, applications can request a full market depth snapshot by sending an [Instrument Snapshot Request](#Instrument-Snapshot-Request) to the *Market Snapshot Service*. The IP address and port for the service can be found in reference data for a given [environment](#Environments), for example: ```xml <instruments> <instrument> <instrument_id>1</instrument_id> <price_decimals>0</price_decimals> <instrument_code>COSP:BTC/USD</instrument_code> <instrument_name>Bitcoin USD Spot</instrument_name> <!-- snip --> <market_data> <!-- snip --> <feed> <type>Snapshot</type> <name>External</name> <protocol>TCP/IP</protocol> <ip>10.50.10.15</ip> <port>65001</port> </feed> </market_data> </instrument> <!-- snip --> </instruments> ``` The [instrument snapshot response](#Snapshot-Success-Response) returned from the service contains a collection of resting `bid` and `ask` orders that are in the market `as of` sequence number returned in the response. Note that the orders are transmitted based on how adventageous they are, ie. how far they are from the inside market. Their ordering matches their position in the matching engine's execution queue. ### Reconciliation Before requesting a full-depth snapshot, the client application should first [join the multicast groups](#Joining-the-Multicast-Group) and start recording any incremental instrument updates into memory. After the [instrument snapshot response](#Snapshot-Success-Response) is successfully returned by the *Market Snapshot Service*, it will contain an `as of` sequence number. Applications should clear their in-memory order book structure for the instrument and bulk insert the orders which arrived in the snapshot. Additionally, consuming applications should traverse the cached incremental updates and discard any update whose sequence number is lesser or equal to the one returned within the instrument snapshot response. Incremental updates with sequence numbers higher than that of the snapshot's should then be normally applied to the order book. Once completed, the instrument's feed is up to date. Binary Marketdata Protocol Specification ======================================== The following sections describe Seed CX binary market data protocol. Seed CX reserves the right to change and evolve the protocol on an as-needed basis. Rest assured, best efforts will be taken to preserve the existing wire layout. That said, in an effort to provide transparency to potential future updates, the below are examples subject to our consideration: 1. Reserved fields may be re-purposed to hold public data; 2. The header and message lengths may be increased; 3. New message types may be introduced. :::info The current protocol version is **2**. This document describes the protocol as of version 1. ::: ## Data Types The following field types are used within the Seed CX binary market data protocol. They appear in little endian byte order unless specifically stated otherwise. ### Byte Byte is the collection of 8 bits. This data type is primarirly used to represent an opaque sequence of data, such as space reserved for future protocol versions or for padding. ### Numeric Numerics are unsigned integral values expressed in little endian byte order. Typically `1`, `2`, `4` or `8` bytes long. ### Price Prices are encoded as 64-bit _signed_ integers in little endian byte order. They have an implied decimal point, which varies from instrument to instrument. Use instrument _reference data_ for a given [environment](#Environments) to obtain the number of places after the decimal point. ### Quantity Quantities are encoded as 64-bit _unsigned_ integers in little endian byte order. They are usually used for representing order sizes. ### Instrument Identifier Instrument Identifiers are encoded as 64-bit unsigned integers in little endian byte order. They may change with each trading session. Use instrument _reference data_ for a given [environment](#Environments) to obtain instrument identifiers. ### Order and Execution Identifiers Order Identifiers, as well as Execution Identifiers, are encoded as 64-bit binary objects. They can be printed as integers in little endian byte order, or as `base58`-encoded alphanumeric strings. ### Timestamp Timestamps are encoded as 64-bit unsigned integers in little endian byte order. They represent the number of nanoseconds since the start of UNIX Epoch (`1970-01-01 00:00:00 UTC`). ### Text Text is represented as a sequence of ASCII-encoded characters. If the text is fixed length, any excess space must be filled with byte value of `0`. ## UDP Multicast Messages Seed CX distributes incremental market data updates using UDP frames on redundant `A-` and `B-` multicast channels. All UDP frames follow this structure: ![UDP wire message structure](https://s3.amazonaws.com/basically_me_images/seed/wire-structure.svg) * Packet Header - Total length - Packet Header Length - Protocol Version - Number of messages - Instrument - Sequence number of first message - Sending time * Message Header - Header Length - Message Length - Type - Sequence number is implied (packet + 0) * Message-specific Body * Message Header - Header Length - Message Length - Type - Sequence number is implied (packet + 1) * Message-specific Body * Message Header - Header Length - Message Length - Type - Sequence number is implied (packet + N) * Message-specific Body ### Packet Header Every market data UDP packet will begin with this packet header. While multiple messages may be combined into a single packet to save bandwidth, they will always pertain to the same instrument. |Name|Type|Byte Length|Encoding|Description| |--- |--- |--- |--- |--- | |Total Length|Int|2|16 bit unsigned|Total length of all payloads in the packet, including this header| |Message Count|Int|2|16 bit unsigned|Number of messages that are included in this packet, and immediately following the Packet Header| |Protocol Version|Int|1|8 bit unsigned|Version of the marketdata protocol.<br/> Be sure to honor the length fields included in headers, in case message length increases between versions.| |Reserved|Int|3|8 bit unsigned|Reserved for future use| |Instrument ID|Int|8|64 bit unsigned|Seed CX Instrument ID, defined in reference data feed| |Sequence Number|Int|8|64 bit unsigned|Sequence number of message immediately following the packet header| |Sending Time|UTC Timestamp|8|64 bit unsigned|Time of packet transmission Nanoseconds since UNIX epoch UTC| #### Sequence Number As described in the [Reading incremental updates from multicast](#Reading-incremental-updates-from-multicast) section, the next expected sequence number can be calculated by adding the message count to packet header's sequence number. :::info Next Expected Sequence Number = PacketHeader.SequenceNumber + PacketHeader.MessageCount ::: #### Heartbeat A packet header with a message count of zero represents a heartbeat. Heartbeats never increment the sequence number, and their sequence effectively represents the next expected sequence number. Heartbeat messages are sent if there was no activity for a given instrument for at least `1 second`. ### Message Header Message Header describes the type of message that follows it. Please note that the header size may grow in the future protocol versions. Applications should ensure that they read lengths specified in headers. |Name|Type|Byte Length|Encoding|Description| |--- |--- |--- |--- |--- | |Message Length|Int|2|16 bit unsigned|Length of the message, including this header| |Message Type|Int|1|8 bit unsigned|<ol start="0"><li>Clear Book</li><li>Add Order</li><li>Replace Order</li><li>Delete Order</li><li>Trading Status</li><li>Trade</li><li>Trade Break</li><li>Session End</li></ol>| |Reserved|Int|5|8 bit unsigned|Reserved for future use| |Reserved|Int|8|64 bit unsigned|Reserved for future use| ### Add Order Message An Add Order message represents a newly accepted visible order on the Seed CX book. It includes a session-specific `Order ID` which will subsequently be used to refer to this order. |Name|Type|Byte Length|Encoding|Description| |--- |--- |--- |--- |--- | |Order ID|Int|8|64 bit unsigned|Unique order identifier for the current session| |Price|Int|8|64 bit signed|Order price| |Size Remaining|Int|8|64 bit unsigned|Order size resting in the book| |Side|Int|1|8 bit unsigned|<ol start="0"><li>Bid</li><li>Ask</li></ol>| |Reserved|Int|7|8 bit unsigned|Reserved for future use| ### Replace Order Message Replace Order messages represent a modification of the order state. They refer to an `Order ID` previously sent via [Add Order](#Add-Order-Message), [Instrument Snapshot](#Snapshot-Success-Response), or a Replace Order message. Orders whose remaining size reaches zero should be removed from the book. |Name|Type|Byte Length|Encoding|Description| |--- |--- |--- |--- |--- | |Original Order ID|Int|8|64 bit unsigned|Previously assigned unique order identifier| |New Order ID|Int|8|64 bit unsigned|New unique order identifier which will be used to refer to this order| |Price|Int|8|64 bit signed|Order price| |Size|Int|8|64 bit unsigned|Order size| |Lost Priority|Int|1|8 bit unsigned|<ol start="0"><li>Did not lose priority</li><li>Lost priority</li></ol>| |Reserved|Int|7|8 bit unsigned|Reserved for future use| ### Delete Order Message Delete Order messages are sent when an order is removed from the order book. This may occur due to a cancellation or when an order has been completely filled. The `Order ID` field refers to an order identifier previously sent via [Add Order](#Add-Order-Message), [Instrument Snapshot](#Snapshot-Success-Response), or a [Replace Order](#Replace-Order-Message) message. |Name|Type|Byte Length|Encoding|Description| |--- |--- |--- |--- |--- | |Order ID|Int|8|64 bit unsigned|Unique order identifier for the current session| ### Trade Message Trade messages are sent whenever an order on Seed CX has been executed in whole or in part. Applications that simply build a full depth order book can safely ignore Trade Break messages. |Name|Type|Byte Length|Encoding|Description| |--- |--- |--- |--- |--- | |Execution ID|Int|8|64 bit unsigned|Execution identifier for the trade| |Price|Int|8|64 bit signed|Execution price| |Size Executed|Int|8|64 bit unsigned|Executed order size| |Reserved|Int|8|64 bit unsigned|Reserved for future use| ### Trade Break Message The Trade Break message is sent whenever an execution on Seed CX is busted. Trade breaks are very rare. Trade Break immediately followed by a new [Trade Message](#Trade-Message) with the same `Execution ID` indicates a trade correction. Again, applications that simply build a full depth order book can safely ignore Trade Break messages. |Name|Type|Byte Length|Encoding|Description| |--- |--- |--- |--- |--- | |Execution ID|Int|8|64 bit unsigned|Execution identifier of the broken/cancelled trade| ### Trading Status Changed Message Trading Status Changed messages are sent whenever an instrument's trading status is updated. This may happen due to the regular exchange schedule, or unforeseen events such as a market halt. |Name|Type|Byte Length|Encoding|Description| |--- |--- |--- |--- |--- | |Trading Status|Int|1|8 bit unsigned|<ol start="0"><li>Closed</li><li>Available</li><li>Opening Auction</li><li>Open</li><li>Pre-Closed</li><li>Halted</li></ol>| |Reserved|Int|7|8 bit unsigned|Reserved for future use| ### Clear Book Message This message type is sent to indicate that the order book for this instrument should be cleared. All cached orders need to be purged. A Clear Book message is usually sent at the beginning and the end of the trading session, as well as during a system fail-over. `The message body is empty.` ### Session End Message This message type is sent when an instrument's trading session has ended. Reset an instrument's sequence numbers to zero. `The message body is empty.` --- ## TCP Binary Snapshot Protocol A full list of an instrument's resting orders can be obtained by establishing a TCP connection to the *Market Snapshot Service* and sending an Instrument Snapshot Request. This section describes the snapshot request as well as the possible responses. Note that snapshots are transmitted using TCP, which is a stream-oriented protocol. Applications should pay attention to lengths specified within headers and messages to ensure successful data processing. ### Instrument Snapshot Request This message should be transmitted by client applications in order to request a full depth market snapshot for a given instrument. Please note that snapshot requests are rate-limited. The current limit is `10` requests per participant per second. |Name|Type|Byte Length|Encoding|Description| |--- |--- |--- |--- |--- | |Length|Int|2|8 bit unsigned|Length of the request| |Message Type|Int|1|8 bit unsigned|20. Snapshot request| |Protocol Version|Int|1|8 bit unsigned|Protocol version. Must be set to `1`.| |Sender Comp ID|Char|12|ASCII|Seed-assigned `SenderCompId` used for authentication and rate limiting. It must be left-justified and padded with `0x00` to the right| |Instrument ID|Int|8|64 bit unsigned|Seed Instrument ID| ### Snapshot Failed Response Instrument snapshot request has failed. Depending on the failure reason, application may choose to re-try. |Name|Type|Byte Length|Encoding|Description| |--- |--- |--- |--- |--- | |Total Length|Int|4|32 bit unsigned|Total length of message| |Message Type|Int|1|8 bit unsigned|21. Snapshot Failed Response| |Protocol Version|Int|1|8 bit unsigned|Version of the marketdata protocol.<br/> Be sure to honor the length fields included in headers, in case message length increases between versions.| |Reserved|Int|2|8 bit unsigned|Reserved for future use| |Sending Time|UTC Timestamp|8|64 bit unsigned|Time of message transmission. Nanoseconds since UNIX epoch UTC| |Instrument ID|Int|8|64 but unsigned|Seed Instrument ID| |Reason|Int|1|8 bit unsigned|<ol start="0"><li>Malformed Request</li><li>Invalid Instrument ID</li><li>Snapshot Not Available</li><li>Invalid Credentials</li><li>Quota Exceeded</li><li>Unsupported protocol</li></ol>| |Reserved|Int|7|8 bit unsigned|Reserved for future use| ### Snapshot Success Response The snapshot request was successful, and the response contains the `as of` sequence number as well as the current trading status for the instrument. Upon successful snapshot, the response will be followed by [Add Order](#Add-Order-Message) messages which will be sorted from most advantageous to least advantageous. Their ordering reflects their position within matching engine's execution queue. |Name|Type|Byte Length|Encoding|Description| |--- |--- |--- |--- |--- | |Total Length|Int|4|32 bit unsigned|Total length of message, including the orders the orders that follow it| |Message Type|Int|1|8 bit unsigned|22. Snapshot Success Response| |Protocol Version|Int|1|8 bit unsigned|Version of the marketdata protocol.<br/> Be sure to honor the length fields included in headers, in case message length increases between versions.| |Reserved|Int|2|8 bit unsigned|Reserved for future use| |Sending Time|UTC Timestamp|8|64 bit unsigned|Time of message transmission. Nanoseconds since UNIX epoch UTC| |Instrument ID|Int|8|64 but unsigned|Seed Instrument ID| |Sequence Number|Int|8|64 bit unsigned|`As of` sequence number| |Trading Status|Int|1|8 bit unsigned|<ol start="0"><li>Closed</li><li>Available</li><li>Opening Auction</li><li>Open</li><li>Pre-Closed</li><li>Halted</li></ol>| |Reserved|Int|3|8 bit unsigned|Reserved for future use| |Order Count|Int|4|32 bit unsigned|Number of [Add Order](#Add-Order-Message) messages to follow this response. Note the Add Order messages will be sorted from most advantageous to least advantageous|