Event Bus Service

--- tags: feature author: Oliver.W, Meng --- # Event Bus Service > Ticket: https://app.asana.com/0/1200011502754281/1200402117339672/f > Event bus is the services that designed by global to support Metrics, Troubleshooting, Dev Mode, Integration with other monitoring services, custom metrics, log search, enable user write functions to response to runtime events. ## Events Event bus records the events that from components CLI and platform, this events records help to do future analytics. ## Architecture Design ![](https://i.imgur.com/iUBsYGE.png) To accomplete the data collection and generate future event analytics report, we will follow the architecture design and extend the services already exist to reuse the services already exist. * Update and deploy the latest event bus services into Tencent to collect the event data and support future event features (eg: logs, dev mode) * Reuse the event integrations services to clean and transform the event data before send to mixpa * All the raw event data are save in Event Bus, compare to the dynamo DB used in global, we will use redis to save those data. * There will have extra events that to track for China users, and all the events are listed here. ## Events ### Platform Events This will tack the events that send from Engine and other serverless platforms, this is used to track the commands or actions executed in serverless platform and its result #### Field details | Property | Datatype | Description | | -------------------------- | --------- | --------------------------------------------------------------------------------------------------- | | `event` | `string*` | Serverless Platform Event type - follows a `domain.object.action` naming convention | | `org_uid` | `string` | ID of org this event falls under | | `user_uid` | `string` | ID of user triggering the event, if applicable for event type | | `client_uid` | `string` | ID of the client. Genereated when first time init | | `created` | `number*` | Event creation time in epoch milliseconds | | `data` | `object*` | Event payload | | `data.object` | `object` | Serverless Platform Object targetted by event | | `data.previous_attributes` | `object` | Only set on `*.updated` event types and contains the properties with values as of before the update | - [x] service.action.succeeded: Emitted when a Service successfully finishes a deploy or remove and etc. | Property | Datatype | Description | | ------------------- | -------- | ------------------------------------------------------------------------------------- | | `org_name` | `string` | The name of the organization. | | `user_uid` | `string` | The UID of the user who performed the Action. | | `action` | `string` | The name of the Action run, like `deploy`, `remove`, `metrics`, etc. | | `app_name` | `string` | The Application name that contains the Service that the Action was performed against. | | `service_name` | `string` | The name of the Service that the Action was performed against. | | `stage_name` | `string` | The name of the Stage of the Service that the Action was performed against. | | `component_name` | `string` | The name of the Component that is being used by the Service. | | `component_version` | `string` | The version of the Component that is being used by the Service. | ```json= { "event": "service.action.succeeded", "org_uid": "1302363968", "user_uid": "1302363968", "data": { "object": { "created": 1601308487731, "org_name": "serverlessinc", "user_uid": "1302363968", "action": "deploy", "app_name": "scf-77e121db", "service_name": "api", "stage_name": "dev", "component_name": "express", "component_version": "2.1.4" }, } } ``` - [x] service.action.failed: Emitted when a Service successfully finishes a deploy or remove and etc. | Property | Datatype | Description | | ------------------- | -------- | ------------------------------------------------------------------------------------- | | `org_name` | `string` | The name of the organization. | | `user_uid` | `string` | The UID of the user who performed the Action. | | `app_name` | `string` | The Application name that contains the Service that the Action was performed against. | | `action` | `string` | The name of the Action run, like `deploy`, `remove`, `metrics`, etc. | | `service_name` | `string` | The name of the Service that the Action was performed against. | | `stage_name` | `string` | The name of the Stage of the Service that the Action was performed against. | | `component_name` | `string` | The name of the Component that is being used by the Service. | | `component_version` | `string` | The version of the Component that is being used by the Service. | | `error_type` | `string` | A categorization of the error to help understand it, like 'action_failed'. | | `error_message` | `string` | The error message. | ```json= { "event": "service.action.failed", "org_uid": "1302363968", "user_uid": "1302363968", "created": 1601308487731, "data": { "object": { "org_name": "serverlessinc", "user_uid": "Lcd7gvBhCnTGzHkP3V", "action": "deploy", "app_name": "fullstack-app", "service_name": "api", "stage_name": "dev", "component_name": "express", "component_version": "2.1.4", "error_type": "action_failed", "error_message": "credentials not found", }, } } ``` - [ ] service.deleted: Emitted when a Service record is removed. | Property | Datatype | Description | | -------------- | -------- | ------------------------------------------------------------------------ | | `object` | `string` | `service` | | `user_uid` | `string` | The UID of the user who deleted the Service. | | `app_name` | `string` | The Application UID that contains the Service that was deleted | | `app_uid` | `string` | The Application UID that contains the Service that was deleted. | | `service_name` | `string` | The name of the Service that was deleted. | | `org_uid` | `string` | The uid of the Organization which contains the service that was deleted. | ```json= { "event": "service.deleted", "org_uid": "1302363968", "user_uid": "1302363968", "created": 1601308487731, "data": { "object": { "org_uid": "Lcd7gvBhCnTGzHkP3V", "user_uid": "Lcd7gvBhCnTGzHkP3V", "app_name": "fullstack-app", "app_uid": "abcd-1234", "service_name": "api", }, } } ``` > 没有找到global相关事件逻辑. ### Components CLI Events The components CLI will send event of each command that execute by users, and all the commands will send to the metrics service | Property | Datatype | Description | | | ------------------------ | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --- | | `timezone` | `string` | Local timezone of user performing the deployment. It uses IANA names, e.g. `Europe/Warsaw` or `America/New_York`. It is available only if `serverless_version` is higher or equal to `2.30.0` or higher or equal to `1.83.3` (for 1.x versions) | | | `components_cli_version` | `string` | Version of Components CLI. | | | `components` | `array` | The arraylist of components used in the project. | | | `provider_name` | `string` | Name of the provider used for deployment. | | | `provider_stage` | `string` | Stage used for deployment. | | | `provider_runtimes` | `array` | Array of runtimes used during deployment. In most cases it will have only one item. Since `2.34.0` version of Serverless Framework, it can also include meta-runtime `$containerimage`, which is set for functions that use `image` instead of `handler` (they use Docker containers). | | | `provider_region` | `string` | Name of the region used for deployment. | | | `user_uid` | `string` | The appid of user | | | `client_uid` | `string` | The unique client id generated | | | `functions_count` | `number` | Number of functions from service configuration used during deployment | | | `events_count` | `number` | Number of events from service configuration used during deployment | | | `event_types` | `array` | Array of unique event types used during deployment | | | `outcome` | `string` | Signifies if command failed or succeded. Can have `success` or `failure` value. | | | `failure_reason` | `string` | error messgae | | | `failure_step` | `string` | The step of error occurs | | | `failure_source` | `string` | The source of error occurs | | | `failure_code` | `string` | The code of errors | | | `requestId` | `string` | The request id of the error request, it's helpful in SCF log | | | `traceId` | `string` | The trace id of the error request, it's helpful in our kibana log system | | | `ci_name` | `string` | Name of the CI engine used for deployment. If it's equal to `untracked`, it means that for that event we were not detecting CI engine at all. In most cases, `ci_name` equal to `untracked` will mean that the deployment wasn't executed in CI environment as previously (versions earlier than `2.30.0` and earlier than `1.83.3` for 1.x versions) we were discarding events coming from CI. If `ci_name` is equal to `undefined`, it means that based on the library that we're using for CI detection, the deployment was not run in CI engine. If `ci_name` is equal to `unknown`, it means that the deployment was executed in CI engine environment, but we weren't able to detect the name of tha specific CI engine. In all other cases, `ci_name` will be equal to CI engine name such as `Circle CI` or `Github Actions` | | - [x] init ```json= { "event":"components.command.init", "org_uid": "Lcd7gvBhCnTGzHkP3V", "user_uid": "z082jZGnkYfxNRNtl0", "client_uid": "uuid", "created": 1601308487731, "data":{ "object":{ "provider_name": "aws", "provider_stage": "prod", "provider_region": "us-east-1", "provider_runtimes": ["python3.8"], "functions_count": 5, "events_count": 5, "event_types": ["http", "sqs"], "npm_dependencies": ["lodash", "middy"], "ci_name": "Github Actions", "timezone": "Europe/Warsaw", "user_id": "m62vrWszWT2TF51111", "user_uid": "m62vrWszWT2TF51111", "step": "获取组件信息", "code": "API_FAILED", "source": "Serverless::Registry", "referral": "www.serverless.com", "requestID": "jid83msdfs-38fdsd-k4j8", "traceId": "zld8-dskj-82ffdsi" } } } ``` - [x] dev ```json= { "eventType":"components.command.dev", "action":"dev", "data":{ "object":{ "action":"dev", "org_name":"1302363968", "user_uid":1302363968, "app_name":"scf-77e121db", "service_name":"scf-nodejs", "stage_name":"dev", "component_name":"scf", "component_version":"0.2.0" }, "created":1623219828936 } } ``` - [x] logs ```json= { "event":"components.command.logs", "action":"logs", "data":{ "object":{ "action":"invoke", "user_uid":"1302363968", "app_name":"scf-77e121db", "stage_name":"dev", // region "component_name":"scf", "error_type":"action_error", "error_messgae":"[FAAS] 无法获取函数 CLS 配置，请检查函数是否配置 CLS 功能" }, "created":1623222952314 } } ``` - [x] invoke ```json= { "event":"components.command.invoke", "action":"invoke", "data":{ "object":{ "action":"invoke", "user_uid":"1302363968", "app_name":"scf-77e121db", "stage_name":"dev", // function // region "component_name":"scf" }, "created":1623222329967 } } ``` - [x] invoke local ```json= { "eventType":"components.command.invoke_local", "action":"invoke_local", "data":{ "object":{ "action":"invoke_local", "app_name":"scf-77e121db", "service_name":"scf-nodejs", "stage_name":"dev", "component_name":"scf" }, "created":1623376745462 } } ``` - [x] deploy ```json= { "event":"components.command.deploy", "action":"deploy", "data":{ "object":{ "org_name":"1302363968", "user_uid":1302363968, "action":"deploy", "app_name":"scf-77e121db", "service_name":"scf-nodejs", "stage_name":"dev", "component_name":"scf", "component_version":"0.2.0" }, "created":1623219820451 } } ``` - [x] remove ```json= { "event":"components.command.remove", "action":"remove", "data":{ "object":{ "org_name":"1302363968", "user_uid":1302363968, "action":"remove", "app_name":"scf-77e121db", "service_name":"scf-nodejs", "stage_name":"dev", "component_name":"scf", "component_version":"0.2.0" }, "created":1623219820451 } } ``` #### Tracked Events --- - [ ] `user.created` : Emitted when a new user is created.https://github.com/serverlessinc/platform-dashboard/blob/438bc1fad85d509de2724750a482755bbf897147/src/webhooks/constants.js#L4 这里面定义了一些events, **user.created** 这个event也是在dashboard中才使用 > This is controlled by Tencent Cloud, we cant easily track this and tell when a new user created. but we can get the new user data by run analytic script. * 复杂, 不考虑腾讯通知方式, 当前中国版本无法得知一个用户是否是新的用户，除非查数据库然后判断数据库中是否有此用户记录，然后判断是否是新用户。此操作代价较大，不可行。 * (用户) 潜在用户(下载) - 创建 - 部署 - 成功 - 2次部署(人周部署次数) - 5次部署 * (用户) 潜在用户(下载) - 当周部署的 - 第二周部署的 * CLI 下载之后, 生成UUID. 并记录在全局配置中 (mac) * 在进行非登陆的操作时候(提交uuid), 默认记录为 -》匿名用户 - 交互时间 * 在部署的时候(提交uuid, appid) - 创建时间(已有的用户使用最早的时间作为创建时间,否则使用当前时间) 1. 下载CLI global 变量记录 uuid 2. 非登陆操作提交uuid, 只存储uuid,以及第一次时间. 下载时间. * uuid - timestamp. (uuid已经存在, 不更新.) ``` { 'user.downloaded: { [uuid]: { downloadAt: 111, } } } ``` 3. 登陆操作关联appid, 用户创建时间. ``` { 'user.downloaded: { [uuid1|appid]: { createdAt: timestamp downloadAt: 111, } [uuid2|appid]: { createdAt: timestamp downloadAt: 222, } } } ``` 4. { 'user.downloaded: { [uuid]: { appid: xxx, createdAt: 1111, downloadAt: 111, } } } - [ ] `interact.action.succeeded` : Emitted when a user successfully runs an 'action' in the interact feature.https://github.com/serverlessinc/platform-dashboard/blob/438bc1fad85d509de2724750a482755bbf897147/src/studio/components/Invoker.js#L349 > we dont need to track this event at this moment - [ ] `interact.action.failed` : Emitted when a user runs an 'action' in the interact feature and it results in an error.https://github.com/serverlessinc/platform-dashboard/blob/438bc1fad85d509de2724750a482755bbf897147/src/studio/components/Invoker.js#L368 > we dont need to track this event at this moment - [x] `service.action.succeeded` : Emitted when a Service successfully finishes a deploy or remove. - [x] `service.action.failed` : Emitted when a Service fails to finish a deploy or remove. ## 设计文档 ### DB object ```json { [EventType]: { [Action]: EventData[] } } e.g: { 'service.action.succeeded': { deploy: [ { object: { org_name: xxx, // orgName, 此字段无用处，如果用户不指定，默认为appid user_uid: xxx, // 用户客户端唯一id user_id: xxx, // 用户appid action: deploy | remove, // 用户执行的动作 app_name: xxxx, // appName service_name: xxx, // instanceName stage_name: xxx, // stageName component_name: xxx, // component name component_version: 1.0.0, // component version }, created: 1111, //timestamp } ] }, 'service.action.failed': { deploy: [ { object: { org_name: xxx, // orgName, 此字段无用处，如果用户不指定，默认为appid user_uid: xxx, // 用户appid action: deploy |remove, // 用户执行的动作 app_name: xxxx, // appName service_name: xxx, // instanceName stage_name: xxx, // stageName component_name: xxx, // component name component_version: '1.0.0', // component version error_type: xxxx, // 错误的类型 error_message: xxxx, // 错误详情 }, created: 1111, //timestamp } ] } } ``` ### 自定义通知数据结构 EventData ```json { event: 'service.action.(succeeded/failed)', action: xxx, //用户的行为 data: { created: 111, //timestampe object: { action: deploy / remove / etc..., // 用户的行为 [key]: value // 每个行为对应的数据，不固定 } } } ``` ### 自定义事件 : 客户端发送通知统计自定义事件: ### Event bus service Apis 1. dev 环境 2. prod 环境 ### API https://docs.qq.com/doc/DRlFuQ0RERGtRdGd0 --- ## Debug 1. CLI 建立websocket通道: https://github.com/serverless/components/blob/b3a14f4fb9ffe31f54b1c34463370e813a276042/src/cli/commands-cn/run.js#L96 2. registry/handler 中对console进行劫持，发送到event 服务中：https://github.com/serverlessinc/platform-cn/blob/8e68bb450a91540acd7d2181daf44962c569106b/sp-registry/layer/src/handler.js#L411 3. Event 服务把接受到的console发来的消息，通过websocket再返回到cli: https://github.com/serverlessinc/platform-cn/blob/8e68bb450a91540acd7d2181daf44962c569106b/sp-events/code/src/handler.js#L81 ## Dev Mode Current we implement the dev mode based on the socket abilities from Tencent SCF. ### 实现模式 1. dev.js 中，通过部署代码之后的返回结果，调用腾讯云的日志接口，建立websocket通道，接受实时日志： https://github.com/serverless/components/blob/b3a14f4fb9ffe31f54b1c34463370e813a276042/src/cli/commands-cn/dev.js#L271 2. 第一步调用的实际上是sdk中封装的腾讯云的日志接口: 1. https://github.com/serverlessinc/platform-cn/blob/8e68bb450a91540acd7d2181daf44962c569106b/sdk/src/utils.js#L572 实例化腾讯云日志实例 2. 调用腾讯云debug实例中的接口: https://github.com/serverlessinc/platform-cn/blob/8e68bb450a91540acd7d2181daf44962c569106b/sdk/src/utils.js#L574, 建立了ws 连接，由腾讯云日志接口负责实时日志推送 > Based on the events list https://github.com/serverlessinc/company/tree/main/docs/analytics#serverless-platform-events --- ## 相关实现逻辑 ### Components CLI 1. 生成客户端 client_uid(~/.serverless/tencent/client_uid-credentials) 1. 如果当前没有 **client_uid-credentials** 文件，会先根据 uuidv1生成一个对应的值和一个当前的时间，作为此 **client_uid** 生成的时间 ``` [uuid] value=e2d6eda0-d2fc-11eb-b6c7-19e7af884fd1 downloadAt=1624326914426 ``` 2. 生成之后，将生成的**client_uid** 存到云数据库redis 中,格式: ```json { "client:e2d6eda0-d2fc-11eb-b6c7-19e7af884fd1": { "client_uid":{"value":"e2d6eda0-d2fc-11eb-b6c7-19e7af884fd1","downloadAt":1624326914426} } } ``` 2. CLI 命令收集metrics 数据 1. 生成 metrics paylaod: `generatePayload` 2. 相关命令执行，并且根据命令的执行结果，修改 **payload** 的 **outcome**, 同时存储到本地文件中 **~/.serverless/tencent/telemetry-cache** 3. 对于 **deploy** 命令，有以下特殊逻辑： 1. 检查 **client_uid-credentials** 文件，检查当前 **appId** 是否存在，如果不存在，那么加上此 **appId**, 数据格式: ``` [uuid] value=e2d6eda0-d2fc-11eb-b6c7-19e7af884fd1 downloadAt=1624326914426 1302363968=true ``` 3. 同时传递**client_uid**到deploy参数中:**options.client_uid=xxxxx**. **engine/run.js** 检查到有此参数的话，会统计此 **appId** 的最早的实例，记录 **创建时间(createdAt)** 到 **client_uid** 的redis字段中, 这样就把appid和client_uid关联起来。格式: ```json { "client:e2d6eda0-d2fc-11eb-b6c7-19e7af884fd1": { "client_uid":{"value":"e2d6eda0-d2fc-11eb-b6c7-19e7af884fd1","downloadAt":1624326914426}, "1302363968(appId)": 1624331617147(createdAt), } } ``` 3. **deploy** 命令会把当前所有存在本地文件中 **~/.serverless/tencent/telemetry-cache** 的数据发送到 **Metircs** 服务器，有服务端进行数据的整合和转发, 发送成功的会，会删除当前文件下的所有统计数据，清空缓存 --- ## Analytics Redis实例为了存储统计数据，并且和业务的数据存储分离开，所以在dev和prod分别新建了新的redis实例，用来储存统计数据 1. dev: https://console.intl.cloud.tencent.com/redis/instance/manage/detail?instanceId=crs-ppn9awj1&regionId=4 2. prod: https://console.intl.cloud.tencent.com/redis/instance/manage/detail?instanceId=crs-10ydekbx&regionId=1 3. 在各环境的服务中，会配置如下统计数据库环境变量: ``` REDIS_ANALYTICS_HOST=***** REDIS_ANALYTICS_INSTANCE_ID=****** REDIS_ANALYTICS_PASSWORD=****** REDIS_ANALYTICS_PORT=**** ``` ### 到现在为止: 2021/06, 主要存储了如下类型的数据: 1. "client:*", 这一类是每个用户的**client_uid** 数据，包含 **匿名用户**(只有一个client_uid)和**关联用户**(用户的appID和client_uid关联起来) 2. "service.action.success/fail": 这是通过 `engine/run.js` 发送到 **events services** 的统计数据，统计用户action的结果和数据字段 3. "component.command.deploy": 这是通过 **components CLI** 进行的数据统计，统计的是命令行的数据，发送到 **metrics services**, 现在只存储 **deploy** 的action到数据库中 --- ## Event-integration server 1. Code repo: https://github.com/serverlessinc/event-integrations-cn 2. Ticket: https://app.asana.com/0/1200011502754281/1200502082991966/f 3. Dev SCF: https://console.intl.cloud.tencent.com/scf/list-detail?rid=4&ns=default&id=event-integration-cn-dev 4. Prod SCF: https://console.intl.cloud.tencent.com/scf/list-detail?rid=1&ns=default&id=event-integration-cn-prod&tab=codeTab

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.