Telemetry Infra
warning
🚧 Cortex.cpp is currently under development. Our documentation outlines the intended behavior of Cortex, which may not yet be fully implemented in the codebase.
Architecture​
The telemetry capability comprises several key components:
- Telemetry Collector: This component gathers telemetry data from the Cortex and exports it to other observability tools for further analysis and monitoring.
- Jan Telemetry Server: This component ingests telemetry data from Cortex. It processes this data and forwards events to Google Logging for further handling.
- Google Logging: This component receives events from the Jan Telemetry Server. It stores and manages these logs and exports them to BigQuery.
- Google BigQuery: This component stores the exported logs from Google Logging.
- Metabase: This component connects to Google BigQuery to perform queries and visualize the data. It provides an interface for analyzing and presenting the telemetry data.
Crash Report Schema​
{ "$schema": "http://json-schema.org/draft-07/schema#", "definitions": { "StringValue": { "type": "object", "properties": { "stringValue": { "type": "string" } }, "required": ["stringValue"] }, "ObjectValue": { "type": "object", "properties": { "kvlist_value": { "type": "object", "properties": { "values": { "type": "array", "items": { "type": "object", "properties": { "key": { "type": "string" }, "value": { "$ref": "#/definitions/StringValue" } }, "required": ["key", "value"] } } }, "required": ["values"] } }, "required": ["kvlist_value"] }, "Attribute": { "type": "object", "properties": { "key": { "type": "string" }, "value": { "oneOf": [ { "$ref": "#/definitions/StringValue" }, { "$ref": "#/definitions/ObjectValue" } ] } }, "required": ["key", "value"] }, "TelemetryLog": { "type": "object", "properties": { "traceId": { "type": "string" }, "startTimeUnixNano": { "type": "string" }, "endTimeUnixNano": { "type": "string" }, "severityText": { "type": "string" }, "body": { "type": "object", "properties": { "stringValue": { "type": "string" } }, "required": ["stringValue"] }, "attributes": { "type": "array", "items": { "$ref": "#/definitions/Attribute" } } }, "required": ["traceId", "severityText", "body", "attributes"] }, "ScopeLog": { "type": "object", "properties": { "scope": { "type": "object" }, "logRecords": { "type": "array", "items": { "$ref": "#/definitions/TelemetryLog" } } }, "required": ["scope", "logRecords"] }, "Resource": { "type": "object", "properties": { "attributes": { "type": "array", "items": { "$ref": "#/definitions/Attribute" } } }, "required": ["attributes"] }, "TelemetryEvent": { "type": "object", "properties": { "resource": { "$ref": "#/definitions/Resource" }, "scopeLogs": { "type": "array", "items": { "$ref": "#/definitions/ScopeLog" } } }, "required": ["resource", "scopeLogs"] } }, "type": "object", "properties": { "resourceLogs": { "type": "array", "items": { "$ref": "#/definitions/TelemetryEvent" } } }, "required": ["resourceLogs"]}
Implementation​
The diagram illustrates the implementation of handling crashes in the Cortex server environment:
- User Interaction
- The user sends a command or requests to the Cortex Server Main Process.
- Cortex Server Main Process
- The main process handles the initial command or request.
- It forks a child process to monitor and handle errors.
- To manage unexpected errors, the main process catches any
uncaughtException
orunhandledRejection
events.
- Child Process
- A child process is forked to perform periodic checks (interval pings) to ensure the main process is not hanging or unresponsive.
- This child process can also capture and log errors.
- Local Directory
- If a crash is detected, the error details and crash reports are inserted into a local directory for further analysis and record-keeping.
info
Learn more about Telemetry: