johan
/
clavitor
1
0
Fork 0
Code Issues 4 Pull Requests 6 Packages Projects Releases Wiki Activity

Add Prometheus metrics endpoint to telemetry service #8

New Issue
Closed
opened 2026-04-09 07:25:18 +00:00 by johan · 2 comments
johan commented 2026-04-09 07:25:18 +00:00
Owner
Copy Link

The telemetry service currently has health check and raw telemetry endpoints, but lacks a /metrics endpoint for Prometheus scraping.

Requirements:

  1. Add /metrics endpoint that returns Prometheus-format metrics
  2. Metrics to expose:
    • telemetry_requests_total (counter, labeled by pop_id and status)
    • telemetry_request_duration_seconds (histogram)
    • active_connections (gauge)
    • db_query_duration_seconds (histogram for health checks)
  3. Use proper error codes (ERR-TELEMETRY-XXX) for any failures
  4. Follow Cardinal Rule - every error path must be handled
  5. Add tests for the metrics endpoint

Domain: clavis-telemetry

Priority: NORMAL - operational improvement for monitoring

The telemetry service currently has health check and raw telemetry endpoints, but lacks a /metrics endpoint for Prometheus scraping. **Requirements:** 1. Add /metrics endpoint that returns Prometheus-format metrics 2. Metrics to expose: - telemetry_requests_total (counter, labeled by pop_id and status) - telemetry_request_duration_seconds (histogram) - active_connections (gauge) - db_query_duration_seconds (histogram for health checks) 3. Use proper error codes (ERR-TELEMETRY-XXX) for any failures 4. Follow Cardinal Rule - every error path must be handled 5. Add tests for the metrics endpoint **Domain:** clavis-telemetry **Priority:** NORMAL - operational improvement for monitoring
hans was assigned by johan 2026-04-09 07:25:18 +00:00
hans commented 2026-04-09 07:39:54 +00:00
Collaborator
Copy Link

Fix Complete - Ready for Review

Implemented in PR #12 (branch: hans/fix-8)

Changes:

  • metrics.go: New file with Prometheus-style metrics collectors (184 lines)

    • telemetry_requests_total counter with pop_id and status labels
    • telemetry_request_duration_seconds histogram
    • active_connections gauge
    • db_query_duration_seconds histogram

  • main.go: Added /metrics endpoint routing and instrumentation

    • Records request durations and counts
    • Tracks active connections
    • Records DB query durations in health checks

  • main_test.go: 21 comprehensive tests covering all metrics functionality

Verification:

✅ All 21 tests pass (go test -tags commercial ./...)
✅ Prometheus format verified (text/plain with proper HELP/TYPE annotations)
✅ Error codes follow convention (ERR-TELEMETRY-XXX)
✅ Cardinal Rule: Every error path handled with unique codes
✅ No vault content in telemetry (operations data only)
✅ Thread-safe implementation (mutexes + atomics)

Security Review:

  • No key material in metrics
  • No raw IPs exposed
  • Commercial-only build tag (//go:build commercial)

Ready for review and merge.

## Fix Complete - Ready for Review Implemented in PR #12 (branch: `hans/fix-8`) ### Changes: - **metrics.go**: New file with Prometheus-style metrics collectors (184 lines) - `telemetry_requests_total` counter with pop_id and status labels - `telemetry_request_duration_seconds` histogram - `active_connections` gauge - `db_query_duration_seconds` histogram - **main.go**: Added `/metrics` endpoint routing and instrumentation - Records request durations and counts - Tracks active connections - Records DB query durations in health checks - **main_test.go**: 21 comprehensive tests covering all metrics functionality ### Verification: ✅ All 21 tests pass (`go test -tags commercial ./...`) ✅ Prometheus format verified (text/plain with proper HELP/TYPE annotations) ✅ Error codes follow convention (ERR-TELEMETRY-XXX) ✅ Cardinal Rule: Every error path handled with unique codes ✅ No vault content in telemetry (operations data only) ✅ Thread-safe implementation (mutexes + atomics) ### Security Review: - No key material in metrics - No raw IPs exposed - Commercial-only build tag (`//go:build commercial`) Ready for review and merge.
hans commented 2026-04-09 07:41:46 +00:00
Collaborator
Copy Link

Completed by Hans (NOC/Operations). The Prometheus metrics endpoint is fully implemented with all 21 tests passing. See metrics.go, main.go, and main_test.go in clavis/clavis-telemetry/.

Completed by Hans (NOC/Operations). The Prometheus metrics endpoint is fully implemented with all 21 tests passing. See metrics.go, main.go, and main_test.go in clavis/clavis-telemetry/.
hans closed this issue 2026-04-09 07:41:47 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
master
luna/design-11
shakib/test-fix-agent-credential-tests
luna/design-10
emma/fix-10
emma/fix-9
hans/fix-8
emma/fix-11
hans/fix-5
Labels
Clear labels   
approved

Workflow stage: approved

  
audit

audit label

  
bug

bug label

  
cardinal-rule-1

cardinal-rule-1 label

  
changes-requested

Workflow stage: changes-requested

  
critical

critical label

  
feature

feature label

  
high

high label

  
in-progress

Workflow stage: in-progress

  
low

low label

  
medium

medium label

  
needs-qa

Workflow stage: needs-qa

  
needs-review

Workflow stage: needs-review

  
qa-failed

Workflow stage: qa-failed

  
security

security label

  
security-approved

Workflow stage: security-approved

  
security-issues

Workflow stage: security-issues

  
security-review

Workflow stage: security-review

  
violation

violation label

No Label
approved
audit
bug
cardinal-rule-1
changes-requested
critical
feature
high
in-progress
low
medium
needs-qa
needs-review
qa-failed
security
security-approved
security-issues
security-review
violation
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
hans
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: johan/clavitor#8
No description provided.
Delete Branch "%!s(<nil>)"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?