Hello, I posted awhile back about an event we were...

# ktor

[Scott Rankin]

Hello, I posted awhile back about an event we were encountering in production and I've finally had a chance to distill down a minimal repro. Sources can be found here [1].

The high level gist is when we enable histogram metrics it changes ktor_http_server_requests_seconds from TYPE summary to histogram but the ktor_http_server_requests_seconds does not contain the le label so our aws-otel-collector produces a warning.

Is this a bug with the MicrometerMetrics plugin or ktor or in the prometheus receiver of the otel collector?

Here's a snippet of what I get when I hit /metrics on my example server with histograms enabled using a DistributionStatisticConfig,

# HELP ktor_http_server_requests_seconds
# TYPE ktor_http_server_requests_seconds histogram
ktor_http_server_requests_seconds{address="0:0:0:0:0:0:0:1:8080",method="GET",route="/metrics",status="200",throwable="n/a",quantile="0.5",} 1.409286144
ktor_http_server_requests_seconds{address="0:0:0:0:0:0:0:1:8080",method="GET",route="/metrics",status="200",throwable="n/a",quantile="0.9",} 1.409286144
ktor_http_server_requests_seconds{address="0:0:0:0:0:0:0:1:8080",method="GET",route="/metrics",status="200",throwable="n/a",quantile="0.95",} 1.409286144
ktor_http_server_requests_seconds{address="0:0:0:0:0:0:0:1:8080",method="GET",route="/metrics",status="200",throwable="n/a",quantile="0.99",} 1.409286144
ktor_http_server_requests_seconds_bucket{address="0:0:0:0:0:0:0:1:8080",method="GET",route="/metrics",status="200",throwable="n/a",le="0.001",} 0.0
ktor_http_server_requests_seconds_bucket{address="0:0:0:0:0:0:0:1:8080",method="GET",route="/metrics",status="200",throwable="n/a",le="0.001048576",} 0.0
ktor_http_server_requests_seconds_bucket{address="0:0:0:0:0:0:0:1:8080",method="GET",route="/metrics",status="200",throwable="n/a",le="0.001398101",} 0.0
ktor_http_server_requests_seconds_bucket{address="0:0:0:0:0:0:0:1:8080",method="GET",route="/metrics",status="200",throwable="n/a",le="0.001747626",} 0.0
...

[1] https://github.com/botscholar-scott/ktor-micrometer

[Scott Rankin]

Or a bug that ktor_http_server_requests_seconds is even sent from the scrape?

[Aleksei Tirman [JB]]

Most likely the problem is that Ktor sends those metrics by default and, unfortunately, which cannot be disabled. Can you please file an issue?

[Scott Rankin]

Filed. Please let me know if I can provide additional information etc.

https://youtrack.jetbrains.com/issue/IDEA-343609/Histogram-metric-ktorhttpserverrequestsseconds-missing-le-label