Hey all. Is anyone instrumenting their project wit...

# ktor

[Austin Pederson]

Hey all. Is anyone instrumenting their project with Open Telemetry? I'm seeing some issues with the datadog java agent so I'm trying Open Telemetry to see if it can resolve the issues. I am seeing issues with Open Telemetry where all my traces were being suppressed and not output to my local Zipkin. I changed the otel.instrumentation.experimental.span-suppression-strategy flag to "span-kind" and now I see traces in Zipkin. Curious if anyone has had a similar experience and/or can help point out why this behavior is occurring.

[AdamW]

I’m using Otel + Ktor (+Dynatrace), haven’t ran into many issues yet. Running Jaeger locally, perhaps it’s worth comparing the spans there with Zipkin 🤔

[Austin Pederson]

I tried using Jaeger to start, but the otel collector image doesn't support it out of the box, unfortunately. Are you not seeing the same issue with the spans being suppressed by default?

[AdamW]

Nope 🤔 Just to clarify, are you using the Otel Java agent or manual Ktor instrumentation with the plugin?

[Austin Pederson]

Otel Java agent

[AdamW]

This is a shot in the dark, but which engine are you running?

[Austin Pederson]

Netty for the server, OkHttp for the clients

[AdamW]

Here’s what I used to run Jaeger locally: https://gist.github.com/organize/3a7a25104a6de07ba10fa4e6d4bfacc2

If it’s feasible, could you verify that you’re also seeing the issue there?

[Austin Pederson]

I will see the issues either way I believe since the otel instrumentation is suppressing the spans. I checked by setting the log level to debug and reading the logs

[Austin Pederson]

I believe the issue lies in the instrumentation. Do you have any specific flags/configuration changes for otel instrumentation? I have it all setup as default currently minus that one flag

[AdamW]

The configuration I’m using is in the comment under that Gist, I’ve nothing else set up. I’m using version 1.34.1 for opentelemetry-bom, opentelemetry-api,opentelemetry-extension-kotlin and 1.32.1 for opentelemetry-instrumentation-annotations. Agent version is 1.32.0.

[Austin Pederson]

Ah so you are not using the automatic instrumentation the?

[AdamW]

I am, but I’m also creating spans manually from e.g. Exposed transactions, so I’ve made some helpers to work with coroutines 👍

[AdamW]

I omitted the agent VM option from the Gist if you meant that

[Austin Pederson]

Well this is quite interesting indeed. I am using a ton of concurrency in this project which maybe that causes some issues?

[AdamW]

As far as I’ve understood, since some-quite-recent-version ™️ of the Java agent, coroutine support comes OOTB. IIRC it has worked ever since Ktor auto-instrumentation was added. Previously, span annotations did not work properly with suspension.

I assume the Java agent comes with the Kotlin extension for the Ktor plugin to work, so that shouldn’t be the issue here.

[Austin Pederson]

I am using the latest Java agent. So I shouldn't need to do anything else in my repo and it should just work, correct?

[AdamW]

Yes. Can you try disabling Netty instrumentation?

[Austin Pederson]

Well oddly enough, it appears to just work toady with the defaults

[Austin Pederson]

Maybe I was just blind to some setting or something. I worked on this for many hours yesterday until later in the evening

[AdamW]

rubber duck 🤝

[Austin Pederson]

Do you know if there's a way to better name this stuff? I am seeing bad naming of my client spans as well as the rate limiter is showing up in my netty URLs lol

[Austin Pederson]

I am also not seeing the spans linked correctly to downstream spans in dd, but that is outside of the scope of this question I guess 😄