refactor(stackable-telemetry)!: Improve stackable-telemetry to make it easier to build the TraceGuard

[NickLarsenNZ]

Description

Part of stackabletech/issues#639.

Problem

It is cumbersome to configured the various subscribers to be enabled at runtime. Using @soenkeliebau's code example:

// Instead of building a TraceGuard, a mutable builer needs to be assigned so that...
let mut builder = Tracing::builder()
    .service_name("whoyougonnacall")
    .with_console_output("WYGC_CONSOLE", LevelFilter::INFO);

// ... builder methods can be called conditionally:
if enable_trace_exporter().context(ParseConfigSnafu)? {
    builder = builder.with_otlp_trace_exporter("TEST_OTLP_TRACE", LevelFilter::TRACE);
}
if enable_log_exporter().context(ParseConfigSnafu)? {
    builder = builder.with_otlp_log_exporter("TEST_OTLP_LOG", LevelFilter::TRACE);
}

// Then finally the trace TraceGuard is returned.
let _tracing_guard = builder.build().init().context(InitializeTelemetrySnafu)?;

The purpose of the builder is to allow overriding of the the defaults, however in the current state, the function parameters for the subscribers (console and otlp exporters) break that, and it will only get messier as config options are added.

Solution

Add common and subscriber specific configuration through a Settings builder (suggested by @Techassi). The previous example would then look like:

let _trace_guard = Tracing::builder()
    .service_name("whoyougonnacall")
    .with_console_output(
        Settings::builder()
            .env_var("WYGC_CONSOLE")
            .default_level(LevelFilter::INFO)
            .enabled(true)
            // 👇 this part is new, and has no analogue in the example from the problem statement above.
            .console_log_settings_builder()
            .log_format(Format::Plain)
            // ☝️
            .build(),
    )
    .with_otlp_trace_exporter(
        Settings::builder()
            .env_var("TEST_OTLP_TRACE")
            .default_level(LevelFilter::TRACE)
            .enabled(enable_trace_exporter().context(ParseConfigSnafu)?)
            .build(),
    )
    .with_otlp_log_exporter(
        Settings::builder()
            .env_var("TEST_OTLP_LOG")
            .default_level(LevelFilter::TRACE)
            .enabled(enable_log_exporter().context(ParseConfigSnafu)?)
            .build(),
    )
    .build();

Definition of Done Checklist

• Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant
• Please make sure all these things are done and tick the boxes

# Author
- [ ] Changes are OpenShift compatible
- [ ] CRD changes approved
- [ ] Integration tests passed (for non trivial changes)

# Reviewer
- [ ] Code contains useful comments
- [ ] (Integration-)Test cases added
- [ ] Documentation added or updated
- [ ] Changelog updated
- [ ] Cargo.toml only contains references to git tags (not specific commits or branches)

[soenkeliebau]

I like it!

[NickLarsenNZ]

I like it!

Thanks.

It is basically done, there is one thing left to refactor (and enable console log formats), but before merging I will document the pub items (and enable warn on missing docs).

[NickLarsenNZ]

This PR was going to include the following items:

• complete docs
• tuples for quicker setup
• console json format output as an option

But they were dropped in order to unblock @xeniape for the RollingFileAppender implementation. I am promising to come back and do those things in a separate PR.

[nightkr]

Not really relevant since it's excluded from this PR, but the tuple variant smells a lot like "From magic" that I would rather avoid.