Fine-tune the volume of logs your app produces

If you’re running a production application, you know the challenges of managing logs. Too few logs and you’re flying blind; too many and you’re drowning in data and paying excessive storage costs. It’s a classic observability dilemma – you want comprehensive information when things go wrong, but you don’t want to store every detail from your happy paths.

Enter log sampling in .NET – a powerful capability that lets you strategically reduce log volume while maintaining observability. Unlike simple log filtering which uses binary decisions (emit or don’t emit logs), sampling gives you fine-grained control, letting you emit a precise percentage of logs from different parts of your application.

Get started

To get started, install the Microsoft.Extensions.Telemetry NuGet package:

dotnet add package Microsoft.Extensions.Telemetry

or add it directly in the C# project file:

<ItemGroup>
  <PackageReference Include="Microsoft.Extensions.Telemetry"
                    Version="*" />
</ItemGroup>

The challenge of logging at scale

As applications grow in complexity and traffic, logging strategies that worked well during development can quickly become problematic in production environments. Consider these common scenarios:

• Your application emits thousands of information-level logs per second, most providing redundant details about normal operations
• Your cloud logging costs increase dramatically as your user base grows
• During incidents, your ability to troubleshoot is hindered by a flood of routine log messages
• Latency increases as your application spends significant resources generating logs

Traditional approaches to these problems often involve simply turning off lower-level logs in production – but this creates blind spots. What if you could instead keep the same logging instrumentation but intelligently sample those logs?

Log sampling to the rescue

Log sampling extends filtering capabilities by giving you more fine-grained control over which logs are emitted. Instead of enabling or disabling entire categories of logs, sampling lets you emit a controlled percentage of them.

For example, while filtering typically uses probabilities like 0 (emit no logs) or 1 (emit all logs), sampling lets you choose any value in between, such as 0.1 to emit 10% of logs, or 0.25 to emit 25%.

.NET provides several sampling strategies out of the box:

1. Random probabilistic sampling: Sample logs based on configured probability rules
2. Trace-based sampling: Sample logs based on the sampling decision of the current trace
3. Custom sampling: Implement your own sampling strategy

Let’s walk through how to implement each approach.

Random probabilistic sampling: The simplest approach

Random probabilistic sampling is usually the easiest place to start. It allows you to define sampling rules based on log category, log level, or event ID.

For basic scenarios, you can configure a single probability value:

builder.Logging.AddRandomProbabilisticSampler(0.01, LogLevel.Information);

This configuration samples 1% of Information logs.

For more complex scenarios, you can define specific sampling rules:

builder.Logging.AddRandomProbabilisticSampler(options =>
{
    // Sample 5% of logs with event ID 1001 across all categories
    options.Rules.Add(
        new RandomProbabilisticSamplerFilterRule(
            probability: 0.05d,
            eventId : 1001));
});

Change the sampling rate dynamically

Random probabilistic sampling supports runtime configuration updates via the <xref:Microsoft.Extensions.Options.IOptionsMonitor%601> interface. If you’re using a configuration provider that supports reloads—such as the File Configuration Provider – you can update sampling rules at runtime without restarting the application. For example, depending on how your application is currently behaving in production, you can increase sampling rate or even enable sampling for all logs in case you need more logs to help with diagnostics. After diagnostics and troubleshooting, you can revert to the original sampling rate – again, all without restarting the application!

Trace-based sampling: Keeping logs and traces in sync

When using distributed tracing, it’s often valuable to ensure that logs and traces are sampled consistently. Trace-based sampling ensures that logs are only emitted if the underlying trace is being recorded:

builder.Logging.AddTraceBasedSampler();

This approach is particularly useful in microservice architectures where you want to maintain correlation between traces and logs.

Going fancy: Implementing your own sampling strategy

For specialized requirements, you can create a custom sampler by deriving from the abstract class.
For example, this is how a rate limiting sampler allowing no more than 1 log record per second might look like:

internal sealed class RateLimitingSampler : LoggingSampler
{
    private readonly Stopwatch stopwatch = Stopwatch.StartNew();
    private readonly long maxBalance = Stopwatch.Frequency;
    private long currentBalance;

    public RateLimitingSampler()
    {
        currentBalance = stopwatch.ElapsedTicks - maxBalance;
    }
    public override bool ShouldSample<TState>(in LogEntry<TState> logEntry)
    {
        long currentTicks;
        long currentBalanceTicks;
        long availableBalanceAfterWithdrawal;
        do
        {
            currentBalanceTicks = Interlocked.Read(ref currentBalance);
            currentTicks = stopwatch.ElapsedTicks;
            var currentAvailableBalance = currentTicks - currentBalanceTicks;
            if (currentAvailableBalance > maxBalance)
            {
                currentAvailableBalance = maxBalance;
            }

            availableBalanceAfterWithdrawal = currentAvailableBalance - Stopwatch.Frequency;
            if (availableBalanceAfterWithdrawal < 0)
            {
                return false;
            }
        }

        while (Interlocked.CompareExchange(ref currentBalance, currentTicks - availableBalanceAfterWithdrawal, currentBalanceTicks) != currentBalanceTicks);
        return true;
    }
}

Register it:

builder.Logging..AddSampler<RateLimitingSampler>();

And voilà! You have a custom log sampler that allows only one log record per second!

Performance considerations

The sampling logic introduces a minimal amount of CPU overhead in the application. However, whenever a specific log record is eliminated, it saves CPU cycles, transmission costs, storage costs.
We expect that for most situations, the savings or costs will be in the noise. But as always for performance issues, it’s always safer to measure and ensure there are no negative effects.

Summary

Log sampling gives you a powerful middle path between capturing every log (expensive) and capturing only high-level logs (insufficient detail). By strategically reducing the volume of routine logs while ensuring critical information is always captured, you can significantly reduce storage costs without sacrificing observability.

Whether you use random sampling, trace-based sampling, or a custom implementation, the ability to fine-tune your logging volume is a valuable tool in your observability toolkit.

References

• Log Sampling Documentation
• OpenTelemetry .NET RateLimiter

The post Fine-tune the volume of logs your app produces appeared first on .NET Blog.