Skip to content

[dotnet] Enable flag eval metrics tests [dotnet@sameerank/FFL-1946/add-flag-eval-metrics] #6689

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
sameerank wants to merge 8 commits into
base: main
Choose a base branch
from
Open

[dotnet] Enable flag eval metrics tests [dotnet@sameerank/FFL-1946/add-flag-eval-metrics] #6689

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
e3c6fcc
fix(dotnet): Handle JsonElement deserialization in FFE weblog controller
sameerank Apr 4, 2026
7e77cb4
Enable dotnet flag eval metrics tests and fix weblog JsonElement hand…
sameerank Apr 4, 2026
c86bfde
Enable PARSE_ERROR and PROVIDER_NOT_READY tests for .NET
sameerank Apr 7, 2026
a38da24
Support local NuGet packages for .NET weblog testing
sameerank Apr 7, 2026
1b913b5
Mark tests/ffe/test_flag_eval_metrics.py as missing_feature for .NET
sameerank Apr 8, 2026
ef91458
Retrigger CI after title fix
sameerank Apr 8, 2026
855e812
Enable flag eval metrics tests for .NET v3.43.0-dev
sameerank Apr 22, 2026
15d88de
Retrigger CI with updated Docker image
sameerank Apr 24, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion manifests/dotnet.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -698,7 +698,7 @@ manifest:
tests/docker_ssi/test_docker_ssi_appsec.py::TestDockerSSIAppsecFeatures::test_telemetry_source_ssi: v3.36.0
tests/ffe/test_dynamic_evaluation.py: v3.36.0
tests/ffe/test_exposures.py: v3.36.0
tests/ffe/test_flag_eval_metrics.py: missing_feature
tests/ffe/test_flag_eval_metrics.py: v3.43.0-dev
tests/integration_frameworks/llm/anthropic/test_anthropic_llmobs.py::TestAnthropicLlmObsMessages::test_create_error: bug (MLOB-1234)
tests/integrations/crossed_integrations/test_kafka.py::Test_Kafka: v2.0.0-prerelease
tests/integrations/crossed_integrations/test_kinesis.py::Test_Kinesis_PROPAGATION_VIA_MESSAGE_ATTRIBUTES: missing_feature
Expand Down
6 changes: 6 additions & 0 deletions utils/build/docker/dotnet/poc.Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,12 @@
FROM mcr.microsoft.com/dotnet/sdk:8.0 AS build-app
WORKDIR /app

# Copy binaries folder for local NuGet packages (for testing with local builds)
COPY binaries/ /binaries/
RUN if ls /binaries/*.nupkg 1> /dev/null 2>&1; then \
dotnet nuget add source /binaries --name local-packages; \
fi

# dotnet restore
COPY utils/build/docker/dotnet/weblog/app.csproj app.csproj
RUN dotnet restore
Expand Down
79 changes: 70 additions & 9 deletions utils/build/docker/dotnet/weblog/Controllers/FeatureFlagEvaluatorController.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
using OpenFeature.Model;
using System;
using System.Collections.Generic;
using System.Text.Json;
using System.Text.Json.Serialization;
using System.Threading.Tasks;

Expand Down Expand Up @@ -52,24 +53,76 @@ public async Task<IActionResult> Evaluate([FromBody] EvaluateRequest request)
{
value = request.VariationType?.ToUpper() switch
{
"BOOLEAN" => await _client.GetBooleanValueAsync(request.Flag, Convert.ToBoolean(request.DefaultValue), context),
Copy link
Copy Markdown
Contributor Author

@sameerank sameerank Apr 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Problem

The EvaluateRequest class has a DefaultValue property typed as object:

public class EvaluateRequest
{
    public object DefaultValue { get; set; }  // Can be bool, int, string, etc.
}

When ASP.NET Core's System.Text.Json deserializes JSON into an object property, it does not convert to native .NET types. Instead, it wraps the value in a JsonElement:

{ "defaultValue": true }     // Becomes JsonElement, not bool
{ "defaultValue": 42 }       // Becomes JsonElement, not int
{ "defaultValue": "hello" }  // Becomes JsonElement, not string

The original code assumed native types:

Convert.ToBoolean(request.DefaultValue)  // Throws! DefaultValue is JsonElement

Convert.ToBoolean() has no knowledge of JsonElement and throws InvalidCastException.

Why Previous Tests Didn't Fail

The original code had a catch block that swallowed all exceptions:

try
{
    value = ... Convert.ToBoolean(request.DefaultValue) ...
}
catch (Exception ex)
{
    value = request.DefaultValue;
    reason = "ERROR";
}

Previous tests (test_dynamic_evaluation.py, test_exposures.py) were asserting on:

  • The evaluated flag value (returned by the endpoint)
  • The reason field in the response

When InvalidCastException was thrown, the catch block returned reason = "ERROR" and value = request.DefaultValue. If a test expected an error case or didn't care about the reason, it would pass.

Why Flag Eval Metrics Tests Failed

The metrics tests need OpenFeature hooks to fire. The exception was thrown before calling the OpenFeature API:

  1. Convert.ToBoolean(JsonElement) throws
  2. The catch block catches it and returns "ERROR"
  3. OpenFeature is never called → hooks never fire → no metrics emitted

The tests assert on metrics received by the OTLP intake, not on the HTTP response. No OpenFeature call = no hooks = no metrics = test failure.

The Fix

Check if the value is a JsonElement and extract the actual value:

private static bool GetDefaultValueAsBool(object defaultValue)
{
    if (defaultValue is JsonElement jsonElement)
    {
        return jsonElement.ValueKind switch
        {
            JsonValueKind.True => true,
            JsonValueKind.False => false,
            JsonValueKind.String => bool.Parse(jsonElement.GetString()),
            _ => false
        };
    }
    return Convert.ToBoolean(defaultValue);  // Fallback for non-JsonElement
}

"STRING" => await _client.GetStringValueAsync(request.Flag, request.DefaultValue?.ToString(), context),
"INTEGER" => await _client.GetIntegerValueAsync(request.Flag, Convert.ToInt32(request.DefaultValue), context),
"NUMERIC" => await _client.GetDoubleValueAsync(request.Flag, Convert.ToDouble(request.DefaultValue), context),
// "JSON" => (await _client.GetObjectValueAsync(request.Flag, Value.FromObject(request.DefaultValue), context)).AsStructure(),
"BOOLEAN" => await _client.GetBooleanValueAsync(request.Flag, GetDefaultValueAsBool(request.DefaultValue), context),
"STRING" => await _client.GetStringValueAsync(request.Flag, GetDefaultValueAsString(request.DefaultValue), context),
"INTEGER" => await _client.GetIntegerValueAsync(request.Flag, GetDefaultValueAsInt(request.DefaultValue), context),
"NUMERIC" => await _client.GetDoubleValueAsync(request.Flag, GetDefaultValueAsDouble(request.DefaultValue), context),
_ => request.DefaultValue
};
}
catch (Exception ex)
catch (Exception)
{
// _logger.LogError(ex, "Error on resolution");
value = request.DefaultValue;
reason = "ERROR";
}

return Ok(new { reason, value });
}

private static bool GetDefaultValueAsBool(object defaultValue)
{
if (defaultValue is JsonElement jsonElement)
{
return jsonElement.ValueKind switch
{
JsonValueKind.True => true,
JsonValueKind.False => false,
JsonValueKind.String => bool.Parse(jsonElement.GetString()),
_ => false
};
}
return Convert.ToBoolean(defaultValue);
}

private static string GetDefaultValueAsString(object defaultValue)
{
if (defaultValue is JsonElement jsonElement)
{
return jsonElement.ValueKind == JsonValueKind.String
? jsonElement.GetString()
: jsonElement.ToString();
}
return defaultValue?.ToString();
}

private static int GetDefaultValueAsInt(object defaultValue)
{
if (defaultValue is JsonElement jsonElement)
{
return jsonElement.ValueKind switch
{
JsonValueKind.Number => jsonElement.GetInt32(),
JsonValueKind.String => int.Parse(jsonElement.GetString()),
_ => 0
};
}
return Convert.ToInt32(defaultValue);
}

private static double GetDefaultValueAsDouble(object defaultValue)
{
if (defaultValue is JsonElement jsonElement)
{
return jsonElement.ValueKind switch
{
JsonValueKind.Number => jsonElement.GetDouble(),
JsonValueKind.String => double.Parse(jsonElement.GetString()),
_ => 0.0
};
}
return Convert.ToDouble(defaultValue);
}

private static EvaluationContext CreateContext(EvaluateRequest request)
{
var builder = EvaluationContext.Builder();
Expand All @@ -79,8 +132,16 @@ private static EvaluationContext CreateContext(EvaluateRequest request)
{
foreach (var attr in request.Attributes)
{
builder.Set(attr.Key, attr.Value as string);
// builder.Set(attr.Key, Value.FromObject(attr.Value));
// System.Text.Json deserializes to JsonElement, not string
var value = attr.Value switch
{
JsonElement jsonElement => jsonElement.ValueKind == JsonValueKind.String
? jsonElement.GetString()
: jsonElement.ToString(),
string s => s,
_ => attr.Value?.ToString()
};
builder.Set(attr.Key, value);
}
}
return builder.Build();
Expand Down
4 changes: 2 additions & 2 deletions utils/build/docker/dotnet/weblog/app.csproj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,8 +46,8 @@
<PackageReference Include="RabbitMQ.Client" Version="6.4.0"/>
<PackageReference Include="MongoDB.Driver" Version="2.23.1"/>
<PackageReference Include="Newtonsoft.Json" Version="13.0.3" />
<PackageReference Include="OpenFeature" Version="2.0.0" />
<PackageReference Include="Datadog.FeatureFlags.OpenFeature" Version="2.0.0" />
<PackageReference Include="OpenFeature" Version="2.3.0" />
<PackageReference Include="Datadog.FeatureFlags.OpenFeature" Version="*" />

<PackageReference Include="Datadog.Trace" Version="*" />
<PackageReference Include="OpenTelemetry.Api" Version="1.10.0" />
Expand Down
Loading