Routing API traffic using health endpoints
Load balancers are typically used to route traffic to different instances of a service. This is done to distribute the load across multiple instances, but is also to ensure that a healthy service is used. A health endpoint will return a JSON response and HTTP 200 for healthy or a HTTP 500 for unhealthy.
The load balancer is configured to call this endpoint periodically. The load balancer will aim to only route traffic to instances that are healthy. Typically load balancers will determine health using two consecutive calls to the health endpoint.
Example unhealthy status json response api/health:
HTTP 502
{
"status": "Unhealthy",
"version": "2023.1.3"
}
Automatically redirecting traffic during deployments
When deploying a new service we have a couple of options to ensure that traffic is not routed to the service when it’s unavailable:
- Manually update the load balancer configuration to remove the service instance (like a caveman); or
- Use a health endpoint to automatically remove the service from the load balancer by returning a non-200 status code.
The second option is the option covered here. As of dotnet 7 we can implement a IHostedLifecycleService service to signal that the service is stopping.
/// <summary>
/// A background service that is started and stopped prior to other services. This allows signalling that the process is starting and stopping.
/// </summary>
internal class LifecycleService(
IOptions<AppOptions> options,
IApplicationHealthProvider healthProvider,
ILogger<LifecycleService> logger)
: IHostedLifecycleService
{
public Task StartingAsync(CancellationToken cancellationToken) => Task.CompletedTask;
public Task StartAsync(CancellationToken cancellationToken) => Task.CompletedTask;
public Task StartedAsync(CancellationToken cancellationToken) => Task.CompletedTask;
public async Task StoppingAsync(CancellationToken cancellationToken)
{
logger.LogInformation("Application is stopping. Deferring shutdown for {Delay}", options.Value.ApplicationShutdownDelay);
healthProvider.ApplicationIsStopping = true;
// Delay 10s to allow load balancer to stop routing traffic
await Task.Delay(options.Value.ApplicationShutdownDelay, cancellationToken);
logger.LogInformation("Application is stopping");
}
public Task StopAsync(CancellationToken cancellationToken) => Task.CompletedTask;
public Task StoppedAsync(CancellationToken cancellationToken) => Task.CompletedTask;
}
The ApplicationIsStopping property is used to signal that the application is stopping. This property is used in the health endpoint to return a degraded status code.
In addition, the StoppingAsync method uses a delay to allow the load balancer to stop routing traffic to the service.
public class ApplicationHealthProvider : IApplicationHealthProvider
{
public HealthReport? HealthReport { get; set; }
public DateTime? LastExecution { get; set; }
public bool ApplicationShuttingDown { get; set; }
/// <summary>
/// Returns the global health based on background checks and whether the application is shutting down.
/// </summary>
public HealthStatus? GetApplicationHealth()
{
var status = HealthReport?.Status;
return status == HealthStatus.Healthy && ApplicationShuttingDown ? HealthStatus.Degraded : status;
}
}
The ApplicationShuttingDown property is used to signal that the application is shutting down. This property is used in the health endpoint to return a degraded status code.
[ApiController, UsedImplicitly]
public class HealthController(
IApplicationHealthProvider healthProvider) : ControllerBase
{
[AllowAnonymous]
[Route(""), HttpGet]
public ActionResult DefaultRoute() => new RedirectResult("/api/systemstatus");
/// <summary>
/// Returns the last global health check result and the application version number.
/// </summary>
[AllowAnonymous]
[Route("api/systemstatus"), HttpGet]
public ActionResult<SystemStatusResult> SystemStatus()
{
var applicationVersion = AppVersionHelper.GetAppVersion(System.Reflection.Assembly.GetExecutingAssembly());
var hostname = MachineNameProvider.GetMachineName();
var result = new SystemStatusResult(
applicationVersion,
Environment,
healthProvider.GetApplicationHealth(),
hostname
);
var statusCode = HttpStatusCodeFromHealth(result.Health);
return StatusCode(statusCode, result);
}
/// <summary>
/// Returns the last global health check result and details per health check.
/// </summary>
[AllowAnonymous]
[Route("api/health/details"), HttpGet]
public ActionResult<HealthDetails> DetailedHealth()
{
var applicationVersion = AppVersionHelper.GetAppVersion(System.Reflection.Assembly.GetExecutingAssembly());
var hostname = MachineNameProvider.GetMachineName();
var healthDetails = new HealthDetails(
healthProvider.HealthReport,
healthProvider.HealthReportCreated,
applicationVersion,
hostname);
var statusCode = HttpStatusCodeFromHealth(healthProvider.GetApplicationHealth());
return StatusCode(statusCode, healthDetails);
}
/// <summary>
/// Determine correct status code based on health status. This is required so the application endpoint can be excluded by the ALB
/// </summary>
private int HttpStatusCodeFromHealth(HealthStatus? healthStatus)
{
return healthStatus switch
{
HealthStatus.Healthy => StatusCodes.Status200OK,
HealthStatus.Degraded => StatusCodes.Status200OK,
HealthStatus.Unhealthy => StatusCodes.Status503ServiceUnavailable,
null => StatusCodes.Status503ServiceUnavailable,
_ => StatusCodes.Status500InternalServerError
};
}
}
Taking it further with client-side health checks
If you are using only gRPC on ASP.NET Core then you probably want to look at gRPC client-side load balancing. However, if you are supporting REST services, or gRPC service on .NET Framework and don’t use an Application Load Balancer then you can still implement client-side health checks including priority routing.
The EndpointHealthProvider class is used to store the health status of service endpoints. The GetBestEndpoints method returns the best endpoints based on health status and priority. The AddOrUpdateHealthStatus method is used to add or update the health status of a service endpoint. This is general purpose and can be used for multiple services provided it is installed transiently.
/// <summary>
/// A utility class to provide health status for service endpoints.
/// </summary>
public class EndpointHealthProvider : IEndpointHealthProvider
{
private readonly ConcurrentDictionary<Uri, EndpointGroup> _healthStatuses = new();
private sealed class EndpointGroup(bool isPreferred, HealthCheckResult healthCheckResult)
{
public bool IsPreferred { get; } = isPreferred;
public HealthCheckResult HealthCheckResult { get; } = healthCheckResult;
};
/// <summary>
/// Get the best endpoints by health status ordered from healthy to unhealthy.
/// Healthy = 2, Degraded = 1, Unhealthy = 0
/// </summary>
public ServiceEndpointSummary GetBestEndpoints()
{
if (_healthStatuses.IsEmpty) return new ServiceEndpointSummary(HealthStatus.Unhealthy, new List<ServiceEndpoint>());
var bestEndpoints = _healthStatuses
.GroupBy(s => s.Value.HealthCheckResult.Status) // Group by health status
.OrderByDescending(g => g.Key) // Order by health status 2, 1, 0
.First()
.GroupBy(item => item.Value.IsPreferred) // Group by IsPreferred
.OrderByDescending(g => g.Key) // Order by IsPreferred True, False
.First()
.Select(item => new ServiceEndpoint(item.Key, item.Value.HealthCheckResult, item.Value.IsPreferred))
.ToList();
var summaryStatus = bestEndpoints[0].HealthCheckResult.Status;
return new ServiceEndpointSummary(summaryStatus, bestEndpoints);
}
/// <summary>
/// Add or update a health status check
/// </summary>
public void AddOrUpdateHealthStatus(Uri serviceUri, HealthCheckResult healthCheckResult, bool isPreferred)
{
var healthGroup = new EndpointGroup(isPreferred, healthCheckResult);
_healthStatuses.AddOrUpdate(serviceUri, healthGroup, (_, _) => healthGroup);
}
}
The EndpointHealthService class is used to select a random endpoint from the best endpoints of a specific service. The GetBestRandomEndpointAsync method returns a random endpoint from the best endpoints. The RefreshHealthAsync method is used to force a refresh of the health statuses.
public class EndpointHealthService(
ILogger<EndpointHealthService> logger,
IEndpointHealthProvider endpointHealthProvider,
IHealthServiceProvider healthServiceProvider,
IOptions<EndpointSettings> settings) : IEndpointHealthService
{
private static readonly ThreadLocal<Random> Random = new(() => new Random());
public IEndpointHealthProvider EndpointHealthProvider => endpointHealthProvider;
/// <summary>
/// Select a random endpoint from the best endpoints. This will trigger health checks if no health statuses are available.
/// </summary>
public async Task<ServiceEndpoint> GetBestRandomEndpointAsync(CancellationToken ct = default)
{
if (settings.Value.BaseUrls == null) throw new InvalidOperationException(nameof(settings.Value.BaseUrls));
var endpointSummary = endpointHealthProvider.GetBestEndpoints();
if (endpointSummary.ServiceEndpoints.Count == 0)
{
logger.LogInformation("No endpoint health statuses available. Refreshing health on {Count} endpoints", settings.Value.BaseUrls.Length);
endpointSummary = await RefreshHealthAsync(ct);
}
// Select a random one from that set
var selectedEndpoint = endpointSummary.ServiceEndpoints.Count == 1
? endpointSummary.ServiceEndpoints[0]
: endpointSummary.ServiceEndpoints[Random.Value.Next(endpointSummary.ServiceEndpoints.Count)];
var logLevel = selectedEndpoint.HealthCheckResult.Status == HealthStatus.Healthy ? LogLevel.Debug : LogLevel.Warning;
logger.Log(logLevel, "Selected endpoint {Url} with status {Status}", selectedEndpoint.ServiceUri, selectedEndpoint.HealthCheckResult.Status);
return selectedEndpoint;
}
/// <summary>
/// Force a refresh of the health statuses
/// </summary>
public async Task<ServiceEndpointSummary> RefreshHealthAsync(CancellationToken ct = default)
{
if (settings.Value.BaseUrls == null) throw new InvalidOperationException(nameof(settings.Value.BaseUrls));
foreach (var baseUrl in settings.Value.BaseUrls)
{
var healthStatus = await GetHealthStatusAsync(baseUrl, ct);
var isPreferred = settings.Value.PreferredBaseUrls != null && settings.Value.PreferredBaseUrls.Contains(baseUrl);
endpointHealthProvider.AddOrUpdateHealthStatus(baseUrl, healthStatus, isPreferred);
}
return endpointHealthProvider.GetBestEndpoints();
}
private async Task<HealthCheckResult> GetHealthStatusAsync(Uri baseUrl, CancellationToken ct = default)
{
try
{
logger.LogTrace("Issuing endpoint health check to {Url}", baseUrl);
var healthService = healthServiceProvider.GetHealthService(baseUrl);
var context = CallContextBuilder.Create(cancellationToken: ct, timeout: settings.Value.HealthCheckTimeout);
var statusResult = await healthService.GetSystemStatusAsync(context);
return CreateHealthCheckResult(baseUrl, statusResult);
}
catch (TaskCanceledException)
{
// We are shutting down
throw;
}
catch (Exception ex)
{
logger.Log(settings.Value.HealthCheckFailureLogLevel, "Error performing endpoint health check to {Url}. Message: {Message}", baseUrl, ex.Message);
return new HealthCheckResult(HealthStatus.Unhealthy, $"Error getting endpoint status from {baseUrl}: {ex.Message}");
}
}
private static HealthCheckResult CreateHealthCheckResult(Uri healthStatusUri, SystemStatusResult statusResult)
{
return new HealthCheckResult(statusResult.Health ?? HealthStatus.Degraded, $"{healthStatusUri} [{statusResult.Version}]");
}
}
The EndpointHealthJob class is used to periodically refresh the health statuses of the service endpoints.
public class EndpointHealthJob(
IEndpointHealthService endpointHealthService,
IOptions<EndpointSettings> settings,
ILogger<EndpointHealthJob> logger)
: BackgroundService
{
private readonly PeriodicTimer _periodicTimer = new(settings.Value.HealthCheckInterval);
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
// Yield to allow the application to start
await Task.Yield();
do
{
try
{
await endpointHealthService.RefreshHealthAsync(stoppingToken);
}
catch (TaskCanceledException)
{
// We are shutting down
return;
}
catch (Exception ex)
{
logger.LogError(ex, "Error refreshing endpoint health");
}
await _periodicTimer.WaitForNextTickAsync(stoppingToken);
} while (!stoppingToken.IsCancellationRequested);
}
}
Use this technique, a job will periodically check the health of all the provided endpoints and store the results. Then GetBestRandomEndpointAsync is used to randomly select the best one based on health status and priority.
This is useful when you have multiple instances of a service and you want to ensure that traffic is routed to the healthiest instances, but also would prefer say local endpoints over remote endpoints e.g. endpoints in a local vs remote data centre.