Resilience & Error Handling
Recover from transient failures automatically, and fail fast on the ones that won’t fix themselves.
Overview
Resilient task handling comes down to a few things: retrying transient errors, failing fast on permanent ones, bounding runtime with timeouts, and recovering cleanly across restarts. EverTask covers each.
Key Features:
- Retry Policies: Automatically retry failed tasks with configurable policies
- Exception Filtering: Fail-fast on permanent errors, retry transient failures
- OnRetry Callbacks: Track and debug retry attempts
- Timeout Management: Prevent runaway tasks with flexible timeouts
- Cancellation Support: Implement cooperative cancellation with CancellationTokens
- Graceful Shutdown: Handle application restarts with automatic task recovery
- Error Observation: React to errors with lifecycle hooks
- Misfire Policies: Decide what a recurring schedule does with the slots a downtime missed
What a downtime does to a recurring schedule
Recovery brings back everything that was pending, but a recurring schedule is a special case: the slots that came due while nobody was running are gone by the time the process is back. By default they are skipped: at most the slot that is still the current one runs, and the schedule moves on. That is the right answer for a heartbeat and the wrong one for a nightly billing job.
OnMisfire is where you say which it is: FireOnce collapses the whole missed run into one occurrence, and CatchUp replays every missed slot inside caps you set. Both give each occurrence a durable row of its own, with its own retries and its own audit trail, and both come with an at-least-once contract: write handlers that can run twice. See Durable Occurrences.
Topics
Overview
Introduction to the resilience features, with quick examples.
Retry Policies
Configure automatic retry behavior for failed tasks. Learn about LinearRetryPolicy, ExponentialRetryPolicy (exponential backoff), custom policies, and Polly integration.
Exception Filtering
Control which exceptions trigger retries and which fail immediately. Use whitelist, blacklist, or predicate-based filtering to save resources and improve error visibility.
Retry Callbacks
Track retry attempts with OnRetry callbacks. Implement metrics tracking, circuit breaker patterns, and debugging for intermittent failures.
Timeout Management
Prevent runaway tasks with global, per-handler, and per-queue timeout configuration.
Cancellation Tokens
Implement cooperative cancellation for graceful shutdown and resource cleanup using CancellationTokens.
Graceful Shutdown
Handle application restarts gracefully with automatic task recovery and progress tracking.
Error Observation
React to errors using OnError lifecycle hooks for alerting, telemetry, and compensation workflows.
Best Practices
Follow best practices for retry policies, exception filtering, timeouts, cancellation, and error handling.
Quick Start
Basic Retry Policy
builder.Services.AddEverTask(opt =>
{
// 3 retries (up to 4 executions) with 500ms delay between attempts
opt.SetDefaultRetryPolicy(new LinearRetryPolicy(3, TimeSpan.FromMilliseconds(500)));
// Or exponential backoff: 500ms, 1s, 2s, 4s, 8s
// opt.SetDefaultRetryPolicy(new ExponentialRetryPolicy(5, TimeSpan.FromMilliseconds(500)));
});
Exception Filtering
public class DatabaseTaskHandler : EverTaskHandler<DatabaseTask>
{
public override IRetryPolicy? RetryPolicy => new LinearRetryPolicy(5, TimeSpan.FromSeconds(2))
.HandleTransientDatabaseErrors(); // Only retry database-related errors
}
Timeout Configuration
public class QuickTaskHandler : EverTaskHandler<QuickTask>
{
public override TimeSpan? Timeout => TimeSpan.FromSeconds(30);
}
Cancellation Token Usage
public override async Task Handle(MyTask task, CancellationToken cancellationToken)
{
foreach (var item in task.Items)
{
cancellationToken.ThrowIfCancellationRequested();
await ProcessItemAsync(item, cancellationToken);
}
}
Common Patterns
Database Handler with Retries
public class RobustDatabaseHandler : EverTaskHandler<DatabaseTask>
{
private readonly ILogger<RobustDatabaseHandler> _logger;
private readonly IMetrics _metrics;
// 1s, 2s, 4s, 8s: exponential backoff, only for database errors
public override IRetryPolicy? RetryPolicy =>
new ExponentialRetryPolicy(4, TimeSpan.FromSeconds(1))
.HandleTransientDatabaseErrors();
public override ValueTask OnRetry(Guid taskId, int attemptNumber, Exception exception, TimeSpan delay)
{
_logger.LogWarning(exception,
"Database task {TaskId} retry {Attempt} after {DelayMs}ms",
taskId, attemptNumber, delay.TotalMilliseconds);
_metrics.IncrementCounter("db_task_retries");
return ValueTask.CompletedTask;
}
public override async Task Handle(DatabaseTask task, CancellationToken ct)
{
await _dbContext.ProcessAsync(task.Data, ct);
}
}
HTTP API with Circuit Breaker
public class HttpApiHandler : EverTaskHandler<HttpApiTask>
{
public override IRetryPolicy? RetryPolicy => new LinearRetryPolicy(3, TimeSpan.FromSeconds(1))
.HandleWhen(ex => ex is HttpRequestException httpEx && httpEx.StatusCode >= 500);
public override async Task Handle(HttpApiTask task, CancellationToken ct)
{
var response = await _httpClient.GetAsync(task.Url, ct);
response.EnsureSuccessStatusCode();
}
}
Next Steps
Start with the Overview to learn about resilience features, or jump directly to:
- Retry Policies - Configure automatic retry behavior
- Exception Filtering - Fail-fast on permanent errors
- Best Practices - Patterns and pitfalls
Note: Start with the defaults and tune to your workload.