These background tasks will run for the entirety of the applications life, and will be reading azure queues so spending a lot of time sitting doing nothing.

I agree with others' comments: by far the best solution is to move these out of the ASP.NET process completely. Azure Functions, for example, has built-in support for Azure Storage Queues. Benefits of Azure Functions over ASP.NET in-process include:

• Your API (ASP.NET) can scale independently from your backend (Azure Function).
• Scaling on the backend is automatic. Up as high as you want, potentially down to zero.
• The backend process has its own thread pool, so there's no interference with the ASP.NET thread pool.

I have read that for I/O you shouldn't use Task.Run() as it's inefficient

Task.Run should be avoided in ASP.NET's request processing pipeline. It's always inefficient regardless of the type of work being done.

Task.Run can be useful for background tasks in an ASP.NET process, in particular if there's blocking work done during startup of the background task.

The main thing with Task.Run on ASP.NET is that you don't want to interfere with the ASP.NET thread pool heuristics. Regularly queueing work to the thread pool is a problem, since you're stealing threads and then re-injecting them on a regular basis. So using Task.Run when processing each HTTP request or when processing each queue item would be a bad idea. A single Task.Run done once at startup doesn't matter.

It seems if I don't use Task.Run() then every time my background task gets something from the queue and so the code after await continues, then it will be blocking some main thread in ASP that spooled up the task in the first place.

This is a valid concern if there's blocking work being done during your message processing.

One way to think about async/await is that each method is broken up into pieces (at each await point), and each of those pieces is scheduled separately, one at a time. Each time a piece is scheduled, it takes a thread pool thread, runs the piece, and then returns the thread. As long as the work is fast, this is fine, but if one of those pieces has blocking work, then the ASP.NET thread pool heuristics can be impacted.

You still can use IHostedService as base for background tasks in combination with BlockingCollection.

Create a wrapper for BlockingCollection so we can inject it as singleton.

public class TasksToRun
{
    private readonly BlockingCollection<SingleTaskData> _tasks;

    public TasksToRun() => _tasks = new BlockingCollection<SingleTaskData>(new ConcurrentQueue<SingleTaskData>());

    public void Enqueue(SingleTaskData taskData) => _tasks.Add(settings);

    public TaskSettings Dequeue(CancellationToken token) => _tasks.Take(token);
}

For background process we can use "built-in" implementation of IHostedService - Microsoft.Extensions.Hosting.BackgroundService.

public class TaskProcessor : BackgroundService
{
    private readonly TasksToRun _tasks;

    public TaskProcessor(TasksToRun tasks) => _tasks = tasks;

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        await Task.Yield(); // This will prevent background service from blocking start up of application

        while (cancellationToken.IsCancellationRequested == false)
        {
            try
            {
                var taskToRun = _tasks.Dequeue(_tokenSource.Token);


                await ExecuteTask(taskToRun);               
            }
            catch (OperationCanceledException)
            {
                // execution cancelled
            }
            catch (Exception e)
            {
                // Catch and log all exceptions,
                // So we can continue processing other tasks
            }
        }
    }
}

Then we can add new tasks from the controller without waiting for them to complete

public class JobController : Controller
{
    private readonly TasksToRun _tasks;

    public JobController(TasksToRun tasks) => _tasks = tasks;

    public IActionResult PostJob()
    {
        var taskData = CreateSingleTaskData();

        _tasks.Enqueue(taskData);

        return Ok();
    }
}

Wrapper for blocking collection should be registered for dependency injection as singleton

services.AddSingleton<TasksToRun, TasksToRun>();

Register background service

services.AddHostedService<TaskProcessor>();

You can use Hangfire It's a fantastic framework for background jobs in ASP.NET.You can find HangFie Tutorial Here.

The best feature from Hangfire is its built in /hangfire dashboard that shows you all your scheduled, processing, succeeded and failed jobs. It's really a nice polished addition.

All these libraries are excellent, are open source, and Available as Nuget Packages.

Some Other Options

QUARTZ.NET

FLUENTSCHEDULER

WEBBACKGROUNDER

In that case you can only rely on the servicelocater-pattern which is an anti-pattern. Also the DbContext must be instance-per-dependency (transient) rather than scoped.

My suggestion is to inject IServiceScopeFactory which is a singleton and the beginn a scope in your background-worker that does the deed.

using (var scope = _serviceScopeFactory.CreateScope())
{
   var context = scope.ServiceProvider.GetRequiredService<DbContext>();
   // now do your work
}

There is no other. These things are not part of the mvc pipeline so they cannot be resolved on the fly. I would also suggest not to directly access the dbcontext somewhere. Wrap things in a repository and then use that repository or even wrap that repository in a so called service class. So your logic is encapsulated and your background-worker just executes things when receiving a message.

Not directly related, but ASP.NET-Core has background-worker built-in with IHostedService.

Edit 2018-07-30: As Steven suggested below, it might not always be an anti-pattern when manually resolving / initiating classes. At least not when the compositionroot is taking care of it. Here is a link he provided, where it is explained quite good.

Edit 2022-07-07:

Using the latest c# syntax and creating an async-scope is now also possible.

await using var scope = _serviceProviderFactory.CreateAsyncScope();
var context = scope.ServiceProvider.GetRequiredService<DbContext>();
// now do your work

Alternatively you can also directly inject IServiceProvider and do the very same thing.

IMHO, your ASP.NET Web API application shouldn't run those background tasks by itself. It should only be responsible to receive the request, stick it inside the queue (as you indicated) and return the response indicating the success or failure of the received message. You can use variety of messaging systems such as RabbitMQ for this approach.

As for the notification, you have a few options. You can have an endpoint which the client can check whether the processing is completed or not. Alternatively, you could provide a streaming API endpoint which your client can subscribe to. This way, the client doesn't have to poll the server; the server can notify the clients which are connected. ASP.NET Web API has a great way of doing this. The following blog post explains how:

• Native HTML5 push notifications with ASP.NET Web API and Knockout.js

You can also consider SignalR for this type of server to client notifications.

The reason why background tasks are hard for ASP.NET Web API application is that you're responsible to keep the AppDomain alive. This is a hassle especially when you are hosting under IIS. The following blog posts explains really good what I mean:

• Returning Early from ASP.NET Requests
• The Dangers of Implementing Recurring Background Tasks In ASP.NET

Nothing good or bad in general, it is all based on how you will use it, and if you will consider any expected future issues, I find that this article and links inside it is very good http://www.hanselman.com/blog/HowToRunBackgroundTasksInASPNET.aspx

A few years back Phil Haack wrote a great article on the dangers of recurring background tasks in ASP.NET. In it he points out a few gotchas that are SO common when folks try to do work in the background. Read it, but here's a summary from his post.

An unhandled exception in a thread not associated with a request will take down the process. If you run your site in a Web Farm, you could end up with multiple instances of your app that all attempt to run the same task at the same time. The AppDomain your site runs in can go down for a number of reasons and take down your background task with it. If you think you can just write a background task yourself, it's likely you'll get it wrong. I'm not impugning your skills, I'm just saying it's subtle. Plus, why should you have to?

There's LOT of great ways for you to do things in the background and a lot of libraries and choices available.

Some ASP.NET apps will be hosted in IIS in your data center and others will be hosted in the Azure cloud. The spectrum of usage is roughly this, in my opinion:

General: Hangfire (or similar similar open source libraries) used for writing background tasks in your ASP.NET website Cloud: Azure WebJobs A formal Azure feature used for offloading running of background tasks outside of your Website and scale the workload Advanced: Azure Worker Role in a Cloud Service scale the background processing workload independently of your Website and you need control over the machine There's lots of great articles and videos on how to use Azure WebJobs, and lots of documentation on how Worker Roles in scalable Azure Cloud Services work, but not a lot about how your hosted ASP.NET application and easily have a background service. Here's a few.

You can use an IHostedService with a number of threads to consume the IBackgroundTaskQueue.

Here is a basic implementation. I assume you're using the same IBackgroundTaskQueue and BackgroundTaskQueue described here.

public class QueuedHostedService : IHostedService
{
    private readonly ILogger _logger;

    private readonly Task[] _executors;
    private readonly int _executorsCount = 2; //--default value: 2
    private CancellationTokenSource _tokenSource;
    public IBackgroundTaskQueue TaskQueue { get; }

    public QueuedHostedService(IBackgroundTaskQueue taskQueue,
        ILoggerFactory loggerFactory,
        IConfiguration configuration)
    {
        TaskQueue = taskQueue;
        _logger = loggerFactory.CreateLogger<QueuedHostedService>();

        if (ushort.TryParse(configuration["App:MaxNumOfParallelOperations"], out var ct))
        {
            _executorsCount = ct;
        }
        _executors = new Task[_executorsCount];
    }

    public Task StartAsync(CancellationToken cancellationToken)
    {
        _logger.LogInformation("Queued Hosted Service is starting.");

        _tokenSource = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken);

        for (var i = 0; i < _executorsCount; i++)
        {
            var executorTask = new Task(
                async () =>
                {
                    while (!cancellationToken.IsCancellationRequested)
                    {
#if DEBUG
                    _logger.LogInformation("Waiting background task...");
#endif
                    var workItem = await TaskQueue.DequeueAsync(cancellationToken);

                        try
                        {
#if DEBUG
                        _logger.LogInformation("Got background task, executing...");
#endif
                        await workItem(cancellationToken);
                        }
                        catch (Exception ex)
                        {
                            _logger.LogError(ex,
                                "Error occurred executing {WorkItem}.", nameof(workItem)
                            );
                        }
                    }
                }, _tokenSource.Token);

            _executors[i] = executorTask;
            executorTask.Start();
        }

        return Task.CompletedTask;
    }

    public Task StopAsync(CancellationToken cancellationToken)
    {
        _logger.LogInformation("Queued Hosted Service is stopping.");
        _tokenSource.Cancel(); // send the cancellation signal

        if (_executors != null)
        {
            // wait for _executors completion
            Task.WaitAll(_executors, cancellationToken);
        }

        return Task.CompletedTask;
    }
}

You need to register the services in ConfigureServices on Startup class.

...
services.AddSingleton<IBackgroundTaskQueue, BackgroundTaskQueue>();
services.AddHostedService<QueuedHostedService>();
...

Aditionally, you can set the number of threads in configuration (appsettings.json)

...
"App": {
    "MaxNumOfParallelOperations": 4
}
...