Upcoming article - speeding up processing of large amounts of data

One of the things I come across quite often is the need to process large amounts of data, apply certain business rules to it, and then write it back to the database it came from.

If the business rules to be applied are not too complex, you can either generate an SQL UPDATE statement that will do the job or use a stored procedure with that same UPDATE statement. That's your best bet.

If, however, the logic is complex and better represented in code, then you're probably stuck creating a data reader, looping through the records, and changing them one at a time. You could simply do this as described and let it take as long as it takes, or you could take advantage of threading to split the job into mutilple concurrent jobs.

Managing these worker threads through a semaphore, creating and maintaining a work queue, dispatching the work units, handling exceptions ... all this will be in the next article.