Menu

Asynchronous, Queue-based File Iterators

June 30, 2013 - C#, Programming


Embarassingly, I have come back to the FileIterator for other reasons only to discover, much to my horror, that it doesn’t work! It seems as if the work queue is empty far too fast or there are any number of other issues with the program that I’ve been unable to identify. It is unclear what the cause of the problem is- I suspect I will be “forced” to rewrite it entirely. If anybody sees what might be causing it please let me know so I can try to address it!

In a previous post, I looked at using the Win32 API through P/Invoke to directly call the Win32 FindFirstFile, FindNextFile, and FindClose() methods to perform a File search and retrieve those results using an Enumerator Method.

Here, we will look at what we can do to create a fully asynchronous search; in terms of allowing the results to process as the search continues, as well as perform that evaluation in a manner that best uses the available processing by processing multiple elements at the same time.

The first thing is that this won’t be an implementation of an interface; this has a large disadvantage in terms of Usability, but a wrapper could be created to provide IEnumerable<?> capabilities, so it won’t be something that should stop us.

The methodology I’ve thought of for this particular implementation is to use A background worker. When a search is started, the class will spin off that background worker, then proceed to perform it’s search, calling the appropriate filter delegate, and then if the resulting file is determined to “match” then it is added to a ConcurrentQueue. That ConcurrentQueue is examined intently by the BackgroundWorker, which waits until the Queue is Empty and the search is complete before it allows itself to exit. This is to protect from the eventuality where the actual Search logic that adds items to the Queue isn’t able to keep up with the BackgroundWorker Dequeue-ing elements.

We want to allow extensibility and custom filtering for both determining if the File counts as a “match” as well as whether a Directory should be recursed into. Appropriate overloads can be added that provide their own definitions of the more complex delegates, but we want to aim for a core level of functionality on which to build that simpler syntax.

The delegates

While we could feasibly use Predicate<> types as the parameters to the constructor of our search class for filtering, we will instead create specific delegates so we can more fully document the parameters and purpose of the delegates in XML documentation.

Since we will be dealing with the Win32 Find API, the appropriate declarations are a must. The required declarations are the struct, WIN32_FIND_DATA, the constants MAX_PATH and MAX_ALTERNATE, the ERROR_NO_MORE_FILES flag, as well as the FindFirstFile,FindNextFile, and FindClose API functions. We place these in the conventional NativeMethods class.

This also adds a few helpers. The actually search will be given two delegates, and we want to be able to pass in this information. So we will wrap the WIN32_FIND_DATA into a “FileSearchResult” class:

The purpose of this class is simple: First, we want to avoid instantiating a FileSystemInfo object if possible. The delegate method may not need that extra information, so we won’t create it ourselves. Instead, we provide easier wrappers around the WIN32_FIND_DATA, which includes wrapping the FILETIME structures and exposing them as the ‘everyday’ DateTime class type, which is far more familiar.

And there it is. It even has the IEnumerable implementation, as a static method. here’s a short program that uses it:

As it is there are some improvements I can think of. One would involve using Multicast Delegates rather than the basic Delegates, which is a fancy way of saying “use events fool!”. But at the same time that would probably fit more appropriately as a subclass which provides it’s own delegate implementations that instead call into events it declares, or something to that effect.

The full source to this Project can be found Here in a small github repo I created for it. Who knows, maybe it will mutate into a File Search API to Rival “BCFile”, but for .NET.

Have something to say about this post? Comment!