Searching Large Text Files in C#
By techmike
Searching Large Files in c#
I needed to search a large row text file very quickly. The results of my initial code took upwards of 10 minutes to search. Slowly changing various things I was able to reduce my search to less than a minute.
Problem: My 800 meg text file was not able to be loaded into memory to do a search
Result: Out Of Memory Exception
Solution 1: I read the file line by line taking several minutes due to severe disk access, toooo SLOW
Solution 2: I split the file into multiple 100 meg files that were loaded into memory 1 at a time
Problem: Reading the files were much quicker, Added an indexOf to do my slowed my search dramatically
Solution: Changed my code from indexOf to Regular Expressions, wow the speed was back
Includes
using System.Text.RegularExpressions;
Did you find this helpful?
See results without votingSample Code
List searchResults = new List();
Regex regexp = new Regex(txtBarcode.Text);
string[] info7 = File.ReadAllLines(drive + @"index\index7.rpt");
foreach (String line in info7)
{
if (regexp.Match(line).Success)
searchResults.Add(line);
}Comments
Well, actually neither. I had to create a image search program for a customer. The text file had the search information in it pointing to an image location. The customer will be able to search for an image out of 8 million that will bring that image up from two 2tb usb drives. The search was no problem in sql server, but it needed to run without having to get additional IT support for setup. I have a pc with 4gig of memory and loading the entire index file would give me an out of memory exception, and I didn't want to chance the fact it would happen to the customer. Of course I could find out after he gets it, that the files are too big for his machine, but hopefully not.
techmike
Thanks for the additional information.
It would seem unlikely that the customer knowing the size of his data wouldn't have an appropriate size machine. Memory and Hard Disks are fairly cheap.
Just remember its not always easy for everyone to get support from their IT department. I thought I had a pretty hefty machine too :).. Thanks for the comments!
techmike
I certainly agree on the IT department comment, it really changes from company to company and sometimes the IT policies escape logic.
ib radmasters 4 months ago
Was it C# or your system that failed?