One of the issues is that these are very large files (30m+) and therefore we don't want the whole file to be in memory at any 1 time.