The framework I developed here should be well-suited for this task. All you have to do is to write a converter from your papers into the list structure. But, it is very easy to do.
To load the framework, you could either grab all code pieces together from that post and evaluate, or (much simpler), call:
Import["https://gist.githubusercontent.com/lshifr/2696189/raw/largeData.m"]
Next, note that by default, a cross-platform but much slower option which uses Compress
is used to store the list chunks. You may want to execute this, after you load the main code, but before you start your work:
$fileNameFunction = mxFileName;
$importFunction = mxImport ;
$exportFunction = mxExport ;
$compressFunction = Identity;
$uncompressFunction = Identity;
to switch to using .mx
files, which are much faster (assuming that you won't frequently change machines / platforms). See the examples in the linked post for more details.
Assuming that you have that code loaded, you basically need something like (the papers
variable is the same as you defined in the question):
initList[allpapers];
Do[appendTo[allpapers,Import[paper]],{paper, papers}];
storeMainList[allpapers, DestinationDirectory :> "your-directory-to-store-chunks"]
and then you can work with the allpapers
variable as with a usual list, in many respects, just that it won't all be in-memory at once. See the mentioned link for examples of use, and the linked questions too.
To load the list in a different session, you have to load the framework's code and use
retrieveMainList[allpapers, DestinationDirectory :> "your-directory-to-store-chunks"]
If you later change the list (add more elements or delete some), you will have to call storeMainList
again, for these changes to persist on disk.