Most efficient method for export/import of non-traditional tensor-style dimensions of matrices/tables/lists-of-lists-of-lists(-of lists)?

Question

Hello World!

(I did it for my first answer, so it only feels proper...I'm not nostalgic, you are!)

OKAY, here we go!

Let's start off with the good-stuff, here's my code, made relatively arbitrary, as I cannot (yet) share the actual code, though I will be happy to test methods with my real code and provide, in the end, a nice display of the collated methods and their efficacies, with attributions to those who contribute:

YourMainFunction[{v1_?NumericQ,v2_?NumericQ,v3_?NumericQ},d1_?NumericQ,n_?IntegerQ]:=
(Export[NoteBookDirectory[]<>ToString[N[v1,8]]<>"v1"<>ToString[N[v2,5]]<>"v2"<>ToString[N[v3,5]]<>"v3.wdx",
Transpose[SortBy[Transpose[Eigensystem[N[f[v1,v2,v3,d1,n]]],Re[#[[1]]]&][[n+1;;2 n]]]);

On the surface, I am running an Eigensystem calculation for a function of 3 arbitrary inputs of numerical values that is size n-by-n, and I output the following format:

{{v1, v2, v3},{eVals,eVecs}}

It must be mentioned that while the eVals are real, the eVecs are complex, and must be kept this way, if remotely possible.

However, I do realize I can input the parameters into the filename, and only contain the Eigensystem outputs in my exported files, like so:

expr=Transpose[SortBy[Transpose[Eigensystem[N[f[v1,v2,v3]]],Re[#[[1]]]&][[n+1;;2 n]]];
Export[NotebookDirectory[]<>ToString[N[v1,8]]<>"v1"<>ToString[N[v2,5]]<>"v2"<>ToString[N[v3,5]]<>"v3.wdx",expr];

Which resulted in the above coding attempts.

I will do this 10,000 times (100-by-100 combinations of v2 & v3) before I change the v1 value, or vice-versa, as there will be 1 main parameter, whilst the others are cycled through all combinations, I do this in a variety of ways, often either ParallelSubmit(most efficient) or ParallelTable(best for reimportation of ParallelSubmit-created data), but currently on a dual-core machine, I am unable to load the sheer volume of files in any feasible manner that can be completed in the length of a day let alone over a research meeting, and in-fact put too much stress on my hexa-core mobile workstation (though it does work! [albeit slowly]), and I only find comfort in my CPU load percentage when using a 16-core behemoth I keep caged at home.

Each of these are stored separately, as this is best for stopping and starting mid-export. I then export the master list of the filenames of the exported files (this filename is what Export will output), but this can easily be recreated and properly collated if need be. It does not seem advantageous to reassemble the intended set format each time when importing, nor does it aid in the subsequent analysis of these datasets to be required to do so. I would also not like to lose accuracy due to truncated values in the filename, nor have lengthy strings for the filename. While I could investigate Parallel Computing methods using this post, I feel this has been addressed quite well already, and I recognize my error is likely in the method of how I import and export, and what file formats I use, as the used format .wdx is slow, but cross-system compatible.

I know the following:

• .wdx is appropriate for cross-compatibility between systems
• .mx is faster, but only able to be loaded on those systems with matching $SystemWordLength
• The most efficient method of exporting is using DumpSave and Get, but it doesn't work for pushing out a non-square table, nor is it entirely applicable for compatibility across systems or versions of the WL & Mathematica.

My question, made gigantic:

What is the most (more than above) efficient (fast, low memory-impact[we can clear it every time, but I am not even aware of what function to investigate for this], low-core-count-requirement) method, given my above code & following intended use, for the export and import of tensor-style (non-traditional dimensions) matrices, or lists-of-lists-of-lists(-of-lists)?

In other words, what is the right format to use in this case, in order to provide for quick export and import of many (~10,000+) files containing matrices of non-traditional tensor-style dimensions, id est, lists-of-lists-of-lists(-of-lists)?

Please note, prior to answering, that this must be a method that is cross-compatible, and appropriate for arbitrary systems, what I mean by this is something like a dual core should be able to load it (assuming enough available memory, of course[if you can solve this too, you are welcome to include it in your answer!]) in a feasible amount of time, thus allowing a 4+ core setup to load it faster, albeit with similarly impacted memory usage.

Here are some other, unmentioned, resources which I found useful, but I am left with a lot of older methods, and nothing that is "current" in such a way that I am unsure if .wdx remains a current format not yet replaced by .wxf or .wdf, what came first the chicken or the egg, or even if .mx is indeed cross-compatible [barring the aforementioned $SystemWordLength] (perhaps we can send around the same dataset and see if it stays consistent(I wonder how the CheckSum changes...don't answer that...but that would be fun wouldn't it?...)):

Relevant References & Resources:

• How do I save a variable or function definition to a file?
• Save expression and load them into another notebook efficiently?
• Fast way to export large amount of data in "Table" format
• How to save all data in all variables so that loading it is fast?

If this seems like an unresearched post, or may be incomplete, I've likely missed something glaring, so please let me know where I can provide clarifications or improvements, as I wish to have a resource for all through this question and subsequent answers, which will alleviate future concerns as to the most ideal method of export and import of lists-of-lists-of-lists(-of-lists) of non-traditional tensor-style dimensions. Also it is very late, and I need to sleep, but I will edit this message of sleep away in the morning tomorrow, in hopes of providing further clarification, or to begin collating time & memory tests.

I've only just stumbled upon the above methods of compression and binary exportation, but the current ability to understand this escapes me....I fear my solution is there, and I may just be posting this only to be marked as a crazy-overdeveloped-duplicate question, but it would be awesome if this method was revitalized here, using the newest methods that have been determined by the community, hint hint...maybe?

Also this is super duper relevant, though it is not cross-compatible...but I need sleep, for real!

Thank you in advance to all who take the time to read through this with a serious eye, I hope that my lack of brevity is not too lax in the way I express my sheer passion and enjoyment of Mathematica & (the) Wolfram Language within this question post. Everyone here is awesome, and I am very excited to see what inputs you all have for this. Enjoy!

Answer 1

You might want to give the HDF5 format a try. It seems to be very efficient, even faster than MX in the example below.

One has only to be careful to use the option "ComplexKeys" -> {"Re", "Im"} upon import (otherwise, each complex number is split into an association containing real and imaginary part, rendering the method very inefficient).

Export:

n = 1000;
A = RandomReal[{-1, 1}, {n, n}];

{λ, U} = Eigensystem[A];
Export["a.h5", {
  "Eigenvalues" -> λ,
  "Eigenvectors" -> U
  }, "Datasets"] // AbsoluteTiming

{0.019481, "a.h5"}

Import:

μ = Import["a.h5", {"Data", "/Eigenvalues"}, 
     "ComplexKeys" -> {"Re", "Im"}]; // AbsoluteTiming // First
V = Import["a.h5", {"Data", "/Eigenvectors"}, 
     "ComplexKeys" -> {"Re", "Im"}]; // AbsoluteTiming // First

0.011885

0.01822

Check:

Max[Abs[λ - μ]]
Max[Abs[U - V]]

0.

0.