Previously I asked a related question about my test data.
Looking at the size of the data
ans // ByteCount
9602605504
9602605504*8/519284064 // N
147.936
I will probably need about 150G at the moment to export to XLSX.
My data has unequal length
Dimensions /@ ans[[1 ;; 20]]
{{80, 6}, {78, 6}, {79, 6}, {79, 6}, {82, 6}, {81, 6}, {78, 6}, {82, 6}, {80, 6}, {77, 6}, {79, 6}, {80, 6}, {79, 6}, {82, 6}, {80, 6}, {81, 6}, {81, 6}, {81, 6}, {80, 6}, {81, 6}}
In theory, none of them should be more than {90,6}.
Some examples of the data entries,
ans=1:eJy1XcuyI7cNleNxxv6M/EGT1KOVnbJUJym3nWVWWbjqrrJw/r+\ iuaOZOUdNHICQvHFpyrpiNwnicXAA/OU///3ltx92u93vH27/+fvb7//77Wf615/pX9/\ jv379ePvw77/W47Fdv32e9vD59Pbd7fNvf+r/WT39+uPXr5bp7dP/+PXTH/\ z8tvv0Z7y29SMr/uD7b1hfLRU+n95+yiw3nfmZfxLrTQ0f8+37L+v9rbeetbW1/\ 7nMb59Wtf6swCmUCzxSze1ygVdpe7nLbf62Q+W2Qz/K9Yznn679bWzL+\ zaauwVfnZbNIw8/Rr1+e5dWJvgf+7cPSrIP8ByXtz9n1sYjbGf47fb+e+\ bOwVcLP8YPUgAtIT7B710/n7y1W3hQJ3wk565Zp7mQQNC92ykpuBifz+/qaPzOH+\ nefVBLL/2ly/l9veHrj0Jc8fP184/wE+O9g7WPOQHEy1ROtI0/qS1Y+/\ egwj0Y2YJa6fPXd6w3KVB3EI+ieltgqT187SqVfTui8OeUfZn7Gvd22t+pewd/\ Rnfw8m4jxk0qqJB20JuMdmbv6PrIcqfPyi3w1duVypmWY/98p8P770Vudrn2RWtErzY86+\ vnpa23RmNYv53pkGgthiWbZvnaqIAeLOqHzGujEvuiSKwdOtBrfxx4bVQSfde0NGnKyN9B\ PX76ZkIGrejG1TW/i9KVNJ0Vn/\ mAe6o98gvdh5zfAr9RWYlJy4mq9sguZE6ZHozPJ3ny9ByoWO8S+Gnn/\ nUaeQ702lGJHORzoJqq+ExeIGEdLbzLzSGTW4ACA9a8rDkXsi1stXdCAskELkmDglK31/\ ccnZVl62IOvyue9nSUar0cSSXnzAi6+\ 1P7HBNaX8WjnJNbu9CNkMvBSbZrzlajU0WWo2pbXfsHcrPVqUOdUC3OOuZH5X3I3Rf0mm9\ 2SskQnunNwU7ZJvI06/\ sjCy8CTDJ5mikdjTLCelIqCRKtOefaT6yKfxDL1SFEw1oPcbEqQ3i0wvj59qpafK1dRnnC\ 317lY+BXb2aAVHhK0Hp4XyT4pKD6mjN/hAxpBMXCzm5Cqa+\ z4YMWBo7Iw5KqBJGX81DwbPi/K/64hnL2dPaoeFNmilBafbkpHFyTyFEztvEQV2/\ 42bOWEQSFY0Z18KT/k6gJmiwK0k8SNyZLt+c7/\ 1X4hxAU9E9WLe8oc3MuzEWAqDyEuTt18ng1ELTaJ+MvEzi+\ SCinGLh3AbU3BHagzwf7W5vOlOCVTwIcuHSbNcCBrsTVOfkByDMSC92CmpQ9L3Fleos+\ 0e7Ru+YCKoTBqozoC3jQUx16VcOINJmBQf8X9U6lC/1dWIApYq50j75XJ4yvfU0CGQYaO+\ 2lt0hw0TGZbsN8z1UbC8QG9jnHDPO3A5ktVzlay+E5nrRNQFR9ed73KF3I0TAZd+\ ztHfE5Z/1t1qQSxkdTiW76wTG9ASQL99EBulFjkCvsIb62S9E3ha1JQJBuO2MJu5QYIJbQ\ zeL1H5MUvbe2dbSLIf1VY6Kz8fgejhGBZlGxHjUUSTEj+aYjprrvnR80RoZ3Yp8L/\ Qqlx6S1pAjJg94jultnz+\ jGL0lliuHSWbtZ6GBnRQgVyZGWVroNfXtKV95T8u86tiQJIo0xDfUchO7j5+\ s35Zb2r9GKaiYSqeMxs9b3ypoT2Lb+dbh5hDk0BW3EVfNRUHVTYmGz9D9p6dj+\ I5wC51m1RkMN9OBApWwcvhh5SFVG24SsoKk9DOkCw7zqoIRgEfycRHjQVk2LVuqcjUvZEP\ II9HKEPSWdB2J3NJ22RpXQcvcLD+\ d2tVWaBc1NScJlxSCQTGcNmjAzBz3FXEhNKU8d3yE8fMy9NeJb04PjL0mG6MyRIkmaVDxB\ 5hqoC4yoAoMfHmQYcVwcTwIV+TmnM5HVQtCU3n2Wz40r/\ VSKERVTXXqORF9gygMtTbOuQkHHFjqJQJdtm/l+bkM4DFV6iLDulkNc0BRPBOFLb6ryU+\ bQymW7XCA9x3BXkmGLO6czA4gsTHsPnoh4CV4yZgvijmc+\ VxJKGfPjdqKXcE1mH5CN0Y21++40ufJLMus69wW7XXT2EY0gxoeHJPRw6Z9EO+gcNO4+\ Gqglyfs7bF4lAqDebGou84bXGXegq9VBPA3dN+\ fSLzdj1bfRTh0Dod6o0z1XJ8Jgr5pRgqFpS3pWa1/+\ 7iz4SBropihyjO0jyaoScQLqUfFcHRQ9YDjx8n+pggqgxejQYKp9SPRxFymhrTO/\ RCQjg6rdiAiR79Bz5g1fKptrP/\ cF6W6VQy4Hauy7xXm3rockz6cu0tqhB4xR47QmLc61L9ClWyzS97SSnttkUJZqk7kyMnZr\ 7sZTMnR+HbM6kpG6htPMLZlmRotUNV/\ oFXz5yQpCylFCAVQBik6Xp8YDBpDjbW03EYVg5lwuMXrmwE4mYlHss6HAkddT0jVUzRqhv\ TSmIyv4hZyESy7OpPM5bH4vct3rnOTkm35Z6VIDDYAV0ynnJCufuXWST72QPI8YyX6QUzt\ UdVtOaL+IDruTLx5JC+CDONUZh/5DTchPy+\ azKkFB4QROW3MUPfIXD3QWaguIuLx1NsZ1HKfB5O4TG5ZMd64qCk+z6g1Hh9qrJgjQ+\ NtJR0HoiB+TorW3iCUX6ZLSY6Jv9oKAgBkFOjxC9YYy55XAWTrXqLts3ZID8kgs/\ TXEEe3/ItdCau5TMQjZQ95qvJTkIYrJ6Rg08JgrPcgabrLNaHCzVZ94e9cwSSENNsVJx5T\ 6mpOJk205SiREozK1ZHsQM2pvYRVTuU4+\ 5ctR0MsOmiSn4lm3HM7QKTMLOVsnJ0CKeGFOhp0hVPKZtFgHoNzGZW3yMpul38lacqpHJx\ 6O5t5YXJlsTn7mYClszz2mVShDgBZx1WlytOHMRUiBysTrpBYpOnDa0pPGAyeqrKTzDBNd\ KF/plSxGiIyI/zjwgFEW5FJKAxeB+JlOCNsBasZD2NViL1y1NKK+fKhA/qqVhnQ/Rm/\ z5mwjYUfHmfqFlh6GyhwXm0JflMdzUh7xWqA8HsNlNv6VjICaTp3NSFlP4LjJg3HSmkjYI\ eZHOr2wqck1X5va0yWj5JmEK4rDVq9lgh3ka6JnIESp12RHPVRPq+\ aVEmxBcpgD17cdZiJySFYpicRR9aEToB/7m9WyMAh64E0DrycSw+\ dBwMZ8fpmoXI1HzmZSKEevA4WVZCvX9wkTkjMbvjjmu3X/x/\ 23qwVoOG0IUIV6ebpIgYMukiQtnW0XcuZ9lmrzBYE/\ aunb8chLTFqaooNc4peUkJP27EQu4/\ kT8jPQF9HOp5FyzYL3E1uccK6dqsxyhopaqGkOeKfoatzJM966aBv5kFdDWz0U9xolxA5k\ ia5PtqAJcx2zLmhC9fJiIsd00Tx4Rr9y5KEz7ZYEFFCWa5ImZfA/+\ niCEe012qFUBGdyYOZwqff0AuqWX6cLS8DnbFOnPQnMR/\ GmFLBz6iJlpsq2t4j5VaJvJolprb9zRQNG2LOwfWnPPRyeUpx/\ 0spqhHoYYGFMmuKKQXvjvmZPn6rXG+0FnE6qojrqto546F5xToBRW7a/\ F0mU12QBP1EJndibwZdU7N2sLkC6xpIgZtQtyY5VZevuR9hXJNXJtmxW99Ci2+\ 4RALctxvxDa/z2+\ fyKkRyhTjy6GG3LQBzXWwM9MbaAX4ZoBndGYzxGtifbbpDicB2jma1kL16MZgu20dXGqXT\ EzBvth9NR3o4Juvt7LyGMhHxTZfZT6jmoIJtyKh7KSLdNS3wkZ8xZxHApO5f3pC4ehe26J\ S+x6dElnJPJ+31fqvqzPQzMCKXRaxQTKdByHG/\ uQqLTxwF1WRet6joNMt4vfaDereut3DVHRLW68av1eggR3iO5QLxZkv0/\ yOvUsUTrzK55qqSudWEuw54mO5kTa4knBezEXcX7SThRFmvCu1olcDq9gJjN5XKwtNPY7k\ pL5940jqFSkSCBGUn7THYWXRRde2wx5UayIkwD39aehv7uiXYABrzHzbN078hmHKLXyD8g\ TNXJFWxVeoam2t+DfpMso2QlX7hrcHUribUkuRieSfOKRyOxrGYxdtDDcb+\ 8g0BGvprU65Sy4NTxTmwy+UzcuSfXJA9/\ j1uCyw03hgsN4QzUKp7uq9r8V6QBKQmk29lM2/\ KA8bQcqvJLvNcczzZIoSjUHu2ivTGE0rM9ZVr/8euikxcXWnrEbhtp/\ S411dCve5K9nL9CTbbJX1GI0UOL6Y2L9VSbAY+\ RhL4SF7Ps5NIB32XiGEBabcRScReXbWe8f9BzxDbE77nvMx2yOn6yiMKON8HjbvRtsOmRX\ fdwcojxRqO8tiQLSJmmHc0o1nVI0Rq+\ o4NacrV7LilgpMTvPaEEUGyoomRXSgq5UNBm7cfk44auwZo4Wac0/\ kNF1YjG7ytaR9lZUfTdgXjXMscsitsdJ2X1G3reU6WXiaef2jVpU9bNUUW2ubLjlMvcUle\ IrXkPCMaUHCBB1ak00jPcYPUBLBwxIX5Yfm8+\ HqEoVc5FpRR57bC7AqaHsBu3pUjEjG6JlyHPgscu5rKCxmzbpnsqUYTFjK2v8HE6Ec0Nre\ SpGCkcFx0JmD7aaSd+tZoEXXLYfYs3j6MkS7a3fbEKgC66a+Pesgerw1kKIB7OdBEqmL+\ D+u/vPWdROXrxrpdlFI1lJzFjqELdB2WPSsbfjfYkg1Nt+\ jLvNExFtU0T33IyT1dZt3eyGsKUsYYwZp/\ YPxo56c1Gtg7H4gOPjS37tlw1hGekRcrWTIz79gipLJx47khd33OoWxh3HCAlWZKxVaeV7\ nN4jsNAoSRZtvsybpETqlK4lZvXVLELw+\ RMhW2vmMBIDBJnzqVR4VNacgYKAdy4kTqdd6J9/\ ph567aFFgf49OOmypo0oPutVAMAHQoLKaOg+\ ZfcjzVXT2Tgh5P2yokpzTZZv3WErYiu9llrKJ5SoEUrUgbCGVPJG0ebdiCN8AL+\ OjfqVwaTEMktRDhuHzqNCANbQFFychpCZ56IKS8WXpUsfSaEk0BX3enZmCnk1lAOIiXObW\ RWIX5+fnTzvUF3DKvNVmMbbdQcnIrqJ3jWXC7LdmT7LOkxqHlaUvcS1R61mq7jWAwBTJp1\ GoDx0CpS5nxoAAp8XpOQyKV/EYqesDZZw1LL8Q9vsFa4a4fWetbzc/\ dK3EZpf8y5H14LolFCplPQS+LjtVmOcFIfoqOdugq4X9Rt/\ Hmy0KQr97lkntyGHCphTcnb69yzNcv+6EFEAarBpHmtBK4n0xIdzlWEr+M2vgrsMmI/\ 97xZRD7R2JZDMkrFpTVHh8Q62+\ EHN5lnwavbxQkT0kwpEac0Aj6Srlu05j26k0pCgOK21jcCT7jzxQIKi0LoWSdBUDlwkao2\ +QEecHF23+ADZp0e4ragrtHzBah67kR/pp0Nu/\ gDXE6td1DmDkkY8tKX56bjC56gZoxnGnI28AriNmqXk4JMHgr/AnbqwHCNh8Lu5/\ MQnZcJQbh4MNlWLsZwmqbJABSeJqfaUym4rrptug7gZ1puGMGcFo0I43F7swcszcHlZrh3\ Uu8ZLbXbkmw3iT5iZalTfOxi9V/GBPig2JlsS/kgnP+\ iF0g9CJEseF5jFHp8aAySwlxKs7IVjmxuZ62NbwE1JYHf0wVer+\ gS0rnXFsbzOPNpl9rm2ThCPVyWfNBt6uPJKHebmXhRGwlLCRBAS5sqWR9XK4V/\ SJJxiM7CDSukzNNUxecrZqn/TJcG0PUVKQ96STqAzO9WJrB0+n89VyxNXf+\ 0GUKEiQ4t2a7O7UZg+WhJ6nDj/KiSrvaCfiI8vkw61+\ kcgiUm6FdqUsVD9JvKlpBfc9AtabaI5lMUrWmvowVqRLe1krbaTAttV7u0a7JDOcaaNMgy\ 7BDULBl6JjlUu0x9StgPyhIjDRzdaQxuTDtFjmq2RtnhRRMGmx3Bjssde8GnYXqTUR/\ xt8i5kVeYelwkm0RQfl/\ XRnYauTxn8Xj0ks5srP3XRg54Ns3GPYbCnidBkMlhrhR7YlwbZ/wSUXg/\ 1uFzhJFlZHeTrSQe+\ 7PLvP7eePMkuYMqyHvtcgOJoGxhC6GbTh9GbpSdjLAeMoKbVlKh7AYldZNQx9I/\ RKeTnWnQvDa69n5g4LLZ4IgLUb1pVREGRdM55G2DgqcK9PrBPBwCOTR6Z623I7ZWvME5d+\ jJxRNUliJRkqHWzwGueKdZauTq1zlHBGxrJw9urYf2xOM8RtKrTjtPZnukXq9XBRXwW9yM\ V0C5kFFwCF6r8RjJoj/CXZzpHhi0cdI2x67b5q0jN2dKDrchbI2hTYcsTIo65+\ huZ1dEqjqoVPqSU42UYFvD+qkla9OrkcvyFMZk9E/Ptgib9BTGaqQ/\ i0dSjeUjH7cxArdXDqhyIbMxyqw5mQNj7OrQwYMiKUyMkWn0Y1/c3XKLAJ/\ hEeqL0ifLxaveDiAPlM8/\ atL4ngQwpWGoyGAOd5UqY1U0fXl3HS6jRtErcQmEZIVr1qSfycVbKbYM4fxM8JWpAlT/\ GPxnm2uixnIIoRg7ZGf4bJtNh+QiC0jRI6M/\ qGth0b5vyxWeQouImTLrbs1WNjtdhDDQOPDal5F7G97xgPViYY1VH0XrP352Diq14CMHWK\ dvMDRLJsSrUVDhjKCzBjdkix6cPiRU6oFutduHJEJJ7HjfEe+VfaJkD924pqOChWwwv+\ 0cG3HpiwdbWLt17st41WM/KWHN/LNd6jGMQvLpIDUd6emVRCQ7nK+\ rcG8Hoa5aG6GfRRw39BNWae1Kh28OV+3/9SsZ+g==
Two main requirements:
- To be able to see the "suits"
- To have a similar "mutiple sheets" function like in XLSX
OpenWrite[...];Write[<couple data pages>];Close[...]
and then put there the data-page using something likeOpenAppend[...];Write[...];Close[...]
. The optimal amount of the data written at the each step can be defined experimentally. TheClose[..]
command have to be evaluated periodically to flush the file system buffers and all rest cashed memory used during the file operations. $\endgroup$