0
$\begingroup$

Previously I asked a related question about my test data.

Looking at the size of the data

ans // ByteCount

9602605504

9602605504*8/519284064 // N

147.936

I will probably need about 150G at the moment to export to XLSX.

My data has unequal length

Dimensions /@ ans[[1 ;; 20]]

{{80, 6}, {78, 6}, {79, 6}, {79, 6}, {82, 6}, {81, 6}, {78, 6}, {82, 6}, {80, 6}, {77, 6}, {79, 6}, {80, 6}, {79, 6}, {82, 6}, {80, 6}, {81, 6}, {81, 6}, {81, 6}, {80, 6}, {81, 6}}

In theory, none of them should be more than {90,6}.

Some examples of the data entries,

ans=1:eJy1XcuyI7cNleNxxv6M/EGT1KOVnbJUJym3nWVWWbjqrrJw/r+\ iuaOZOUdNHICQvHFpyrpiNwnicXAA/OU///3ltx92u93vH27/+fvb7//77Wf615/pX9/\ jv379ePvw77/W47Fdv32e9vD59Pbd7fNvf+r/WT39+uPXr5bp7dP/+PXTH/\ z8tvv0Z7y29SMr/uD7b1hfLRU+n95+yiw3nfmZfxLrTQ0f8+37L+v9rbeetbW1/\ 7nMb59Wtf6swCmUCzxSze1ygVdpe7nLbf62Q+W2Qz/K9Yznn679bWzL+\ zaauwVfnZbNIw8/Rr1+e5dWJvgf+7cPSrIP8ByXtz9n1sYjbGf47fb+e+\ bOwVcLP8YPUgAtIT7B710/n7y1W3hQJ3wk565Zp7mQQNC92ykpuBifz+/qaPzOH+\ nefVBLL/2ly/l9veHrj0Jc8fP184/wE+O9g7WPOQHEy1ROtI0/qS1Y+/\ egwj0Y2YJa6fPXd6w3KVB3EI+ieltgqT187SqVfTui8OeUfZn7Gvd22t+pewd/\ Rnfw8m4jxk0qqJB20JuMdmbv6PrIcqfPyi3w1duVypmWY/98p8P770Vudrn2RWtErzY86+\ vnpa23RmNYv53pkGgthiWbZvnaqIAeLOqHzGujEvuiSKwdOtBrfxx4bVQSfde0NGnKyN9B\ PX76ZkIGrejG1TW/i9KVNJ0Vn/\ mAe6o98gvdh5zfAr9RWYlJy4mq9sguZE6ZHozPJ3ny9ByoWO8S+Gnn/\ nUaeQ702lGJHORzoJqq+ExeIGEdLbzLzSGTW4ACA9a8rDkXsi1stXdCAskELkmDglK31/\ ccnZVl62IOvyue9nSUar0cSSXnzAi6+\ 1P7HBNaX8WjnJNbu9CNkMvBSbZrzlajU0WWo2pbXfsHcrPVqUOdUC3OOuZH5X3I3Rf0mm9\ 2SskQnunNwU7ZJvI06/\ sjCy8CTDJ5mikdjTLCelIqCRKtOefaT6yKfxDL1SFEw1oPcbEqQ3i0wvj59qpafK1dRnnC\ 317lY+BXb2aAVHhK0Hp4XyT4pKD6mjN/hAxpBMXCzm5Cqa+\ z4YMWBo7Iw5KqBJGX81DwbPi/K/64hnL2dPaoeFNmilBafbkpHFyTyFEztvEQV2/\ 42bOWEQSFY0Z18KT/k6gJmiwK0k8SNyZLt+c7/\ 1X4hxAU9E9WLe8oc3MuzEWAqDyEuTt18ng1ELTaJ+MvEzi+\ SCinGLh3AbU3BHagzwf7W5vOlOCVTwIcuHSbNcCBrsTVOfkByDMSC92CmpQ9L3Fleos+\ 0e7Ru+YCKoTBqozoC3jQUx16VcOINJmBQf8X9U6lC/1dWIApYq50j75XJ4yvfU0CGQYaO+\ 2lt0hw0TGZbsN8z1UbC8QG9jnHDPO3A5ktVzlay+E5nrRNQFR9ed73KF3I0TAZd+\ ztHfE5Z/1t1qQSxkdTiW76wTG9ASQL99EBulFjkCvsIb62S9E3ha1JQJBuO2MJu5QYIJbQ\ zeL1H5MUvbe2dbSLIf1VY6Kz8fgejhGBZlGxHjUUSTEj+aYjprrvnR80RoZ3Yp8L/\ Qqlx6S1pAjJg94jultnz+\ jGL0lliuHSWbtZ6GBnRQgVyZGWVroNfXtKV95T8u86tiQJIo0xDfUchO7j5+\ s35Zb2r9GKaiYSqeMxs9b3ypoT2Lb+dbh5hDk0BW3EVfNRUHVTYmGz9D9p6dj+\ I5wC51m1RkMN9OBApWwcvhh5SFVG24SsoKk9DOkCw7zqoIRgEfycRHjQVk2LVuqcjUvZEP\ II9HKEPSWdB2J3NJ22RpXQcvcLD+\ d2tVWaBc1NScJlxSCQTGcNmjAzBz3FXEhNKU8d3yE8fMy9NeJb04PjL0mG6MyRIkmaVDxB\ 5hqoC4yoAoMfHmQYcVwcTwIV+TmnM5HVQtCU3n2Wz40r/\ VSKERVTXXqORF9gygMtTbOuQkHHFjqJQJdtm/l+bkM4DFV6iLDulkNc0BRPBOFLb6ryU+\ bQymW7XCA9x3BXkmGLO6czA4gsTHsPnoh4CV4yZgvijmc+\ VxJKGfPjdqKXcE1mH5CN0Y21++40ufJLMus69wW7XXT2EY0gxoeHJPRw6Z9EO+gcNO4+\ Gqglyfs7bF4lAqDebGou84bXGXegq9VBPA3dN+\ fSLzdj1bfRTh0Dod6o0z1XJ8Jgr5pRgqFpS3pWa1/+\ 7iz4SBropihyjO0jyaoScQLqUfFcHRQ9YDjx8n+pggqgxejQYKp9SPRxFymhrTO/\ RCQjg6rdiAiR79Bz5g1fKptrP/\ cF6W6VQy4Hauy7xXm3rockz6cu0tqhB4xR47QmLc61L9ClWyzS97SSnttkUJZqk7kyMnZr\ 7sZTMnR+HbM6kpG6htPMLZlmRotUNV/\ oFXz5yQpCylFCAVQBik6Xp8YDBpDjbW03EYVg5lwuMXrmwE4mYlHss6HAkddT0jVUzRqhv\ TSmIyv4hZyESy7OpPM5bH4vct3rnOTkm35Z6VIDDYAV0ynnJCufuXWST72QPI8YyX6QUzt\ UdVtOaL+IDruTLx5JC+CDONUZh/5DTchPy+\ azKkFB4QROW3MUPfIXD3QWaguIuLx1NsZ1HKfB5O4TG5ZMd64qCk+z6g1Hh9qrJgjQ+\ NtJR0HoiB+TorW3iCUX6ZLSY6Jv9oKAgBkFOjxC9YYy55XAWTrXqLts3ZID8kgs/\ TXEEe3/ItdCau5TMQjZQ95qvJTkIYrJ6Rg08JgrPcgabrLNaHCzVZ94e9cwSSENNsVJx5T\ 6mpOJk205SiREozK1ZHsQM2pvYRVTuU4+\ 5ctR0MsOmiSn4lm3HM7QKTMLOVsnJ0CKeGFOhp0hVPKZtFgHoNzGZW3yMpul38lacqpHJx\ 6O5t5YXJlsTn7mYClszz2mVShDgBZx1WlytOHMRUiBysTrpBYpOnDa0pPGAyeqrKTzDBNd\ KF/plSxGiIyI/zjwgFEW5FJKAxeB+JlOCNsBasZD2NViL1y1NKK+fKhA/qqVhnQ/Rm/\ z5mwjYUfHmfqFlh6GyhwXm0JflMdzUh7xWqA8HsNlNv6VjICaTp3NSFlP4LjJg3HSmkjYI\ eZHOr2wqck1X5va0yWj5JmEK4rDVq9lgh3ka6JnIESp12RHPVRPq+\ aVEmxBcpgD17cdZiJySFYpicRR9aEToB/7m9WyMAh64E0DrycSw+\ dBwMZ8fpmoXI1HzmZSKEevA4WVZCvX9wkTkjMbvjjmu3X/x/\ 23qwVoOG0IUIV6ebpIgYMukiQtnW0XcuZ9lmrzBYE/\ aunb8chLTFqaooNc4peUkJP27EQu4/\ kT8jPQF9HOp5FyzYL3E1uccK6dqsxyhopaqGkOeKfoatzJM966aBv5kFdDWz0U9xolxA5k\ ia5PtqAJcx2zLmhC9fJiIsd00Tx4Rr9y5KEz7ZYEFFCWa5ImZfA/+\ niCEe012qFUBGdyYOZwqff0AuqWX6cLS8DnbFOnPQnMR/\ GmFLBz6iJlpsq2t4j5VaJvJolprb9zRQNG2LOwfWnPPRyeUpx/\ 0spqhHoYYGFMmuKKQXvjvmZPn6rXG+0FnE6qojrqto546F5xToBRW7a/\ F0mU12QBP1EJndibwZdU7N2sLkC6xpIgZtQtyY5VZevuR9hXJNXJtmxW99Ci2+\ 4RALctxvxDa/z2+\ fyKkRyhTjy6GG3LQBzXWwM9MbaAX4ZoBndGYzxGtifbbpDicB2jma1kL16MZgu20dXGqXT\ EzBvth9NR3o4Juvt7LyGMhHxTZfZT6jmoIJtyKh7KSLdNS3wkZ8xZxHApO5f3pC4ehe26J\ S+x6dElnJPJ+31fqvqzPQzMCKXRaxQTKdByHG/\ uQqLTxwF1WRet6joNMt4vfaDereut3DVHRLW68av1eggR3iO5QLxZkv0/\ yOvUsUTrzK55qqSudWEuw54mO5kTa4knBezEXcX7SThRFmvCu1olcDq9gJjN5XKwtNPY7k\ pL5940jqFSkSCBGUn7THYWXRRde2wx5UayIkwD39aehv7uiXYABrzHzbN078hmHKLXyD8g\ TNXJFWxVeoam2t+DfpMso2QlX7hrcHUribUkuRieSfOKRyOxrGYxdtDDcb+\ 8g0BGvprU65Sy4NTxTmwy+UzcuSfXJA9/\ j1uCyw03hgsN4QzUKp7uq9r8V6QBKQmk29lM2/\ KA8bQcqvJLvNcczzZIoSjUHu2ivTGE0rM9ZVr/8euikxcXWnrEbhtp/\ S411dCve5K9nL9CTbbJX1GI0UOL6Y2L9VSbAY+\ RhL4SF7Ps5NIB32XiGEBabcRScReXbWe8f9BzxDbE77nvMx2yOn6yiMKON8HjbvRtsOmRX\ fdwcojxRqO8tiQLSJmmHc0o1nVI0Rq+\ o4NacrV7LilgpMTvPaEEUGyoomRXSgq5UNBm7cfk44auwZo4Wac0/\ kNF1YjG7ytaR9lZUfTdgXjXMscsitsdJ2X1G3reU6WXiaef2jVpU9bNUUW2ubLjlMvcUle\ IrXkPCMaUHCBB1ak00jPcYPUBLBwxIX5Yfm8+\ HqEoVc5FpRR57bC7AqaHsBu3pUjEjG6JlyHPgscu5rKCxmzbpnsqUYTFjK2v8HE6Ec0Nre\ SpGCkcFx0JmD7aaSd+tZoEXXLYfYs3j6MkS7a3fbEKgC66a+Pesgerw1kKIB7OdBEqmL+\ D+u/vPWdROXrxrpdlFI1lJzFjqELdB2WPSsbfjfYkg1Nt+\ jLvNExFtU0T33IyT1dZt3eyGsKUsYYwZp/\ YPxo56c1Gtg7H4gOPjS37tlw1hGekRcrWTIz79gipLJx47khd33OoWxh3HCAlWZKxVaeV7\ nN4jsNAoSRZtvsybpETqlK4lZvXVLELw+\ RMhW2vmMBIDBJnzqVR4VNacgYKAdy4kTqdd6J9/\ ph567aFFgf49OOmypo0oPutVAMAHQoLKaOg+\ ZfcjzVXT2Tgh5P2yokpzTZZv3WErYiu9llrKJ5SoEUrUgbCGVPJG0ebdiCN8AL+\ OjfqVwaTEMktRDhuHzqNCANbQFFychpCZ56IKS8WXpUsfSaEk0BX3enZmCnk1lAOIiXObW\ RWIX5+fnTzvUF3DKvNVmMbbdQcnIrqJ3jWXC7LdmT7LOkxqHlaUvcS1R61mq7jWAwBTJp1\ GoDx0CpS5nxoAAp8XpOQyKV/EYqesDZZw1LL8Q9vsFa4a4fWetbzc/\ dK3EZpf8y5H14LolFCplPQS+LjtVmOcFIfoqOdugq4X9Rt/\ Hmy0KQr97lkntyGHCphTcnb69yzNcv+6EFEAarBpHmtBK4n0xIdzlWEr+M2vgrsMmI/\ 97xZRD7R2JZDMkrFpTVHh8Q62+\ EHN5lnwavbxQkT0kwpEac0Aj6Srlu05j26k0pCgOK21jcCT7jzxQIKi0LoWSdBUDlwkao2\ +QEecHF23+ADZp0e4ragrtHzBah67kR/pp0Nu/\ gDXE6td1DmDkkY8tKX56bjC56gZoxnGnI28AriNmqXk4JMHgr/AnbqwHCNh8Lu5/\ MQnZcJQbh4MNlWLsZwmqbJABSeJqfaUym4rrptug7gZ1puGMGcFo0I43F7swcszcHlZrh3\ Uu8ZLbXbkmw3iT5iZalTfOxi9V/GBPig2JlsS/kgnP+\ iF0g9CJEseF5jFHp8aAySwlxKs7IVjmxuZ62NbwE1JYHf0wVer+\ gS0rnXFsbzOPNpl9rm2ThCPVyWfNBt6uPJKHebmXhRGwlLCRBAS5sqWR9XK4V/\ SJJxiM7CDSukzNNUxecrZqn/TJcG0PUVKQ96STqAzO9WJrB0+n89VyxNXf+\ 0GUKEiQ4t2a7O7UZg+WhJ6nDj/KiSrvaCfiI8vkw61+\ kcgiUm6FdqUsVD9JvKlpBfc9AtabaI5lMUrWmvowVqRLe1krbaTAttV7u0a7JDOcaaNMgy\ 7BDULBl6JjlUu0x9StgPyhIjDRzdaQxuTDtFjmq2RtnhRRMGmx3Bjssde8GnYXqTUR/\ xt8i5kVeYelwkm0RQfl/\ XRnYauTxn8Xj0ks5srP3XRg54Ns3GPYbCnidBkMlhrhR7YlwbZ/wSUXg/\ 1uFzhJFlZHeTrSQe+\ 7PLvP7eePMkuYMqyHvtcgOJoGxhC6GbTh9GbpSdjLAeMoKbVlKh7AYldZNQx9I/\ RKeTnWnQvDa69n5g4LLZ4IgLUb1pVREGRdM55G2DgqcK9PrBPBwCOTR6Z623I7ZWvME5d+\ jJxRNUliJRkqHWzwGueKdZauTq1zlHBGxrJw9urYf2xOM8RtKrTjtPZnukXq9XBRXwW9yM\ V0C5kFFwCF6r8RjJoj/CXZzpHhi0cdI2x67b5q0jN2dKDrchbI2hTYcsTIo65+\ huZ1dEqjqoVPqSU42UYFvD+qkla9OrkcvyFMZk9E/Ptgib9BTGaqQ/\ i0dSjeUjH7cxArdXDqhyIbMxyqw5mQNj7OrQwYMiKUyMkWn0Y1/c3XKLAJ/\ hEeqL0ifLxaveDiAPlM8/\ atL4ngQwpWGoyGAOd5UqY1U0fXl3HS6jRtErcQmEZIVr1qSfycVbKbYM4fxM8JWpAlT/\ GPxnm2uixnIIoRg7ZGf4bJtNh+QiC0jRI6M/\ qGth0b5vyxWeQouImTLrbs1WNjtdhDDQOPDal5F7G97xgPViYY1VH0XrP352Diq14CMHWK\ dvMDRLJsSrUVDhjKCzBjdkix6cPiRU6oFutduHJEJJ7HjfEe+VfaJkD924pqOChWwwv+\ 0cG3HpiwdbWLt17st41WM/KWHN/LNd6jGMQvLpIDUd6emVRCQ7nK+\ rcG8Hoa5aG6GfRRw39BNWae1Kh28OV+3/9SsZ+g==

Two main requirements:

  1. To be able to see the "suits"
  2. To have a similar "mutiple sheets" function like in XLSX
$\endgroup$
8
  • $\begingroup$ But here in particular, what is the end goal of the data? Do you want to be able to open it up in excel? Another program? Or just back into Mathematica? $\endgroup$
    – Jason B.
    Nov 16, 2017 at 22:24
  • $\begingroup$ @JasonB. To be able to "View" it using free/common package. Don't have to be Excel. $\endgroup$ Nov 16, 2017 at 22:53
  • $\begingroup$ @JasonB. To note this is NOT a duplicate as my data is non-numerical (most of it). Also, I have want to have a single file that holds the 200,000 sheets. $\endgroup$ Nov 16, 2017 at 22:57
  • 1
    $\begingroup$ there doesn't seem to be a question here at all..?? What are you asking? $\endgroup$
    – george2079
    Nov 17, 2017 at 0:18
  • 1
    $\begingroup$ The common way to "export" such big datasets is to write it binary in step bu step manner. I mean, you should create a file using OpenWrite[...];Write[<couple data pages>];Close[...]and then put there the data-page using something like OpenAppend[...];Write[...];Close[...]. The optimal amount of the data written at the each step can be defined experimentally. The Close[..] command have to be evaluated periodically to flush the file system buffers and all rest cashed memory used during the file operations. $\endgroup$
    – Rom38
    Nov 17, 2017 at 8:03

0