Convert a huge txt-file into a dataset

Question

My friend has this huge txt-log of sea levels. He wants to organize it into a dataset.

After importing it this file a used StringSplit to separate it into rows, then to singular elements

rawData = Import["rawData.txt"];
splitRawData = StringSplit[rawData, "%%"];
dataIwant = splitRawData[[19]];
FullForm[dataIwant];
splitDataIntoRows = StringSplit[dataIwant, "\n"];
splitData1 = StringSplit[splitDataIntoRows, " "];

I want to use this function to split the data into 6 columns.

convertListToAssociation = 
 list \[Function] 
  AssociationThread[{"Time (kyr BP)", "Sea level (m)", "T_NH(deg C)",
"T_dw (deg C)", "delta_w", "delta_T"}, list]

What are further steps to be taken?

Answer 1

A slightly different approach is to split the data into lines first, then split each line into fields. Since we know the data begins on line 20, we can do this

rawData = Import["rawData.txt", Path -> NotebookDirectory[]];
textLines = StringSplit[rawData, "\n"];
dataIwant = ToExpression[StringSplit /@ textLines[[20 ;;]]];

We used ToExpression to convert from text strings to numbers. Now we can put the numbers into an association. We probably want to use the first column, time, as our key, but floating point numbers are not good keys. So don't do this

poor = Association @@ (First[#] -> Rest[#] & /@ dataIwant);
poor[-39999.8]

If you get the right answer, it was just luck. A better way to treat this data is to convert the time from floating point kiloyears to integer centuries. Then we can create a better association like this

better = Association @@ (Round[10 First[#]] -> Rest[#] & /@ dataIwant);

Now our keys are exact numbers, but we still want to use kiloyears, so we write a function that converts our time in kiloyears to centuries and rounds off for us, like this

getData[kyr_] := better[Round[10 kyr]]
getData[-3999.8123]

(*   {68.766, 27.806, 4.047, -1.184, 2.377}   *)

Alternate versions of getData could interpolate the data or just give specific columns.

Answer 2

You actually should work with arrays in this case as the dataset is well-strcutured and quite large. You can import the table in one go as follows.

data = Import[
   "rawData.txt",
   "Table",
   "HeaderLines" -> 19
   ];

columns = Transpose[Developer`ToPackedArray[N[data]]];

I extracted only the data columns without column titles so that they can be stored in a packed array. This should speed up further working with the data.