5
$\begingroup$

I want to import some data that has been formatted using C language. I am not sure if there is a way to read in data by specifying column widths in WL. The data has been formatted using "%5d%-5s%5s%5d%8.3f%8.3f%8.3f\n" integer (5 positions) string (5 characters) string (5 chars) integer (5 chars) real (8 chars with 3 decimal places) real (8 chars with 3 decimal places) real (8 chars with 3 decimal places). To give some examples,

    1SOL   HW1    2  -0.542  -0.399  -0.468
   43NA     NA32678   1.224  -0.131   0.941
 1048ISO   H8332635  -3.331  -1.372   3.843

Specifically, the last row should be interpreted as "1048", "ISO", "H83", "32635", and the last 3 as real numbers. I appreciate any help provided by the community.

Thank you,

$\endgroup$
3
  • $\begingroup$ StringTake? StringSplit? $\endgroup$ Jun 10, 2019 at 21:25
  • $\begingroup$ I thought the file data might have been tab delimited (tab separated value). If it was, then you could just use "TSV" as the type on Import and it will do what you want. Can you coerce the data into TSV format? Otherwise, I think the previous comment is the way to go with the caution that I think you'll find everything is coming in as strings so you'll have to convert them to Expressions. $\endgroup$
    – Mark R
    Jun 10, 2019 at 23:14
  • $\begingroup$ Sorry, I wasn't reading your initial problem statement carefully enough. You have explicit positions for the data. I do notice that the sum of the sizes of the fields seems to be greater than the test data that you provided. Total@{5, 5, 5, 5, 8, 8, 8} is 44 and the lines have 43 characters. Looking at the data, I think the first field is coming out as 4. I'll propose a solution with {4,5,5,5,8,8,8} below. $\endgroup$
    – Mark R
    Jun 10, 2019 at 23:38

3 Answers 3

1
$\begingroup$

Given a data assignment similar to what you show, you could convert it to a tab separated value (TSV) file using logic similar to this and then the import would work just as you'd hoped.

fieldSizes = {5, 5, 5, 5, 8, 8, 8};
originalFileName = "testData2.txt";
originalData = ReadList[originalFileName, String]; 
partitionedData = 
 With[{aLine = #, 
     fieldRanges = (# + {1, 0}) & /@ Partition[Accumulate @
         Prepend[fieldSizes,0], 2, 1]},
    StringTrim[#] & /@ StringTake[aLine, fieldRanges]
    ] & /@ originalData;
newFileName = StringSplit[originalFileName, "."][[1]] <> ".tsv";
Export[newFileName, partitionedData]
formattedData = Import[newFileName, "TSV"]

After making sure that the input file conforms to the sizes listed, I get this for your original data:

{{1, "SOL", "HW1", 2, -0.542, -0.399, -0.468}, {43, "NA", "NA", 32678,{{1, "SOL", "HW1", 2, -0.542, -0.399, -0.468}, {43, "NA", "NA", 32678,1.224, -0.131, 0.941}, {1048, "ISO", "H83", 32635, -3.331, -1.372, 3.843}}

I think this is what you want. Here was my reformatted input data to get this to work (I think your cut and paste missed a few characters.

    1SOL    HW1    2  -0.542  -0.399  -0.468
   43NA   NA   32678   1.224  -0.131   0.941
 1048ISO    H8332635  -3.331  -1.372   3.843
$\endgroup$
5
  • $\begingroup$ A slight issue is that the 2nd field is left justified while the 3rd field is right justified which is the default. This comes from the "%-5s%5s" format string. I am not sure why that was done for the original input data. $\endgroup$
    – Somos
    Jun 11, 2019 at 0:38
  • $\begingroup$ Good to learn - I am unfamiliar with Fortran formatting. I think the StringTrim fixes this, right? $\endgroup$
    – Mark R
    Jun 11, 2019 at 0:51
  • $\begingroup$ And I will note that the original input data didn't appear to have quite the right format - I had to massage it so it was in the {5,5,5,5,8,8,8} format originally specified. If I take what was in your code post and StringTake with the sposlist, it generates an error. I think it will work OK if you use what I wrote into the bottom of my answer. $\endgroup$
    – Mark R
    Jun 11, 2019 at 0:54
  • $\begingroup$ Oops! Correct again. I was off by a space in my original edit. By the way, it is "C" print format spacing. Fortran is another story. $\endgroup$
    – Somos
    Jun 11, 2019 at 0:59
  • $\begingroup$ It will be much more efficient to just use StringCases once you’re already reading the entire string into memory... $\endgroup$
    – b3m2a1
    Jun 11, 2019 at 15:27
3
$\begingroup$

There are a few ways to import data into Mathematica as a list of strings. So here is just one example using a string to simulate an input file :

stringlist = ReadList[  StringToStream[
"    1SOL    HW1    2  -0.542  -0.399  -0.468 \n
   43NA      NA32678   1.224  -0.131   0.941 \n
 1048ISO    H8332635  -3.331  -1.372   3.843 "], 
    Record, RecordSeparators -> "\n"];
stringlist // InputForm

which returns the following

{"    1SOL    HW1    2  -0.542  -0.399  -0.468 ", 
 "   43NA      NA32678   1.224  -0.131   0.941 ", 
 " 1048ISO    H8332635  -3.331  -1.372   3.843 "}   

We can extract the data from each string using a list of positions

sposlist = (# + {1, 0}) & /@ Partition[Accumulate @
    {0, 5, 5, 5, 5, 8, 8, 8}, 2, 1];    
(* {{1, 5}, {6, 10}, {11, 15}, {16, 20}, {21, 28}, {29, 36}, {37, 44}} *)

and the function StringTake:

rawdata = StringTake[ stringlist, sposlist];
rawdata // InputForm

with results:

{{{"    1", "SOL  ", "  HW1", "    2", "  -0.542", "  -0.399", "  -0.468"}},
 {{"   43", "NA   ", "   NA", "32678", "   1.224", "  -0.131", "   0.941"}},
 {{" 1048", "ISO  ", "  H83", "32635", "  -3.331", "  -1.372", "   3.843"}}

Now we postprocess the raw data to its proper format:

data = Join[{ToExpression@#[[1]]}, #[[2 ;; 3]], ToExpression@#[[4 ;; 7]]] & /@ rawdata;
data // InputForm

which returns

{{1, "SOL  ", "  HW1", 2, -0.542, -0.399, -0.468}, 
{43, "NA   ", "   NA", 32678, 1.224, -0.131, 0.941}, 
{1048, "ISO  ", "  H83", 32635, -3.331, -1.372, 3.843}}    
$\endgroup$
6
  • $\begingroup$ I like what this solution. If you are repeatedly using the data, you might find it beneficial to convert them to a readily importable format, such as TSV. If you are doing it once, then the solution above avoids the back and forth with files. $\endgroup$
    – Mark R
    Jun 11, 2019 at 0:16
  • $\begingroup$ You may or may not want to add StringTrim to the strings. The above is obviously not trimming the strings so you do get things like "SOL ". Additionally, I think the first column is an integer, so it should probably be converted with ToExpression as you did with the other fields. $\endgroup$
    – Mark R
    Jun 11, 2019 at 0:18
  • $\begingroup$ @MarkR Oops! You are correct about 1st field. Fixed it. $\endgroup$
    – Somos
    Jun 11, 2019 at 0:19
  • $\begingroup$ Another minor issue I see is that you used "==" for sposlist. $\endgroup$
    – Mark R
    Jun 11, 2019 at 0:30
  • $\begingroup$ @MarkR Oops! You are correct. I was doing something else. $\endgroup$
    – Somos
    Jun 11, 2019 at 0:33
0
$\begingroup$

It's pitty that Mathematica doesn't support this import directly, but as the other answers pointed out, the key is to import the string successively according to the partitions of the string.

Typically, Mathematica will get the correct rows with ReadList, so we can focus on constructing the rule of one line. Here is my solution (for v10.2 and later):

lineString="    1SOL   HW1    2  -0.542  -0.399  -0.468";
FoldPairList[{StringTake[#1, #2], StringDrop[#1, #2]} &, lineString, {5,5,5,5,8,8,8}]

A little bit work of encapsulation will make our daily lives easier:

LineRagSplit[str_String, ragLen : {_Integer?Positive ..}, offset : _Integer?NonNegative : 0] := 
 Module[{targetStr, newRagList},

  targetStr = StringDrop[str, offset];

  (*Most of the case, the string length will be exactly the sum of the partitions*)
  If[StringLength[targetStr] == Total[ragLen],

   FoldPairList[{StringTake[#1, #2], StringDrop[#1, #2]} &, targetStr, ragLen],

   (*If it is not the case 
   Find the smallest j such that Total@ragLen[[;;,j]] \[GreaterEqual] 
   StringLength[targetStr]
   If the StringLength[targetStr] > the total of ragLen, 
   then the rest of the targetStr will be discarded*)

   newRagList = Pick[ragLen, Positive@Most@FoldList[#1 - #2 &, StringLength[str], ragLen]];

   (*The last element should be encapsulated with UpTo *)
   newRagList = #1~Join~(UpTo /@ #2) & @@ 
     TakeDrop[newRagList, Length@newRagList - 1];

   FoldPairList[{StringTake[#1, #2], StringDrop[#1, #2]} &, targetStr,
     newRagList]
   ]
  ]
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.