1
$\begingroup$

I have a file which contains 6 columns but each column has a different number of entires. The way that this file is imported into mathematica is that the 'missing' entries for each row are simply entered as an empty string. This of course is not an issue for Import, but ReadList appears to return a list with the wrong number of elements.

This is my current script:

readMyFile[filename_, colPick_:{1,2}, colLen_:2, header_:1, footer_:0,skip_:1] := Block[{strm,data},
(* 
filename_ : name of the file.
colLen_   : number of columns in the file to be read. Integer if one column, a list if multiple.
colPick_  : which columns to pick
header_   : number of lines to skip starting from the top.
footer_   : number of lines to skip from the bottom
skip_     : take every 'skip'th element. set to 1 to take all. Set to 2 to take every 2nd element. 
*)
strm = OpenRead[filename]; (* opean a stream with the file*)
Skip[strm, String, header]; (* drop the first line corresponding to the headers*)
data=ReadList[strm,Number&/@Range@colLen,RecordSeparators->{"\n","\t"}][[;; -(footer + 1);;skip,colPick]]; (*Read the stream, use tab separated - the default type is Expression. Unhash to specify.*)
Close[strm];
data
]

Effectively, it requires the number of columns so that I can address the second argument of ReadList, and then I can specify which columns I am interested in. The Header/Footer are to exclude potential type mismatch elements.

I keep getting inconsistent size arrays (for example a file with 240k rows, will be read as if it had 176k after ReadList.

I suspect the issue is because most rows will have the format

{Number,Number,Number,Number,Number,Number}, but after a certain row number, you will start getting {Number,Number,Number,Number,String,String} as the empty parts are interpreted as strings.

This link contains useful info, but unfortunately nothing that can help. Replacing "Number" with "Expression" does not seem to help either.

Many thanks!

EDIT: You can find a subset of the data here: https://pastebin.com/SnPQ3VMF Looking at the TableForm of the data imported via Import and that by ReadList, it seems that not only the length is different, but also that ReadList somehow fills in the gaps for places where there shouldn't be any data.

Improve this question
$\endgroup$
2
  • 2
    $\begingroup$ Please include an accessible link to the data file (if it is not too large) or include (say) 100 lines of data that are representative enough. Doing so will enable the respondents to work with concrete data. Thanks. $\endgroup$
    – Syed
    Apr 26, 2022 at 13:56
  • $\begingroup$ @Syed, thank you, I have amended the question with a pastebin sample. $\endgroup$
    – alex
    Apr 26, 2022 at 14:13

1 Answer 1

Reset to default
1
$\begingroup$

I saved the pastebin data to the C-drive and read the records first.

data = ReadList["C:/test.txt", Record, 
  RecordSeparators -> {"\n", "\r\n", "\r"}]

dataOut = Interpreter[DelimitedSequence["Number"]] /@ data;

Length /@ dataOut

> {6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 4, 4, 4,
> \ 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4}

or

dataOut2=ToExpression[StringSplit /@ data]

for a more efficient readout as file size gets larger.

Improve this answer
$\endgroup$
5
  • 1
    $\begingroup$ This is brilliant. $\endgroup$
    – alex
    Apr 27, 2022 at 8:56
  • 1
    $\begingroup$ The dataOut version is very slow for large files, but if you instead have ToExpression[StringSplit /@ data], it works very well. $\endgroup$
    – alex
    Apr 27, 2022 at 9:02
  • 1
    $\begingroup$ Interpreter function is general purpose and is often slow. The key point was to read records first and then process them. $\endgroup$
    – Syed
    Apr 27, 2022 at 9:05
  • $\begingroup$ I have taken the liberty to add this solution to your answer for clarity. let me know if this not part of the rules. $\endgroup$
    – alex
    Apr 27, 2022 at 9:15
  • $\begingroup$ Thanks for that. I took the lazy route as I didn't have to contend with the actual file size. $\endgroup$
    – Syed
    Apr 27, 2022 at 9:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.