I have a file which contains 6 columns but each column has a different number of entires. The way that this file is imported into mathematica is that the 'missing' entries for each row are simply entered as an empty string. This of course is not an issue for Import, but ReadList appears to return a list with the wrong number of elements.
This is my current script:
readMyFile[filename_, colPick_:{1,2}, colLen_:2, header_:1, footer_:0,skip_:1] := Block[{strm,data},
(*
filename_ : name of the file.
colLen_ : number of columns in the file to be read. Integer if one column, a list if multiple.
colPick_ : which columns to pick
header_ : number of lines to skip starting from the top.
footer_ : number of lines to skip from the bottom
skip_ : take every 'skip'th element. set to 1 to take all. Set to 2 to take every 2nd element.
*)
strm = OpenRead[filename]; (* opean a stream with the file*)
Skip[strm, String, header]; (* drop the first line corresponding to the headers*)
data=ReadList[strm,Number&/@Range@colLen,RecordSeparators->{"\n","\t"}][[;; -(footer + 1);;skip,colPick]]; (*Read the stream, use tab separated - the default type is Expression. Unhash to specify.*)
Close[strm];
data
]
Effectively, it requires the number of columns so that I can address the second argument of ReadList, and then I can specify which columns I am interested in. The Header/Footer are to exclude potential type mismatch elements.
I keep getting inconsistent size arrays (for example a file with 240k rows, will be read as if it had 176k after ReadList.
I suspect the issue is because most rows will have the format
{Number,Number,Number,Number,Number,Number}, but after a certain row number, you will start getting {Number,Number,Number,Number,String,String} as the empty parts are interpreted as strings.
This link contains useful info, but unfortunately nothing that can help. Replacing "Number" with "Expression" does not seem to help either.
Many thanks!
EDIT: You can find a subset of the data here: https://pastebin.com/SnPQ3VMF Looking at the TableForm of the data imported via Import and that by ReadList, it seems that not only the length is different, but also that ReadList somehow fills in the gaps for places where there shouldn't be any data.