# ReadList problem related to Record type

A simple file reading issue is baffling me here. I am reading a around 200 MB textual file in .dat format. Now the file contains many lines but to phrase my problem I am taking here first two lines from the example file.

str = "Tue 1 Jan 2013 23 : 00 : 01; 17; {}; 32.5; 0.\nTue 2 Jan 2013 2 : 20 : 01; 47; {3,4}; 3.5; 110.";


Now I want to use ; and newline \n as my RecordSeparators and it works as expected.

ReadList[StringToStream[str],{Record, Record, Record, Record, Record},
RecordSeparators -> {"\n", ";"}]


{{Tue 1 Jan 2013 23 : 00 : 01, 17, {}, 32.5, 0.},{Tue 2 Jan 2013 2 : 20 : 01, 47, {3,4}, 3.5, 110.}}

But if I want to specify the Type of each separate Record using for example {String, Number, Expression, Number, number} above code fails to work. The error is the following.

Read::readn: Invalid real number found when reading from StringToStream[Tue 1 Jan 2013 23 : 00 : 01; 17; {}; 32.5; 0. Tue 2 Jan 2013 2 : 20 : 01; 47; {3,4}; 3.5; 110.]. >>

Is this problem solvable or I need to change the formatting of my input .dat file in a way so that it works with ReadList?

PS: If I need to change the file format what will be best separator to use in this context?

-

Your problem is that you've got a slight misunderstanding of the different types of items that ReadList can read. That's OK, it can be a little confusing.

To begin with: String, Number, Expression, etc. are not sub-types of Record. They are all separate types with their own rules for how they are read. The RecordSeparators option is only applied to Records and Words.

Probably if you have some complicated input format whose parsing is best controlled by RecordSeparators/WordSeparators, you should just use Record/Word types, which will give you strings; afterwards, convert the ones you know to be numeric by using ToExpression.

On the other hand, if you want to gain power over this area of Mathematica, read on.

Let's make up some terms to help explain things. At their base, the Mathematica functions Read, ReadList, and Skip read in ITEMS from an input stream. An Item is a number like 3.14159e-26, or a string like "peg and awl". There are different TYPES of Items: Record, Word, String, Number, Real, Character, and Byte. These Types correspond to either a string or a number, with different rules for how the input is parsed. There is also an Expression Type which corresponds to an Item which is a general Mathematica expression of any form.

## WHAT ARE THE ITEMS AND HOW DO THEY WORK?

The simplest cases:

• a Character is a single character from the stream, represented as a one-letter string.
• a Byte is a single character from the stream, represented as an integer that is the Character Code of the character.

Both Character and Byte have the same rule -- read one character -- but different representations, string vs. number.

To describe the other Item Types, you need to understand that they are all assembled from sequences of characters read from the stream, with some particular character that marks their end. That character is not part of the Item being read! It is a TERMINATOR, or TERMINATING CHARACTER. We say that the Item was TERMINATED by a particular character in the stream.

• a Record is a string, a sequence of characters terminated by a RecordSeparator.
• a Word is a string, a sequence of characters terminated by a WordSeparator, RecordSeparator, or TokenWord.
• a String is a sequence of characters terminated by a newline (\n character). Basically it's a LINE of text input, starting at the current stream position.
• a Number is any sequence of characters that can be interpreted as a number (in Fortran syntax), terminated by any character that can't be part of the number. Any whitespace (spaces, newlines, tabs) preceding the number is quietly skipped over first.
• a Real is the same as a Number but it's always a floating-point value, never an integer.
• an Expression is a sequence of one or more newline-terminated lines that form a parseable Mathematica expression. It's terminated by whatever newline ends the last line of the expression. If you've ever typed in a multi-line input to the raw text kernel, you know how this works.

Records, Words, Strings, and Characters become Mathematica strings.
Numbers, Reals, and Bytes become Mathematica numbers.
Expressions become Mathematica expressions.

Numbers, Reals, Strings, and Expressions pay no attention to RecordSeparators and WordSeparators. They have their own rules for when they stop taking characters from the stream.

(The end of the stream, represented in Mathematica by the symbol EndOfFile, is nearly always a terminator. It's not a character, though.)

## OBJECTS: GROUPS OF ITEMS

I have just told you the only Types of Items that can be read. However, there's another term that has to be introduced. The second argument of Read, ReadList, and Skip -- the input specification -- can be a complex expression which contains one or more of these Types. Let's call that an OBJECT. For instance,

Read[stream, {String, Number, Plus[Number, Real], Hold[Expression]}]


reads an Object: a sequence of five Items. Several of the Items are placed inside larger expressions, and the whole thing is placed inside a List head.

The degenerate case of an Object is a single naked Item: ReadList[stream, Byte]

If you don't specify a second argument to Read, ReadList, or Skip, it defaults to Expression. ReadList[stream] == ReadList[stream, Expression]

Read, ReadList, and Skip proceed left to right through the Object; each Item Type they encounter causes an Item to be read from the stream. As I listed above, each Type has its own rules for how many characters it will snatch up, what it will do with them, and when it will stop.

If you are constructing complex Objects consisting of several Item Types, you need to know especially when will each one stop. This requires understanding the TERMINATORS for each type. Just as importantly, you need to know what is done with those terminating characters.

## WHAT HAPPENS TO THE CHARACTERS THAT TERMINATE AN ITEM

Terminating characters are not part of the Item that is read. They simply mark that Item's end in the stream. Different Types apply different rules to how they treat the terminator -- that is, where they leave the position of the stream pointer after they are done.

Bytes and Characters don't have terminators, of course.

Expressions have terminators that are newlines. The stream pointer is left sitting at the newline. An Expression like

1+2*
3/4-
5


has three newlines in it, at the end of each line. The newline after 5 is the terminator for this Expression, and after Read[stream, Expression] that character's position is the stream's position. StreamPosition[stream] == 11. If you followed Read[Expression] with Read[Character] you'd get a \n.

Strings also have newline terminators. But they CONSUME the newline, skipping over it, leaving the stream pointer after it. The newline character is not part of the String, but if you read a Character after reading a String you wouldn't get a \n, you'd get whatever is at the beginning of the next line.

Numbers, like Expressions, do not consume their terminating characters. They leave the stream pointer at that character, whatever it is. For instance, if you read a Number and then a Character from "64+32*3", the Number would be 64, and the Character would be "+". I think you can see why this is what you want.

Records and Words leave the stream pointer pointing at whatever character terminated them. This character is a RecordSeparator or WordSeparator; only Records or Words care about those options. However, if you then read another Record or Word subsequently, the stream will first SKIP OVER the RecordSeparator or WordSeparator that the stream is pointing to, the terminator for the previous Record or Word. Then it will proceed to read the next Record or Word. (Exception: this skipping does not happen if you're about to Read another Word and the separator was a TokenWord.)

If the input stream were

an,a,tev,ka
0123456789


and RecordSeparators->",", then reading one Record would give you "an", and the stream position would be 2. If it reads another Record then it will skip the comma, move to position 3, and then read "a". The stream position would be left at 4, the second comma.

In general this is what you want. You want to be able to read multiple Records without having the terminating characters interfere, but you also want to be able to grab those characters if desired. You might have RecordSeparators->{"+", "-", "*", "/"}, and you need to inspect the Character after the Record to find out which particular separator stopped it.

I hope this is an adequate explanation. I am not going to talk about Record and Word behavior when you have left-and-right matched delimiters as RecordSeparators or WordSeparators (like parentheses); nor NullRecords and NullWords; nor RecordLists; and there's one very useful special case where Numbers can consume RecordSeparator terminators. Please let me know if there's anything unclear and I'll hack on this response to make it unclearer.

-
Doing it Leonid style, eh? Looks like a great overview. +1 – Mr.Wizard Jan 16 '13 at 9:25
@Mr.Wizard Had exactly the same thoughts :) – Ajasja Jan 16 '13 at 9:50
The Bounty is for you. Enjoy. :-) – Mr.Wizard Mar 17 '13 at 12:10
librik, after rereading this fine answer I am left feeling slightly tantalized by your final paragraph. If you have time I'd love to have your analysis of the behavior with "left-and-right matched delimiters" description of the "one very useful special case where Numbers can consume RecordSeparator terminators." – Mr.Wizard Sep 15 '13 at 18:15

Seems not to be possible to mix Number and Expression and String. I would just do something like

ReadList[StringToStream[str], {Record, Record, Record, Record,
Record},
RecordSeparators -> {"\n", ";"}] /. {d_String, b__String} :>
Prepend[ToExpression[{b}], DateList[d]]


or

 ReadList[StringToStream[str], {Word, Word, Word, Word, Word},
RecordSeparators -> {"\n"},
WordSeparators -> {";"}] /. {d_String, b__String} :>
Prepend[ToExpression[{b}], DateList[d]]


If all your lines are exactly of the same structure then this should be as fast as possible (in pure Mathematica, of course you could use MathLink and do things in C):

    str = "Tue 1 Jan 2013 23 : 00 : 01; 17; {}; 32.5; 0.\nTue 2 Jan 2013 \
2 : 20 : 01; 47; {3,4}; 3.5; 110.";

o = StringToStream[str];
result = Reap[
Label[begin];
Skip[o, Character];
Skip[o, Character];
Read[o, Hold[Expression]] /. CompoundExpression -> Sequence}];
If[# === "\n", Goto[begin]] &@Read[o, Character]];
Close[o];
result // Last // First


This gives:

{{"Tue 1 Jan 2013 23 : 00 : 01", 17, {}, 32.5,
0.}, {"Tue 2 Jan 2013 2 : 20 : 01", 47, {3, 4}, 3.5, 110.}}


So, I have never used Label and Reap together, but this was a semi-quick solution. If anybody cannot see Label and Goto in a Mathematica program in 2013, then please: improve it.

-
Similar solution I have but the use of ToExpression I want to avoid as for a big file it makes the total reading pretty slow! See my updated question. – PlatoManiac Jan 15 '13 at 22:31
Rolf, you're correct, but because String tells Read to load in the rest of the line in its entirety. – rcollyer Jan 15 '13 at 23:23
@rcollyer next time you talk to someone powerful at Wolfram, please tell him or her to improve ReadList. It really should be made more versatile. – Rolf Mertig Jan 15 '13 at 23:47
@RolfMertig Thx for your help! Cant test your solution with my file right now in Germany. Will check it once I am at the office desk tomorrow. By the way how are you doing and is it also snowing in Berlin? – PlatoManiac Jan 15 '13 at 23:51
@rcollyer what are "druthers" ? – Rolf Mertig Jan 16 '13 at 0:05

I suspect you want something like this:

str = "Tue 1 Jan 2013 23 : 00 : 01; 17; {}; 32.5; 0.\nTue 2 Jan 2013 2 : 20 : 01; 47; \
{3,4}; 3.5; 110."

blank[] = blank[Character];
blank[_String] = Sequence[];

expr[] = expr[Word];
expr[x_String] := ToExpression[x]

{Word, blank[], Number, blank[], expr[], blank[], Number, blank[], Number},
WordSeparators -> {"\n", ";"}
]

{{"Tue 1 Jan 2013 23 : 00 : 01", 17, {}, 32.5, 0.},
{"Tue 2 Jan 2013 2 : 20 : 01", 47, {3, 4}, 3.5, 110.}}

-
but according to platomaniac ToExpression should be avoided for speed reason. – Rolf Mertig Jan 16 '13 at 9:32
@Rolf I don't see that in the question. In fact I don't see anything about speed in the question. – Mr.Wizard Jan 16 '13 at 9:34
it's in his first comment to my answer – Rolf Mertig Jan 16 '13 at 9:37
+1. Fantastic! Your solution is 12 times faster than mine. Should I just delete my answer? – Rolf Mertig Jan 16 '13 at 11:58
Yeah, this is how I would do it too. Know where each item type (Word, Number) leaves the stream position, and insert "dummy" Character-items in the ReadList spec to consume the terminators. You could write blank[_String] = Sequence[] and it might be faster. – librik Jan 16 '13 at 12:04

I'm not sure the problem here lies with the mixture of types for the objects to read, it seems to be related to the record separators.

str = "Tue 1 Jan 2013 23 : 00 : 01; 17; {}; 32.5; 0.\nTue 2 Jan 2013 \
2 : 20 : 01; 47; {3,4}; 3.5; 110.";


If the semicolon separators are replaced with linefeeds, everything works as expected.

ReadList[StringToStream[StringReplace[str, {";" -> "\n"}]],
{String, Number, Expression, Number, Number}]


{{"Tue 1 Jan 2013 23 : 00 : 01", 17, {}, 32.5, 0.}, {"Tue 2 Jan 2013 2 : 20 : 01", 47, {3, 4}, 3.5, 110.}}

So as you mention one solution would be to replace the separators in the input file. If you are on Linux or OS X something along the lines of sed -i 's/;/\n/g' myfile.dat will work. Taking a backup might be useful as this is an in place replacement.

-
+1; nice and clean. Though Mr.Wizards solution is still 20% faster (for 200000 lines) – Rolf Mertig Jan 16 '13 at 12:05
@RolfMertig Thank you for checking out the timing. – image_doctor Jan 16 '13 at 12:19