# How to speed up interpreting numbers from strings?

In processing a large text file I have batches from ReadList of 100,000 Records of the form.

SeedRandom[123];
"1\t2023.203\t" <> #1 <> "\t4.932\t" <> #2 <> "\t" <> #3 <>"\t430.334" &,
{RandomChoice[{.95, .05} -> {"73.030", "4.3103e+008"}, 100000],
RandomChoice[{.95, .05} -> {"23.335", "-1.02847e+007"}, 100000],
RandomChoice[{.4, .4, .2} -> {"Cake", "Cookies", "Muffins"}, 100000]}];


There are numbers scattered throughout the file that are in a (scientific) format that ToExpression does not recognize. Such that when I process the batch like so

res = MapThread[#1[#2] &, {Insert[Identity, 6]@ConstantArray[ToExpression, 6], #}] & /@


There are entries that are not numbers that should be. (e.g. res[[2, 3]] gives 8 + 4.3103 e).

If Interpreter["Number"] is swapped for ToExpression then one batch takes far too long to complete considering that the file has 400+ batches to process.

MapThread[#1[#2] &, {Insert[Identity, 6]@ConstantArray[Interpreter["Number"], 6], #}] & /@


The unrecognized number strings can be in any of the number locations. How can I either speed up Interpreter["Number"] or apply some other technique to process the numbers in the batch.

I am placing the results each batch into a database ("HSQL(Standalone)") using SQLInsert and am open to any shortcuts that could take advantage of this.

• Try using InternalStringToDouble in place of ToExpression. – Leonid Shifrin Jun 14 '16 at 16:30
• @Leonid The problem with that function is that it won't tell you if the conversion fails. – Szabolcs Jun 14 '16 at 16:38
• @Szabolcs Yes, that sucks. I wasn't aware of that. – Leonid Shifrin Jun 14 '16 at 16:39
• Import and ImportString handle the e number format okay. res2 = fromReadList~StringRiffle~"\n"~ImportString~"TSV"~Cases~{__,"Cake"|"Cookies",_}; – Simon Woods Jun 14 '16 at 18:05
• Have you tried either of the other two methods I posted in (1737)? Particularly SystemConvertTableDumpParseTable seemed quite fast the last time I had need of it. – Mr.Wizard Jun 15 '16 at 0:12

Import and ImportString handle the e number format okay. You might be able to Import directly from file, or use ImportString to process the data you've already read in:

res = fromReadList ~StringRiffle~ "\n" ~ImportString~ "TSV" ~Cases~ {__,"Cake"|"Cookies",_};

• (+1) Its also faster than my initial code. – Edmund Jun 14 '16 at 21:05
• To those who found the infix difficult to decipher: Cases[ImportString[StringRiffle[fromReadList, "\n"], "TSV"], {__, "Cake" | "Cookies", _}] – Henrik Hansen Jun 18 '16 at 13:58

I would like to link this W Community thread here where I asked for functionality like this in 2015 September, and explained why it's critical to have it. I can't link to individual posts but you can find it by searching the page for "StringToDouble".

As Leonid mentioned, there is InternalStringToDouble. This function is very fast, but it does not report errors. This makes it unsuitable for applications where not all inputs are numbers, especially when the type of the input is unpredictable.

InternalStringToDouble["1e2"]
(* 100. *)

InternalStringToDouble["foo"] (* not a number *)
(* 0. *)


As a workaround we can make a small LibraryLink function that parses numbers in this format. Fortunately it's very easy to do in C++.

I am going to use LTemplate for reasons of laziness (as I always do recently). LTemplate is absolutely not needed here, it just makes it quicker for me to set everything up.

First, put this C++ code in Parser.h:

#include <sstream>

class Parser {
bool good;

public:
double parseReal(const char *s) {
std::istringstream str(s);
mma::disownString(s);
double res;
str >> res;
if (str.fail() || ! str.eof()) {
good = false;
return 0;
}
good = true;
return res;
}

mint parseInteger(const char *s) {
std::istringstream str(s);
mma::disownString(s);
mint res;
str >> res;
if (str.fail() || ! str.eof()) {
good = false;
return 0;
}
good = true;
return res;
}

bool success() const { return good; }
};


Then from Mathematica, make sure that Directory[] is where Parser.h is and evaluate:

<<LTemplate

template = LClass[
"Parser",
{
LFun["parseReal", {"UTF8String"}, Real],
LFun["parseInteger", {"UTF8String"}, Integer],
LFun["success", {}, True | False]
}
];

CompileTemplate[template]

The parseReal method parses a real number in e-notation. The success method tells us if the parsing was successful. Then we can build on top of this.
The only thing this provides over InternalStringToDouble` is a way to check for errors.