7
$\begingroup$

In processing a large text file I have batches from ReadList of 100,000 Records of the form.

SeedRandom[123];
fromReadList = 
  MapThread[
   "1\t2023.203\t" <> #1 <> "\t4.932\t" <> #2 <> "\t" <> #3 <>"\t430.334" &,
   {RandomChoice[{.95, .05} -> {"73.030", "4.3103e+008"}, 100000],
    RandomChoice[{.95, .05} -> {"23.335", "-1.02847e+007"}, 100000],
    RandomChoice[{.4, .4, .2} -> {"Cake", "Cookies", "Muffins"}, 100000]}];

There are numbers scattered throughout the file that are in a (scientific) format that ToExpression does not recognize. Such that when I process the batch like so

res = MapThread[#1[#2] &, {Insert[Identity, 6]@ConstantArray[ToExpression, 6], #}] & /@ 
 Select[ContainsAny[{#[[6]]}, {"Cake", "Cookies"}] &]@
  StringSplit[fromReadList]

There are entries that are not numbers that should be. (e.g. res[[2, 3]] gives 8 + 4.3103 e).

If Interpreter["Number"] is swapped for ToExpression then one batch takes far too long to complete considering that the file has 400+ batches to process.

MapThread[#1[#2] &, {Insert[Identity, 6]@ConstantArray[Interpreter["Number"], 6], #}] & /@ 
 Select[ContainsAny[{#[[6]]}, {"Cake", "Cookies"}] &]@
  StringSplit[fromReadList]

The unrecognized number strings can be in any of the number locations. How can I either speed up Interpreter["Number"] or apply some other technique to process the numbers in the batch.

I am placing the results each batch into a database ("HSQL(Standalone)") using SQLInsert and am open to any shortcuts that could take advantage of this.

$\endgroup$
  • 2
    $\begingroup$ Try using Internal`StringToDouble in place of ToExpression. $\endgroup$ – Leonid Shifrin Jun 14 '16 at 16:30
  • 1
    $\begingroup$ @Leonid The problem with that function is that it won't tell you if the conversion fails. $\endgroup$ – Szabolcs Jun 14 '16 at 16:38
  • 1
    $\begingroup$ @Szabolcs Yes, that sucks. I wasn't aware of that. $\endgroup$ – Leonid Shifrin Jun 14 '16 at 16:39
  • 1
    $\begingroup$ Import and ImportString handle the e number format okay. res2 = fromReadList~StringRiffle~"\n"~ImportString~"TSV"~Cases~{__,"Cake"|"Cookies",_}; $\endgroup$ – Simon Woods Jun 14 '16 at 18:05
  • 1
    $\begingroup$ Have you tried either of the other two methods I posted in (1737)? Particularly System`Convert`TableDump`ParseTable seemed quite fast the last time I had need of it. $\endgroup$ – Mr.Wizard Jun 15 '16 at 0:12
4
$\begingroup$

Import and ImportString handle the e number format okay. You might be able to Import directly from file, or use ImportString to process the data you've already read in:

res = fromReadList ~StringRiffle~ "\n" ~ImportString~ "TSV" ~Cases~ {__,"Cake"|"Cookies",_};
$\endgroup$
  • $\begingroup$ (+1) Its also faster than my initial code. $\endgroup$ – Edmund Jun 14 '16 at 21:05
  • $\begingroup$ To those who found the infix difficult to decipher: Cases[ImportString[StringRiffle[fromReadList, "\n"], "TSV"], {__, "Cake" | "Cookies", _}] $\endgroup$ – Henrik Hansen Jun 18 '16 at 13:58
13
$\begingroup$

I would like to link this W Community thread here where I asked for functionality like this in 2015 September, and explained why it's critical to have it. I can't link to individual posts but you can find it by searching the page for "StringToDouble".


As Leonid mentioned, there is Internal`StringToDouble. This function is very fast, but it does not report errors. This makes it unsuitable for applications where not all inputs are numbers, especially when the type of the input is unpredictable.

Internal`StringToDouble["1e2"]
(* 100. *)

Internal`StringToDouble["foo"] (* not a number *)
(* 0. *)

As a workaround we can make a small LibraryLink function that parses numbers in this format. Fortunately it's very easy to do in C++.

I am going to use LTemplate for reasons of laziness (as I always do recently). LTemplate is absolutely not needed here, it just makes it quicker for me to set everything up.

First, put this C++ code in Parser.h:

#include <sstream>

class Parser {
    bool good;

public:
    double parseReal(const char *s) {
        std::istringstream str(s);
        mma::disownString(s);
        double res;
        str >> res;
        if (str.fail() || ! str.eof()) {
            good = false;
            return 0;
        }
        good = true;
        return res;
    }

    mint parseInteger(const char *s) {
        std::istringstream str(s);
        mma::disownString(s);
        mint res;
        str >> res;
        if (str.fail() || ! str.eof()) {
            good = false;
            return 0;
        }
        good = true;
        return res;
    }

    bool success() const { return good; }
};

Then from Mathematica, make sure that Directory[] is where Parser.h is and evaluate:

<<LTemplate`

template = LClass[
  "Parser",
  {
    LFun["parseReal", {"UTF8String"}, Real],
    LFun["parseInteger", {"UTF8String"}, Integer],
    LFun["success", {}, True | False]
  }
];

CompileTemplate[template]
LoadTemplate[template]

Then use like this:

enter image description here

The parseReal method parses a real number in e-notation. The success method tells us if the parsing was successful. Then we can build on top of this.

I used this for personal projects. You can get the code here, but LTemplate must be installed first, and you must also have a working C++ compiler installed... Remember that I made this for personal use. It's a really basic package and you will probably be better off writing your own, tailed to your own needs.

The only thing this provides over Internal`StringToDouble is a way to check for errors.

$\endgroup$
  • $\begingroup$ I didn't realize that it doesn't throw a message on fail and that it needed the extra bits. Not certain it will work for deploying to Player Pro. Thanks for sharing. (+1) $\endgroup$ – Edmund Jun 14 '16 at 17:08
  • 2
    $\begingroup$ @Edmund I don't have Player Pro, but you're right ... it might not work ... it would also make deployment more complicated because you would need to ship a package which has binaries for all platforms: that means one binary for Windows, one carefully compiled one for Linux to make sure it works on many distros, and two for OS X (for 10.0-10.3 and a different one for 10.4) ... $\endgroup$ – Szabolcs Jun 14 '16 at 17:20
  • 1
    $\begingroup$ Amazing work, this and the LTemplate package. Thank you very much for documenting and sharing. $\endgroup$ – Theo Tiger Oct 8 '18 at 8:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.