I have a huge string database with the form such as {{position},{1,2,3,4,5,..}}, and I want to convert such database to Interger data in a very quick way.

In the following example, I created a string database (instead of my real simulation database).

StringData = {};
steps=5000; (* for testing, in real simulation it is large*)
Do[AppendTo[StringData, {"Position"<>ToString[ii],"0,1,1,1,22,1,2,14,5,2,2,1,5,"}], {ii, 1, steps}];

strtolist = ConstantArray[{}, Length[StringData]];
For[ii = 1, ii <= Length[StringData], ii++,
   strtolist[[ii]] = ToExpression[StringSplit[StringData[[ii]][[2]], ","]];
   ]; // AbsoluteTiming

strtolist = ConstantArray[{}, Length[StringData]];
For[ii = 1, ii <= Length[StringData], ii++,
   strtolist[[ii]] = IntegerPart/@Internal`StringToDouble/@StringSplit[StringData[[ii]][[2]], ","];
   ]; // AbsoluteTiming  

{0.248431, Null}

{0.100303, Null}

The second way is much fast. My real simulation is a large database and I wonder whether there are even quicker way to do such converting? for example without doing the outside for-loop? Thank you very much!

one additional problem using Internal`StringToDouble:

when the number is very large as the following example

test = {"0", "33837677493872221", "311462297063636041906"};
numstr1 = IntegerPart /@ Internal`StringToDouble /@ test
numstr2 = IntegerPart /@ Internal`StringToDouble /@ test[[3]]

the results:

{0, 33837677493872220, IntegerPart[$Failed["Bignum"]]}


Why numstr2 works good while numstr1 doesn't work? It seems Internal`StringToDouble works fine with single string not string lists?

What if the StringData contains number like "-1","-2" and so on? Only thinking about Integer number (including negative and positive). Is there any other way to do the same work instead of using ToExpression?

  • 1
    $\begingroup$ try Map[FromDigits, StringSplit[StringData[[All, 2]], ","], {-1}]? $\endgroup$
    – kglr
    Aug 30 '19 at 20:42
  • $\begingroup$ Thank you very much! I will test your method. In addition I add one small question in the end. Seems Internal`StringToDouble does not work with with a lists containing large number but fine with one single string. Do you know why?@kglr $\endgroup$
    – Xuemei
    Aug 30 '19 at 21:00
  • $\begingroup$ Map[FromDigits, StringSplit[StringData[[All, 2]], ","], {-1}] works good. Is it possible works with string list with negitave number such as "-1" and so on? @kglr $\endgroup$
    – Xuemei
    Aug 30 '19 at 21:07
  • 2
    $\begingroup$ because Internal`StringToDouble does not have the Listable attribute. You can make listable version using istd = Internal`StringToDouble; SetAttributes[istd, Listable] $\endgroup$
    – kglr
    Aug 30 '19 at 21:08
  • 1
    $\begingroup$ @kglr that's an interesting subtlety to Listable... I hadn't realized it would take precedence over evaluating to Internal`StringToDouble. I assumed the Head would evaluate, then any Attributes would apply but I suppose it's the other way around? $\endgroup$
    – b3m2a1
    Aug 30 '19 at 21:17
strtolist2 = Map[FromDigits, StringSplit[StringData[[All, 2]], ","], {-1}]

strtolist3 = IntegerPart @ Map[Internal`StringToDouble, 
   StringSplit[StringData[[All, 2]], ","], {-1}];

strtolist3  == strtolist2 == strtolist


Both are about twice as fast as For loop with IntegerPart/@Internal`StringToDouble/@...

  • $\begingroup$ Yes, it is fast but one problem is it cannot work with negative number such as “-1” because of FromDigits. Is there any chance also working with negative number string?@kglr $\endgroup$
    – Xuemei
    Aug 31 '19 at 0:27
  • $\begingroup$ @XuemeiGu, please see the update. The second alternative can handle signed numbers. $\endgroup$
    – kglr
    Aug 31 '19 at 0:43
  • $\begingroup$ Thank you very much. Using istd[x_] := InternalStringToDouble[x];` SetAttributes[istd, Listable];strtolist6 = IntegerPart@istd@StringSplit[StringData[[All, 2]], ","]; cannot solve large number strings such as str={{"Position1", "0,1,201111111111111111,1,-1"}, {"Position2", "0,1,201111111111111111,1,-1"}}. $\endgroup$
    – Xuemei
    Aug 31 '19 at 18:31

ToExpression is very fast for correct Mathematica syntax input. So the key idea is to create an input string for ToExpression that delivers the expected result for the huge string database in one go:

StringData // 
   Extract[{All, 2}] // 
   StringRiffle[#, {"{{", "Nothing},{", "Nothing}}"}] & // 

The odd looking "Nothing}" in StringRiffle is required to make the parser ignore the terminating comma (i.e., "0,1,1,1,22,1,2,14,5,2,2,1,5,") in each input string.

ToExpression also handles negative and very large numbers correctly.

  • $\begingroup$ Of course, one should always be careful with ToExpression[]. $\endgroup$
    – J. M.'s torpor
    Sep 15 '19 at 14:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.