5
$\begingroup$

I have a huge string database with the form such as {{position},{1,2,3,4,5,..}}, and I want to convert such database to Interger data in a very quick way.

In the following example, I created a string database (instead of my real simulation database).

StringData = {};
steps=5000; (* for testing, in real simulation it is large*)
Do[AppendTo[StringData, {"Position"<>ToString[ii],"0,1,1,1,22,1,2,14,5,2,2,1,5,"}], {ii, 1, steps}];

strtolist = ConstantArray[{}, Length[StringData]];
For[ii = 1, ii <= Length[StringData], ii++,
   strtolist[[ii]] = ToExpression[StringSplit[StringData[[ii]][[2]], ","]];
   ]; // AbsoluteTiming

strtolist = ConstantArray[{}, Length[StringData]];
For[ii = 1, ii <= Length[StringData], ii++,
   strtolist[[ii]] = IntegerPart/@Internal`StringToDouble/@StringSplit[StringData[[ii]][[2]], ","];
   ]; // AbsoluteTiming  

{0.248431, Null}

{0.100303, Null}

The second way is much fast. My real simulation is a large database and I wonder whether there are even quicker way to do such converting? for example without doing the outside for-loop? Thank you very much!

one additional problem using Internal`StringToDouble:

when the number is very large as the following example

test = {"0", "33837677493872221", "311462297063636041906"};
numstr1 = IntegerPart /@ Internal`StringToDouble /@ test
numstr2 = IntegerPart /@ Internal`StringToDouble /@ test[[3]]

the results:

{0, 33837677493872220, IntegerPart[$Failed["Bignum"]]}

311462297063636041906

Why numstr2 works good while numstr1 doesn't work? It seems Internal`StringToDouble works fine with single string not string lists?

What if the StringData contains number like "-1","-2" and so on? Only thinking about Integer number (including negative and positive). Is there any other way to do the same work instead of using ToExpression?

$\endgroup$
  • 1
    $\begingroup$ try Map[FromDigits, StringSplit[StringData[[All, 2]], ","], {-1}]? $\endgroup$ – kglr Aug 30 at 20:42
  • $\begingroup$ Thank you very much! I will test your method. In addition I add one small question in the end. Seems Internal`StringToDouble does not work with with a lists containing large number but fine with one single string. Do you know why?@kglr $\endgroup$ – Xuemei Gu Aug 30 at 21:00
  • $\begingroup$ Map[FromDigits, StringSplit[StringData[[All, 2]], ","], {-1}] works good. Is it possible works with string list with negitave number such as "-1" and so on? @kglr $\endgroup$ – Xuemei Gu Aug 30 at 21:07
  • 2
    $\begingroup$ because Internal`StringToDouble does not have the Listable attribute. You can make listable version using istd = Internal`StringToDouble; SetAttributes[istd, Listable] $\endgroup$ – kglr Aug 30 at 21:08
  • 1
    $\begingroup$ @kglr that's an interesting subtlety to Listable... I hadn't realized it would take precedence over evaluating to Internal`StringToDouble. I assumed the Head would evaluate, then any Attributes would apply but I suppose it's the other way around? $\endgroup$ – b3m2a1 Aug 30 at 21:17
5
$\begingroup$

ToExpression is very fast for correct Mathematica syntax input. So the key idea is to create an input string for ToExpression that delivers the expected result for the huge string database in one go:

StringData // 
   Extract[{All, 2}] // 
   StringRiffle[#, {"{{", "Nothing},{", "Nothing}}"}] & // 
   ToExpression

The odd looking "Nothing}" in StringRiffle is required to make the parser ignore the terminating comma (i.e., "0,1,1,1,22,1,2,14,5,2,2,1,5,") in each input string.

ToExpression also handles negative and very large numbers correctly.

$\endgroup$
  • $\begingroup$ Of course, one should always be careful with ToExpression[]. $\endgroup$ – J. M. will be back soon Sep 15 at 14:06
4
$\begingroup$
strtolist2 = Map[FromDigits, StringSplit[StringData[[All, 2]], ","], {-1}]

strtolist3 = IntegerPart @ Map[Internal`StringToDouble, 
   StringSplit[StringData[[All, 2]], ","], {-1}];

strtolist3  == strtolist2 == strtolist

True

Both are about twice as fast as For loop with IntegerPart/@Internal`StringToDouble/@...

$\endgroup$
  • $\begingroup$ Yes, it is fast but one problem is it cannot work with negative number such as “-1” because of FromDigits. Is there any chance also working with negative number string?@kglr $\endgroup$ – Xuemei Gu Aug 31 at 0:27
  • $\begingroup$ @XuemeiGu, please see the update. The second alternative can handle signed numbers. $\endgroup$ – kglr Aug 31 at 0:43
  • $\begingroup$ Thank you very much. Using istd[x_] := InternalStringToDouble[x];` SetAttributes[istd, Listable];strtolist6 = IntegerPart@istd@StringSplit[StringData[[All, 2]], ","]; cannot solve large number strings such as str={{"Position1", "0,1,201111111111111111,1,-1"}, {"Position2", "0,1,201111111111111111,1,-1"}}. $\endgroup$ – Xuemei Gu Aug 31 at 18:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.