# Convert String to Integer from a huge string database in a very fast way?

I have a huge string database with the form such as {{position},{1,2,3,4,5,..}}, and I want to convert such database to Interger data in a very quick way.

In the following example, I created a string database (instead of my real simulation database).

StringData = {};
steps=5000; (* for testing, in real simulation it is large*)
Do[AppendTo[StringData, {"Position"<>ToString[ii],"0,1,1,1,22,1,2,14,5,2,2,1,5,"}], {ii, 1, steps}];

strtolist = ConstantArray[{}, Length[StringData]];
For[ii = 1, ii <= Length[StringData], ii++,
strtolist[[ii]] = ToExpression[StringSplit[StringData[[ii]][[2]], ","]];
]; // AbsoluteTiming

strtolist = ConstantArray[{}, Length[StringData]];
For[ii = 1, ii <= Length[StringData], ii++,
strtolist[[ii]] = IntegerPart/@InternalStringToDouble/@StringSplit[StringData[[ii]][[2]], ","];
]; // AbsoluteTiming


{0.248431, Null}

{0.100303, Null}

The second way is much fast. My real simulation is a large database and I wonder whether there are even quicker way to do such converting? for example without doing the outside for-loop? Thank you very much!

one additional problem using InternalStringToDouble:

when the number is very large as the following example

test = {"0", "33837677493872221", "311462297063636041906"};
numstr1 = IntegerPart /@ InternalStringToDouble /@ test
numstr2 = IntegerPart /@ InternalStringToDouble /@ test[[3]]


the results:

{0, 33837677493872220, IntegerPart[\$Failed["Bignum"]]}

311462297063636041906

Why numstr2 works good while numstr1 doesn't work? It seems InternalStringToDouble works fine with single string not string lists?

What if the StringData contains number like "-1","-2" and so on? Only thinking about Integer number (including negative and positive). Is there any other way to do the same work instead of using ToExpression?

• try Map[FromDigits, StringSplit[StringData[[All, 2]], ","], {-1}]?
– kglr
Aug 30, 2019 at 20:42
• Thank you very much! I will test your method. In addition I add one small question in the end. Seems InternalStringToDouble does not work with with a lists containing large number but fine with one single string. Do you know why?@kglr Aug 30, 2019 at 21:00
• Map[FromDigits, StringSplit[StringData[[All, 2]], ","], {-1}] works good. Is it possible works with string list with negitave number such as "-1" and so on? @kglr Aug 30, 2019 at 21:07
• because InternalStringToDouble does not have the Listable attribute. You can make listable version using istd = InternalStringToDouble; SetAttributes[istd, Listable]
– kglr
Aug 30, 2019 at 21:08
• @kglr that's an interesting subtlety to Listable... I hadn't realized it would take precedence over evaluating to InternalStringToDouble. I assumed the Head would evaluate, then any Attributes would apply but I suppose it's the other way around? Aug 30, 2019 at 21:17

strtolist2 = Map[FromDigits, StringSplit[StringData[[All, 2]], ","], {-1}]

strtolist3 = IntegerPart @ Map[InternalStringToDouble,
StringSplit[StringData[[All, 2]], ","], {-1}];

strtolist3  == strtolist2 == strtolist


True

Both are about twice as fast as For loop with IntegerPart/@InternalStringToDouble/@...

• Yes, it is fast but one problem is it cannot work with negative number such as “-1” because of FromDigits. Is there any chance also working with negative number string?@kglr Aug 31, 2019 at 0:27
• @XuemeiGu, please see the update. The second alternative can handle signed numbers.
– kglr
Aug 31, 2019 at 0:43
• Thank you very much. Using istd[x_] := InternalStringToDouble[x]; SetAttributes[istd, Listable];strtolist6 = IntegerPart@istd@StringSplit[StringData[[All, 2]], ","];  cannot solve large number strings such as str={{"Position1", "0,1,201111111111111111,1,-1"}, {"Position2", "0,1,201111111111111111,1,-1"}}. Aug 31, 2019 at 18:31

ToExpression is very fast for correct Mathematica syntax input. So the key idea is to create an input string for ToExpression that delivers the expected result for the huge string database in one go:

StringData //
Extract[{All, 2}] //
StringRiffle[#, {"{{", "Nothing},{", "Nothing}}"}] & //
ToExpression


The odd looking "Nothing}" in StringRiffle is required to make the parser ignore the terminating comma (i.e., "0,1,1,1,22,1,2,14,5,2,2,1,5,") in each input string.

ToExpression also handles negative and very large numbers correctly.

• Of course, one should always be careful with ToExpression[]. Sep 15, 2019 at 14:06