5
$\begingroup$

I have a huge string database with the form such as {{position},{1,2,3,4,5,..}}, and I want to convert such database to Interger data in a very quick way.

In the following example, I created a string database (instead of my real simulation database).

StringData = {};
steps=5000; (* for testing, in real simulation it is large*)
Do[AppendTo[StringData, {"Position"<>ToString[ii],"0,1,1,1,22,1,2,14,5,2,2,1,5,"}], {ii, 1, steps}];

strtolist = ConstantArray[{}, Length[StringData]];
For[ii = 1, ii <= Length[StringData], ii++,
   strtolist[[ii]] = ToExpression[StringSplit[StringData[[ii]][[2]], ","]];
   ]; // AbsoluteTiming

strtolist = ConstantArray[{}, Length[StringData]];
For[ii = 1, ii <= Length[StringData], ii++,
   strtolist[[ii]] = IntegerPart/@Internal`StringToDouble/@StringSplit[StringData[[ii]][[2]], ","];
   ]; // AbsoluteTiming

{0.248431, Null}

{0.100303, Null}

The second way is much fast. My real simulation is a large database and I wonder whether there are even quicker way to do such converting? for example without doing the outside for-loop? Thank you very much!

one additional problem using Internal`StringToDouble:

when the number is very large as the following example

test = {"0", "33837677493872221", "311462297063636041906"};
numstr1 = IntegerPart /@ Internal`StringToDouble /@ test
numstr2 = IntegerPart /@ Internal`StringToDouble /@ test[[3]]

the results:

{0, 33837677493872220, IntegerPart[$Failed["Bignum"]]}

311462297063636041906

Why numstr2 works good while numstr1 doesn't work? It seems Internal`StringToDouble works fine with single string not string lists?

What if the StringData contains number like "-1","-2" and so on? Only thinking about Integer number (including negative and positive). Is there any other way to do the same work instead of using ToExpression?

Improve this question
$\endgroup$
10
  • 1
    $\begingroup$ try Map[FromDigits, StringSplit[StringData[[All, 2]], ","], {-1}]? $\endgroup$
    – kglr
    Aug 30, 2019 at 20:42
  • $\begingroup$ Thank you very much! I will test your method. In addition I add one small question in the end. Seems Internal`StringToDouble does not work with with a lists containing large number but fine with one single string. Do you know why?@kglr $\endgroup$
    – Xuemei
    Aug 30, 2019 at 21:00
  • $\begingroup$ Map[FromDigits, StringSplit[StringData[[All, 2]], ","], {-1}] works good. Is it possible works with string list with negitave number such as "-1" and so on? @kglr $\endgroup$
    – Xuemei
    Aug 30, 2019 at 21:07
  • 2
    $\begingroup$ because Internal`StringToDouble does not have the Listable attribute. You can make listable version using istd = Internal`StringToDouble; SetAttributes[istd, Listable] $\endgroup$
    – kglr
    Aug 30, 2019 at 21:08
  • 1
    $\begingroup$ @kglr that's an interesting subtlety to Listable... I hadn't realized it would take precedence over evaluating to Internal`StringToDouble. I assumed the Head would evaluate, then any Attributes would apply but I suppose it's the other way around? $\endgroup$
    – b3m2a1
    Aug 30, 2019 at 21:17

2 Answers 2

Reset to default
4
$\begingroup$ 
strtolist2 = Map[FromDigits, StringSplit[StringData[[All, 2]], ","], {-1}]

strtolist3 = IntegerPart @ Map[Internal`StringToDouble, 
   StringSplit[StringData[[All, 2]], ","], {-1}];

strtolist3  == strtolist2 == strtolist

True

Both are about twice as fast as For loop with IntegerPart/@Internal`StringToDouble/@...

Improve this answer
$\endgroup$
3
  • $\begingroup$ Yes, it is fast but one problem is it cannot work with negative number such as “-1” because of FromDigits. Is there any chance also working with negative number string?@kglr $\endgroup$
    – Xuemei
    Aug 31, 2019 at 0:27
  • $\begingroup$ @XuemeiGu, please see the update. The second alternative can handle signed numbers. $\endgroup$
    – kglr
    Aug 31, 2019 at 0:43
  • $\begingroup$ Thank you very much. Using istd[x_] := InternalStringToDouble[x];` SetAttributes[istd, Listable];strtolist6 = IntegerPart@istd@StringSplit[StringData[[All, 2]], ","]; cannot solve large number strings such as str={{"Position1", "0,1,201111111111111111,1,-1"}, {"Position2", "0,1,201111111111111111,1,-1"}}. $\endgroup$
    – Xuemei
    Aug 31, 2019 at 18:31
6
$\begingroup$

ToExpression is very fast for correct Mathematica syntax input. So the key idea is to create an input string for ToExpression that delivers the expected result for the huge string database in one go:

StringData // 
   Extract[{All, 2}] // 
   StringRiffle[#, {"{{", "Nothing},{", "Nothing}}"}] & // 
   ToExpression

The odd looking "Nothing}" in StringRiffle is required to make the parser ignore the terminating comma (i.e., "0,1,1,1,22,1,2,14,5,2,2,1,5,") in each input string.

ToExpression also handles negative and very large numbers correctly.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Of course, one should always be careful with ToExpression[]. $\endgroup$
    – J. M.'s lack of A.I. ♦
    Sep 15, 2019 at 14:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.