# How to speed up loading 20k lines tsv-file with complex numbers?

I have a long (20k lines) file which looks like this:

[...]
1.96    -0.03   (433.489,0) (-13408.6,212.4)    (0,421.9)   (433.489,0)
1.96    -0.02   (163.961,0) (-13423.2,5212.2)   (13423.2,0) (163.961,0)
1.96    -0.01   (-105.342,0)    (-13436.4,0)    (13436.4,0) (-105.342,-0.4124)
1.96    0   (-374.421,0)    (-13448.3,0)    (13448.3,0) (-374.421,42.12)
1.96    0.01    (-643.276,0)    (-13458.9,0)    (0, -129.479)   (-643.276,0)
1.96    0.02    (-911.905,-0.421)   (-13468.1,9912.23)  (13468.1,0) (-911.905,0)
[...]


The first two numbers are floating point numbers, the rest of them are tuples, where the first number is the real part and the second number the imaginary one.

I have put together the following solution:

convert2complex[lst_, index_] :=
Flatten@{lst[[1 ;; index]],
Complex @@
Interpreter["Number"][StringSplit[#, "(" | ")" | ","]] & /@
lst[[index + 1 ;;]]}

amplitude["D1232_scattering"] =
convert2complex[#, 2] & /@
Drop[Import["data/amplitude_D1232_scattering.tsv"], 1];


The index parameter basically tells the function where the strings begin (since the first two numbers need not be transformed).

I'd like to speed this up. I can change the format of the input file, if there is a better input format.

Here is one way to tackle your problem:

1. Import the file as a string
rawData = Import["C:\\data.tsv", "String"];

1. Remove (,) and replace , with \t and continuous spaces with \t (used FixedPoint to make sure nothing is left)
data = Rest@
ImportString[
FixedPoint[
StringReplace[#, {"," -> "\t", "(" -> "", ")" -> "",
RegularExpression@"\)? +\(?" -> "\t",
RegularExpression@"\t+" -> "\t"}] &, rawData], "TSV"];

1. Make the specified parts Imaginary and build the output
temp = {1, 1, 1, I, 1, I, 1, I, 1, I} * Transpose[data];

result = Transpose[Join[temp[[;; 2]], temp[[3 ;; ;; 2]] + temp[[4 ;; ;; 2]]]]


Check the result:

Dimensions[result]

(* Out: {6, 6} *)

result[[1, 4]]

(* Out: -13408.6 + 212.4 I *)

• Since I can control the tsv format, I guess it'd be even faster to do the transformations in the C++ code that generates it and only start from step 3, right? Commented Jun 24, 2022 at 9:37
• @infinitezero Yes, that would be faster (you'll still need the Rest @ ImportString[ ... , "TSV"] from step 2). Commented Jun 24, 2022 at 9:42
• @infinitezero If you have control over the output, you could already output in Mathematica format. Commented Jun 24, 2022 at 10:38
• I used this solution ( and customized it a bit to account for various key lengths in the beginning), but this gave a whopping speedup of factor 250 (my version took 200s, this took 0.8 seconds). Commented Jun 24, 2022 at 10:57
• @BenIzd or directly Get. Commented Jun 24, 2022 at 12:12

Here I offer two solutions, Sol 1 is twice as fast as the one by @BenIzd, and Sol 2 is twice as slow.

# Sol 1

This one is very fast

With[
{
path = "data/amplitude_D1232_scattering.tsv",
tf = Join[#[[1;;2]], Apply[Complex,Partition[#[[3;;-1]],2],{1}] ]&
},
tf/@Import[
File[path]
,"Table"
,"FieldSeparators"->{","," ","∖t","(",")"}
]
]


# Sol 2

Alternatively, a similar by much slower solution

With[
{
tf = BlockMap[Apply[Complex], Insert[#, 0, {{2},{3}}], 2]&
},
tf/@Import[
File[path]
,"Table"
,"FieldSeparators"->{","," ","∖t","(",")"}
]
]

• @infinitezero I find that Sol 1 is significantly faster than the solution by Ben Izd. Commented Jun 24, 2022 at 13:42