2
$\begingroup$

I have a data output from a device in a form of a string. I need to transform it into such a form that I can work with it, such as plot columns with respect to one another and such. After a apply Characters to it it takes the form:

   lst={" ", " ", " ", "1", "6", ".", "4", "1", "0", "7", " ", " ", " ", " \
", " ", " ", " ", " ", " ", " ", " ", "4", ".", "4", "3", "1", "5", \
"e", "-", "0", "1", " ", " ", " ", " ", " ", " ", " ", "5", ".", "4", \
"0", "0", "7", "e", "+", "0", "2", " ", " ", " ", " ", " ", " ", "1", \
".", "9", "8", "3", "8", "e", "-", "0", "1", " ", " ", " ", " ", " ", \
" ", " ", "2", ".", "4", "1", "7", "7", "e", "+", "0", "2", " ", " ", \
" ", " ", " ", " ", " ", "7", ".", "4", "8", "4", "5", "e", "+", "0", \
"0", " ", " ", " ", " ", " ", " ", " ", "9", ".", "1", "2", "1", "5", \
"e", "+", "0", "3", "\\n", " ", " ", " ", " ", " ", " ", "1", "8", \
".", "6", "8", "2", "5", " ", " ", " ", " ", " ", " ", " ", " ", " ", \
" ", " ", "1", ".", "2", "2", "0", "1", "e", "+", "0", "0", " ", " ", \
" ", " ", " ", " ", " ", "1", ".", "3", "7", "1", "8", "e", "+", "0", \
"3", " ", " ", " ", " ", " ", " ", "3", ".", "3", "6", "3", "5", "e", \
"-", "0", "1", " ", " ", " ", " ", " ", " ", " ", "3", ".", "6", "0", \
"0", "7", "e", "+", "0", "2", " ", " ", " ", " ", " ", " ", " ", "1", \
".", "4", "4", "5", "1", "e", "+", "0", "1", " ", " ", " ", " ", " ", \
" ", " ", "1", ".", "5", "4", "7", "0", "e", "+", "0", "4", "\\n", " \
", " ", " ", " ", " ", " "}

This is a small part of the data. StringJoinyields 7 columns, which is how the data is organized

enter image description here

So, any idea of how to transform it into a normal list?

Later edit: to address the question of bbgodfrey: this is a part of the original output. I add it as an image:

enter image description here

After I transformed in into characters and cut away first 1360 ones containing the useless billhead, and then combine them back to the string we see 7 columns with numbers that I already posted above.

$\endgroup$
  • $\begingroup$ As a rough start, try something like SemanticImportString[StringJoin@StringDelete[lst, "\\n"]]. This will give you a Dataset from which you could get your data, and possibly Partition the result into a usable table. $\endgroup$ – MarcoB Oct 26 '15 at 18:52
  • $\begingroup$ You mean you want to convert the whole lot into the real-valued list {16.4107, 4.4315e-01, …}? $\endgroup$ – Patrick Stevens Oct 26 '15 at 18:52
  • $\begingroup$ What does the original output look like? If one giant string, can you post a part of it? $\endgroup$ – bbgodfrey Oct 26 '15 at 19:05
  • 1
    $\begingroup$ Looks like the kind of job that fits like a glove to Awk, Perl or similars $\endgroup$ – Dr. belisarius Oct 26 '15 at 19:25
  • 1
    $\begingroup$ @AlexeiBoulbitch If you're going to work a lot with semi-formatted text files you should try them. Awk is easier, Perl is more modern and powerful. Google them :) $\endgroup$ – Dr. belisarius Oct 26 '15 at 20:01
2
$\begingroup$

Mathematica has a very rich set of methods to deal with strings.

lst = {" ", " ", " ", "1", "6", ".", "4", "1", "0", "7", " ", " ", 
  " ", " ", " ", " ", " ", " ", " ", " ", " ", "4", ".", "4", "3", 
  "1", "5", "e", "-", "0", "1", " ", " ", " ", " ", " ", " ", " ", 
  "5", ".", "4", "0", "0", "7", "e", "+", "0", "2", " ", " ", " ", 
  " ", " ", " ", "1", ".", "9", "8", "3", "8", "e", "-", "0", "1", 
  " ", " ", " ", " ", " ", " ", " ", "2", ".", "4", "1", "7", "7", 
  "e", "+", "0", "2", " ", " ", " ", " ", " ", " ", " ", "7", ".", 
  "4", "8", "4", "5", "e", "+", "0", "0", " ", " ", " ", " ", " ", 
  " ", " ", "9", ".", "1", "2", "1", "5", "e", "+", "0", "3", "\\n", 
  " ", " ", " ", " ", " ", " ", "1", "8", ".", "6", "8", "2", "5", 
  " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", " ", "1", ".", 
  "2", "2", "0", "1", "e", "+", "0", "0", " ", " ", " ", " ", " ", 
  " ", " ", "1", ".", "3", "7", "1", "8", "e", "+", "0", "3", " ", 
  " ", " ", " ", " ", " ", "3", ".", "3", "6", "3", "5", "e", "-", 
  "0", "1", " ", " ", " ", " ", " ", " ", " ", "3", ".", "6", "0", 
  "0", "7", "e", "+", "0", "2", " ", " ", " ", " ", " ", " ", " ", 
  "1", ".", "4", "4", "5", "1", "e", "+", "0", "1", " ", " ", " ", 
  " ", " ", " ", " ", "1", ".", "5", "4", "7", "0", "e", "+", "0", 
  "4", "\\n", " ", " ", " ", " ", " ", " "}

For this answer it will be easier if the string is not broken into characters.

text = StringJoin[lst];

Step 1

Replace the letter "e" with the string "*10^".

text1 = StringReplace[text, "e" -> "*10^"]

Step 2

Break the complete string into a list of lines.

lines = StringSplit[text1, "\\n"]

For your actual data you would remove the first few lines until you get down to the column of numbers.

Step 3

This is the workhorse portion.

We will take the contiguous characters between the spaces and convert them to a number.

data = Map[
  StringCases[#, 
    WordBoundary ~~ x : Characters["01234567890-+.*^"] .. ~~ 
      WordBoundary :> ToExpression[x]] &,
  lines
  ]

The output is

{{16.4107, 0.44315, 540.07, 0.19838, 241.77, 7.4845, 
  9121.5}, {18.6825, 1.2201, 1371.8, 0.33635, 360.07, 14.451, 
  15470.}, {}}

Step 4

Observe that there is an empty list. The last step is to remove all empty lists.

data = DeleteCases[data, {}]

yielding the final result

{{16.4107, 0.44315, 540.07, 0.19838, 241.77, 7.4845, 
  9121.5}, {18.6825, 1.2201, 1371.8, 0.33635, 360.07, 14.451, 15470.}}
$\endgroup$
  • $\begingroup$ Jack, this looks very promising, but by application to my txt file I get unexpected results on the step 3. The warning: ToExpression::sntx: Invalid syntax in or before "*10^". and instead of the removing the whole "cap" each word is replaced by $Failed. Besides, the figures are not splitted into sublists, as expected. The most simple would be to show you the initial txt file, but I do not know, how to do it. $\endgroup$ – Alexei Boulbitch Oct 27 '15 at 12:57
  • $\begingroup$ @AlexeiBoulbitch I looked at the text from your answer. For that text in Step 2 I had to use: lines = StringSplit[text1, EndOfLine]. The rest of it, Step 3 included worked fine. Make sure if you go to your most raw text you removed the lines that precede the column of numbers. $\endgroup$ – Jack LaVigne Oct 27 '15 at 23:11
1
$\begingroup$

I think I understand now, how to solve the question partially based on the @ MarcoB advice. Observing that the billhead ends up with "cc/g\n" let us first delete the billhead assuming that text is the freshly imported string:

text1 = StringDelete[text, __ ~~ "cc/g\n"]

This yields the following

     16.4540           4.0763e-02       4.9548e+01      1.9462e-02       \
2.3656e+01       7.3634e-01       8.9503e+02
      18.7036           1.1419e-01       1.2807e+02      3.0536e-02   \
    3.2653e+01       1.3133e+00       1.4043e+03
      21.3054           2.0390e-01       2.1228e+02      3.2050e-02   \
    3.0086e+01       1.5700e+00       1.4738e+03
      24.3856           2.1418e-01       2.2071e+02      3.0579e-03   \
    2.5080e+00       1.7143e-01       1.4060e+02
      26.7185           2.1639e-01       2.2236e+02      1.6974e-03   \
    1.2706e+00       1.0440e-01       7.8151e+01
      28.1997           2.1876e-01       2.2404e+02      1.4286e-03   \
    1.0132e+00       9.2736e-02       6.5771e+01
      29.9055           2.2086e-01       2.2545e+02      1.1975e-03   \
    8.0084e-01       8.2435e-02       5.5130e+01
      31.7795           2.2299e-01       2.2679e+02      1.0700e-03   \
    6.7342e-01       7.8275e-02       4.9261e+01
      33.8090           2.2493e-01       2.2794e+02      9.3685e-04   \
    5.5420e-01       7.2909e-02       4.3130e+01
      36.3742           2.2723e-01       2.2920e+02      7.5219e-04   \
    4.1359e-01       6.2962e-02       3.4619e+01
      38.8557           2.2862e-01       2.2991e+02      7.2847e-04   \
    3.7496e-01       6.5162e-02       3.3541e+01
      41.8657           2.3096e-01       2.3103e+02      5.6815e-04   \
    2.7142e-01       5.4726e-02       2.6143e+01
      45.1997           2.3233e-01       2.3164e+02      5.3931e-04   \
    2.3864e-01       5.6115e-02       2.4830e+01
      49.3039           2.3460e-01       2.3256e+02      4.0052e-04   \
    1.6247e-01       4.5419e-02       1.8424e+01
      53.9869           2.3595e-01       2.3306e+02      3.6589e-04   \
    1.3555e-01       4.5465e-02       1.6843e+01
      60.1244           2.3830e-01       2.3384e+02      2.7388e-04   \
    9.1106e-02       3.7853e-02       1.2591e+01
      67.5290           2.3959e-01       2.3423e+02      2.0645e-04   \
    6.1143e-02       3.2078e-02       9.5004e+00
      75.6420           2.4133e-01       2.3469e+02      1.7462e-04   \
    4.6170e-02       3.0370e-02       8.0299e+00
      87.3612           2.4310e-01       2.3509e+02      1.3141e-04   \
    3.0085e-02       2.6382e-02       6.0398e+00
     102.3268           2.4499e-01       2.3546e+02      1.1469e-04   \
    2.2417e-02       2.6965e-02       5.2704e+00
     130.3970           2.4721e-01       2.3580e+02      5.5938e-05   \
    8.5796e-03       1.6665e-02       2.5560e+00
     166.7821           2.4845e-01       2.3595e+02      3.7325e-05   \
    4.4759e-03       1.4287e-02       1.7132e+00
     248.5847           2.5013e-01       2.3608e+02      1.2880e-05   \
    1.0363e-03       7.2000e-03       5.7928e-01
     419.7893           2.5065e-01       2.3611e+02      2.4805e-06   \
    1.1818e-04       2.3458e-03       1.1176e-01

Now the idea of MarcoB can be applied:

SemanticImportString[text1] // Normal

returning this:

    {{16.454, 0.040763, 49.548, 0.019462, 23.656, 0.73634, 
  895.03}, {18.7036, 0.11419, 128.07, 0.030536, 32.653, 1.3133, 
  1404.3}, {21.3054, 0.2039, 212.28, 0.03205, 30.086, 1.57, 
  1473.8}, {24.3856, 0.21418, 220.71, 0.0030579, 2.508, 0.17143, 
  140.6}, {26.7185, 0.21639, 222.36, 0.0016974, 1.2706, 0.1044, 
  78.151}, {28.1997, 0.21876, 224.04, 0.0014286, 1.0132, 0.092736, 
  65.771}, {29.9055, 0.22086, 225.45, 0.0011975, 0.80084, 0.082435, 
  55.13}, {31.7795, 0.22299, 226.79, 0.00107, 0.67342, 0.078275, 
  49.261}, {33.809, 0.22493, 227.94, 0.00093685, 0.5542, 0.072909, 
  43.13}, {36.3742, 0.22723, 229.2, 0.00075219, 0.41359, 0.062962, 
  34.619}, {38.8557, 0.22862, 229.91, 0.00072847, 0.37496, 0.065162, 
  33.541}, {41.8657, 0.23096, 231.03, 0.00056815, 0.27142, 0.054726, 
  26.143}, {45.1997, 0.23233, 231.64, 0.00053931, 0.23864, 0.056115, 
  24.83}, {49.3039, 0.2346, 232.56, 0.00040052, 0.16247, 0.045419, 
  18.424}, {53.9869, 0.23595, 233.06, 0.00036589, 0.13555, 0.045465, 
  16.843}, {60.1244, 0.2383, 233.84, 0.00027388, 0.091106, 0.037853, 
  12.591}, {67.529, 0.23959, 234.23, 0.00020645, 0.061143, 0.032078, 
  9.5004}, {75.642, 0.24133, 234.69, 0.00017462, 0.04617, 0.03037, 
  8.0299}, {87.3612, 0.2431, 235.09, 0.00013141, 0.030085, 0.026382, 
  6.0398}, {102.327, 0.24499, 235.46, 0.00011469, 0.022417, 0.026965, 
  5.2704}, {130.397, 0.24721, 235.8, 0.000055938, 0.0085796, 0.016665,
   2.556}, {166.782, 0.24845, 235.95, 0.000037325, 0.0044759, 
  0.014287, 1.7132}, {248.585, 0.25013, 236.08, 0.00001288, 0.0010363,
   0.0072, 0.57928}, {419.789, 0.25065, 236.11, 2.4805*10^-6, 
  0.00011818, 0.0023458, 0.11176}}

I thank you for the discussion.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.