4
$\begingroup$

Problem Description

I have my data arranged in a tab delimited file and I'm trying to parse/access certain parts of it.

As an example, I'd like to extract the number of points and the sampling frequency as well as the time series which is saved like below (>> = tab, bullet = space):

Data file

Before and after this excerpt, there is a lot of other data that I don't need.

Attempt

I tried to find the position of interest in the file with the Find command and then readin the values with Read. I must be doing something wrong as an invalid data format error is thrown.

stream = OpenRead["test.txt"];
(* Find the line, that begins with "Number of points" *)
remarks = Find[stream,"Number of points"] (* now pointer at end of line *)

(* The next line is: Sampling frequency [Hz]    12000*)
Read[stream, {String, String, Real}, RecordSeparators->{"\t"}]

Output:

Out[256]= Number of points      100000
During evaluation of In[255]:= Read::readn: Invalid real number found when reading from test.txt. >>
Out[257]= $Failed

Sample Data Set (tabs get replaced by 4xwhitespace)

Here the position of interest would be e.g. "2. dataset". From there one I'd like to read the sampling frequency and the number of sample points, which then can be used to read in the {time, value} paired data.

1. dataset

Number of points        3
Sampling frequency  [Hz]    12000


Time [s]    Value
--------    -----

0.000000    0.8972
0.003333    0.9015
0.006667    0.9061

2. dataset

Number of points        4
Sampling frequency  [Hz]    12000


Time [s]    Value
--------    -----

0.000000    0.8972
0.003333    0.518
0.006667    0.8971
0.006667    0.8648

3. dataset

Number of points        3
Sampling frequency  [Hz]    12000


Time [s]    Value
--------    -----

0.000000    0.7716
0.003333    0.2674
0.006667    0.8899
$\endgroup$
  • $\begingroup$ Please add a relevant sample from your datafile so we can try and play with it to help you. It would be very hard to figure out exactly what is going wrong otherwise. Also, readNumber is not a built-in Mathematica function: where did you get it? $\endgroup$ – MarcoB Nov 9 '15 at 15:43
  • $\begingroup$ Thanks for your input. I've added a sample but on stackexchange the tabs get replaced by whitespaces. I removed the reference to readNumber - I found it in an old Mathematica file with no source stated. $\endgroup$ – xphan Nov 9 '15 at 16:21
1
$\begingroup$

Your sample data set (tabs replaced by spaces)

str =

  "1. dataset

   Number of points 3  
   Sampling frequency[Hz] 12000
   Time[s] Value
   -------------
   0.000000 0.8972
   0.003333 0.9015
   0.006667 0.9061 

   2. dataset

   Number of points 4
   Sampling frequency[Hz] 12000
   Time[s] Value
   -------------
   0.000000 0.8972  
   0.003333 0.518   
   0.006667 0.8971   
   0.006667 0.8648      

   3. dataset 

   Number of points 3
   Sampling frequency[Hz] 12000
   Time[s] Value
   -------------
   0.000000 0.7716    
   0.003333 0.2674   
   0.006667 0.8899";

Get all numbers

Alternatives of digits, space and point

alt = Alternatives @@ Join[CharacterRange["0", "9"], {" ", "."}]

"0" | "1" | "2" | "3" | "4" | "5" | "6" | "7" | "8" | "9" | " " | "."

vec = ToExpression @ StringSplit @ StringJoin @ StringCases[str, alt]

{1., 3, 12000, 0., ... 0.2674, 0.006667, 0.8899}

Positions of datasets

pos = 
  Transpose[{#, Append[Rest @ # - 1, Length @ vec]}] &[
     Flatten @ Position[vec, _Integer][[;; ;; 2]] - 1]

{{1, 9}, {10, 20}, {21, 29}}

Matrix of datasets

(mat = Take[vec, #] & /@ pos) // MatrixForm

enter image description here

Partition headers and points

mat =
  Transpose[{
    mat[[All, ;; 3]],
    Map[Partition[#, 2] &, mat[[All, 4 ;;]]]}];

MatrixForm[%]

enter image description here

Examples

Get all points of second dataset

mat[[2, 2]]

{{0., 0.8972}, {0.003333, 0.518}, {0.006667, 0.8971}, {0.006667, 0.8648}}

Get all frequencies

mat[[All, 1, 3]]

{12000, 12000, 12000}

$\endgroup$
  • $\begingroup$ Thanks a lot @eldo. I'm marking your reponse as an answer because it additionally allows to parse multiple datasets. $\endgroup$ – xphan Nov 11 '15 at 15:12
3
$\begingroup$

Once you have your data in the form of a string, Mathematica has very powerful string operations that can parse the data.

I am not claiming that this is the most efficient method but it will work with your data (or possibly some minor tweaks).

Starting with your first example:

string1 = "Number of points 3

  Sampling frequency  [Hz]  12000


  Time [s]  Value
  --------  -----

  0.000000  0.8972
  0.003333  0.9015
  0.006667  0.9061";

First we will develop the code step by step and then create a function to do the entire procedure.

Develop Code

Step 1 - Split the string into lines

string2 = StringSplit[data1, EndOfLine]

This creates a list with each line being a row.

Step 2 - Extract the number of points

Here (and in the other parts of the code) we will use StringCases to get the relevant data. This will output an empty list for each row that doesn't match. We will wrap this in DeleteCases to remove the empty list and finally use Part to remove the curly brackets.

numberPoints = DeleteCases[
   StringCases[string2, 
    "Number of points" ~~ Whitespace ~~ np : NumberString -> 
     ToExpression[np]], {}][[1, 1]]

(* "3" *)

Step 2 - Extract the frequency

frequency = DeleteCases[
   StringCases[string2, 
    "Sampling frequency  [Hz]" ~~ Whitespace ~~ freq : NumberString ->
      ToExpression[freq]], {}][[1, 1]]

(* "12000" *)

step 3 - Extract the data

This is the same as the previous steps except we will use Flatten to remove one curly bracket.

data = Flatten[
   DeleteCases[
   StringCases[string2, 
    time : NumberString ~~ Whitespace ~~ 
      value : NumberString -> {ToExpression[time], 
      ToExpression[value]}], {}],
  1]
(* {{"0.000000", "0.8972"}, {"0.003333", 
  "0.9015"}, {"0.006667", "0.9061"}} *)

Define the function

Now that we have tested the individual steps a function is created to do everything.

extractData[string_String] := Module[
  {
   string2,
   numberPoints,
   frequency,
   data
   },

  string2 = StringSplit[string, EndOfLine];

  numberPoints = DeleteCases[
     StringCases[string2, 
      "Number of points" ~~ Whitespace ~~ np : NumberString -> 
       ToExpression[np]], {}][[1, 1]];

  frequency = DeleteCases[
     StringCases[string2, 
      "Sampling frequency  [Hz]" ~~ Whitespace ~~ 
        freq : NumberString -> ToExpression[freq]], {}][[1, 1]];

  data = Flatten[
     DeleteCases[
     StringCases[string2, 
      time : NumberString ~~ Whitespace ~~ 
        value : NumberString -> {ToExpression[time], 
        ToExpression[value]}], {}],
    1];

  {numberPoints, frequency, data}
  ]

Test the function

Execute it on the test string

{numberPoints, frequency, data} = extractData[string1]

(* {"3", "12000", {{"0.000000", "0.8972"}, {"0.003333", 
   "0.9015"}, {"0.006667", "0.9061"}}} *)
$\endgroup$
  • $\begingroup$ Thank you for your clear instructions. One problem arises if you have multiple sets that might have the same naming conventions (e.g. Number of points) as in the dataset of my edit. Is there a way to focus the selection to the first match below a certain unique marker such as "2. dataset"? $\endgroup$ – xphan Nov 10 '15 at 12:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.