I am working with a lot of daily weather files in the txt format. I need to calculate a daily quantity from each of them, e.g., daily light intensity, which involves a simple summation of the light intensity column in Excel. As I am working with over three years of data, it is not possible to do this manually.

How can I automate this in Mathematica? Currently I import all the files using the following method. (Example shown is for Jan 1 to Jan 31)

filenames = FileNames[];  
TestList = Flatten[Table[Files[[j, i]], {j, 1, 31}, {i, 2, 1437}], 1];  
LightIntensity = TestList[[1 ;;, 5]];  
DailyLightIntensity = Partition[LightIntensity, 1436];  
DailySummation = Table[Total[DailyLightIntensity[[i]]],  {i, 1, 31}]

The method above only works assuming all the txt files contain rows from 2 to 1436, so I can use Partition, but I also need to handle cases with missing data.

I want to automate this process with Mathematica , import each txt file separately, perform the necessary calculations, and store the answer. How can this be done?

The naming convention of the txt is given below:

[112] 2012-01-01
[112] 2012-01-02
[112] 2012-01-03
[112] 2012-01-04
[112] 2012-12-31

  • $\begingroup$ Please see editing help. Thanks. $\endgroup$
    – Mr.Wizard
    Feb 15 '13 at 2:25
  • $\begingroup$ Could you post a sample data file with and without missing data? What are you using Paritition for? $\endgroup$
    – s0rce
    Feb 15 '13 at 3:32
  • $\begingroup$ I am using the partition function to separate the long list of values into separate days. Eg: When I Import all the values for the light intensity I get a long list of values for 31 days. Therefore I will partition it into 31 parts to see the values for each day. $\endgroup$ Feb 15 '13 at 3:42
  • $\begingroup$ Partition is not the better way to do that. How to you import your txt? Do you have a sample txt? $\endgroup$
    – Murta
    Feb 15 '13 at 4:18
  • $\begingroup$ @Murta The file can be found at link $\endgroup$ Feb 15 '13 at 4:25

This can be a starting point:

data=Rest@Import["http://dl.dropbox.com/u/10371498/SampleFile.txt", "TSV"];
{First@StringSplit[data[[1, 1]]],Total[data[[All,2]]]}

So you get:

{"1/1/12", 308439.}

For many files and for filter data you can create a function like:

    data=Rest@Import[fileName, "TSV"];
    data=Select[data, #[[2]] > 50 &];
    {First@StringSplit[data[[1, 1]]],Total[data[[All,2]]]}

And make it operate in each txt like:


You should get a list:

{{"1/2/12", 308439.},{"1/2/12", 208389.}, ... , {"1/3/12", 508439.}}
  • $\begingroup$ Thank you for choosing my answer. I hope it helped. :) $\endgroup$
    – Murta
    Feb 15 '13 at 10:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.