I am working with a lot of daily weather files in the txt format. I need to calculate a daily quantity from each of them, e.g., daily light intensity, which involves a simple summation of the light intensity column in Excel. As I am working with over three years of data, it is not possible to do this manually.

How can I automate this in Mathematica? Currently I import all the files using the following method. (Example shown is for Jan 1 to Jan 31)

filenames = FileNames[];  
TestList = Flatten[Table[Files[[j, i]], {j, 1, 31}, {i, 2, 1437}], 1];  
LightIntensity = TestList[[1 ;;, 5]];  
DailyLightIntensity = Partition[LightIntensity, 1436];  
DailySummation = Table[Total[DailyLightIntensity[[i]]],  {i, 1, 31}]

The method above only works assuming all the txt files contain rows from 2 to 1436, so I can use Partition, but I also need to handle cases with missing data.

I want to automate this process with Mathematica , import each txt file separately, perform the necessary calculations, and store the answer. How can this be done?

The naming convention of the txt is given below:

[112] 2012-01-01
[112] 2012-01-02
[112] 2012-01-03
[112] 2012-01-04
[112] 2012-12-31

  • $\begingroup$ Please see editing help. Thanks. $\endgroup$
    – Mr.Wizard
    Commented Feb 15, 2013 at 2:25
  • $\begingroup$ Could you post a sample data file with and without missing data? What are you using Paritition for? $\endgroup$
    – s0rce
    Commented Feb 15, 2013 at 3:32
  • $\begingroup$ I am using the partition function to separate the long list of values into separate days. Eg: When I Import all the values for the light intensity I get a long list of values for 31 days. Therefore I will partition it into 31 parts to see the values for each day. $\endgroup$ Commented Feb 15, 2013 at 3:42
  • $\begingroup$ Partition is not the better way to do that. How to you import your txt? Do you have a sample txt? $\endgroup$
    – Murta
    Commented Feb 15, 2013 at 4:18
  • $\begingroup$ @Murta The file can be found at link $\endgroup$ Commented Feb 15, 2013 at 4:25

1 Answer 1


This can be a starting point:

data=Rest@Import["http://dl.dropbox.com/u/10371498/SampleFile.txt", "TSV"];
{First@StringSplit[data[[1, 1]]],Total[data[[All,2]]]}

So you get:

{"1/1/12", 308439.}

For many files and for filter data you can create a function like:

    data=Rest@Import[fileName, "TSV"];
    data=Select[data, #[[2]] > 50 &];
    {First@StringSplit[data[[1, 1]]],Total[data[[All,2]]]}

And make it operate in each txt like:


You should get a list:

{{"1/2/12", 308439.},{"1/2/12", 208389.}, ... , {"1/3/12", 508439.}}
  • $\begingroup$ Thank you for choosing my answer. I hope it helped. :) $\endgroup$
    – Murta
    Commented Feb 15, 2013 at 10:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.