Take the 2-minute tour ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

I am working with a lot of daily weather files in the txt format. I need to calculate a daily quantity from each of them, e.g., daily light intensity, which involves a simple summation of the light intensity column in Excel. As I am working with over three years of data, it is not possible to do this manually.

How can I automate this in Mathematica? Currently I import all the files using the following method. (Example shown is for Jan 1 to Jan 31)

filenames = FileNames[];  
TestList = Flatten[Table[Files[[j, i]], {j, 1, 31}, {i, 2, 1437}], 1];  
LightIntensity = TestList[[1 ;;, 5]];  
DailyLightIntensity = Partition[LightIntensity, 1436];  
DailySummation = Table[Total[DailyLightIntensity[[i]]],  {i, 1, 31}]

The method above only works assuming all the txt files contain rows from 2 to 1436, so I can use Partition, but I also need to handle cases with missing data.

I want to automate this process with Mathematica , import each txt file separately, perform the necessary calculations, and store the answer. How can this be done?

The naming convention of the txt is given below:

[112] 2012-01-01
[112] 2012-01-02
[112] 2012-01-03
[112] 2012-01-04
...
[112] 2012-12-31

share|improve this question
    
Please see editing help. Thanks. –  Mr.Wizard Feb 15 '13 at 2:25
    
Could you post a sample data file with and without missing data? What are you using Paritition for? –  s0rce Feb 15 '13 at 3:32
    
I am using the partition function to separate the long list of values into separate days. Eg: When I Import all the values for the light intensity I get a long list of values for 31 days. Therefore I will partition it into 31 parts to see the values for each day. –  Raghav Malhotra Feb 15 '13 at 3:42
    
Partition is not the better way to do that. How to you import your txt? Do you have a sample txt? –  Murta Feb 15 '13 at 4:18
    
@Murta The file can be found at link –  Raghav Malhotra Feb 15 '13 at 4:25
show 2 more comments

1 Answer

up vote 3 down vote accepted

This can be a starting point:

data=Rest@Import["http://dl.dropbox.com/u/10371498/SampleFile.txt", "TSV"];
{First@StringSplit[data[[1, 1]]],Total[data[[All,2]]]}

So you get:

{"1/1/12", 308439.}

For many files and for filter data you can create a function like:

resumeFile[fileName_]:=Module[{data},
    data=Rest@Import[fileName, "TSV"];
    data=Select[data, #[[2]] > 50 &];
    {First@StringSplit[data[[1, 1]]],Total[data[[All,2]]]}
]

And make it operate in each txt like:

SetDirectory@"C:/txtPath"
files=FileNames["[112]*.txt"]
resumeFile/@files

You should get a list:

{{"1/2/12", 308439.},{"1/2/12", 208389.}, ... , {"1/3/12", 508439.}}
share|improve this answer
    
Thank you! It works well! –  Raghav Malhotra Feb 15 '13 at 10:18
    
Thank you for choosing my answer. I hope it helped. :) –  Murta Feb 15 '13 at 10:22
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.