Create new file, based on 500 articles

Question

I've got a folder with 500 newspaper articles. Each article is saved in a seperate file with a number in the title like "1.txt", "2.txt". Each article has a date but it is not clear in which line the date is printed. I would like to seperate the date from the article and put them in two columns. The purpose is that I can make selection of articles based on periods of time.

A article looks like this:

The Light of Truth ENORMOUS IPO MAKES THE BILLIONAIRE 2013/12/18 CENTRUM, Tethys - the president and PRESIDENT international Sten can add another description to its name:.......

fname = FileNameJoin[{"E:\\data\\MC1 Data\\MC1 Data\\articles\\", 
"testfile3.txt"}];

s = OpenWrite[fname]
For[i = 1, i < 20, i++,
test1 = Import["E:\\data\\MC1 Data\\MC1 Data\\articles\\" <> ToString[i] <>".txt","Text"];

test2 = {
  StringCases[test1, DatePattern[{"Year", "Month", "Day"}]],StringCases[ToLowerCase[test1], RegularExpression["\\w(?<!\\d)[\\w'-]*"]]};

WriteString[s, test2, "\n"];

]

When I open the file,

test1 = Import["E:\\data\\MC1 Data\\MC1 Data\\articles\\testfile3.txt"]

I will see this:

{{2013/12/18}, {the, light, of, truth, enormous, ipo, makes, the, 
billionaire, of, sanjorge, centrum,}}
{2009/06/21}, {news, online, today, protest, turns, violent, in, 
abila, abila, kronos, a, rally, of, hundreds, of, protestors, allied, 
with, the, protectors, of, kronos, movement, turned, to, violence}

The structure of this file is like this :{{..},{..}} and I can't make a selection like

Select[file[All,{1,2}]], #[[1]]<"2009/06/21"&]

Anyone a suggestion how to solve this?

Answer 1

The problem is you are using methods to write and read the data which don't preserve the expression structure.

WriteString puts the OutputForm of an expression into the file, and so does not preserve the quote marks around strings. Here is a simple example:

data = {{"one", "two"}, {"three", "four"}};

Do[WriteString["test.txt", data[[i]], "\n"], {i, 2}];
Close["test.txt"];

Here' s the file it creates :

Looking at the file contents above, you can see that there is going to be an interpretation problem however you import the data. Is it a list of lists containing symbols one, two etc? Or is it a list of strings, where the strings just happen to contain curly brackets? Or maybe the whole file is one big string with carriage returns in it?

Import goes for the latter interpretation, because you are using the .txt format which is expected to contain text.

Import["test.txt"] // InputForm

(* "{one, two}\n{three, four}" *)

This is what happens with your data - it is imported as a single string.

What you can do is to use Write and ReadList instead of WriteString and Import. Write writes expressions in InputForm, so you get this:

Do[Write["test.txt", data[[i]]], {i, 2}];
Close["test.txt"];

Notice that the strings now have quote marks around them. This file can be imported using ReadList, with each line being correctly interpreted as a Mathematica expression:

ReadList["test.txt"] // InputForm

(* {{"one", "two"}, {"three", "four"}} *)

Comparing dates

For the final part of your code, you can use DateDifference to compare date strings. For example

isbefore[date1_String, date2_String] := DateDifference[date1, date2] >= 0

Select[{"2013/12/18", "2010/12/18", "2001/12/18"}, # ~isbefore~ "2009/06/21" &]

(* {"2001/12/18"} *)