1
$\begingroup$

I would like to import all .csv files in a directory and combine them into a single file using Join. I can do this manually but when I try to automate it using &/@ I run into problems.

The datafiles and example Mathematica notebook described in this question is available at this google drive link

Details

There are 7 files in the notebook directory, each with 6 header rows and ten rows of data. (Each row has 8 columns).

I use the code below to do the following:

  1. Search a directory for all file names containing ".csv"
  2. Import them
  3. Remove the header from each file.
root = NotebookDirectory[];

headerROWS = 6;
filenames = FileNames["*.csv", root];
files = Import[#] & /@ filenames;
files2 = Drop[#, headerROWS] & /@ files;

I verify that everything imported corrected as shown in screenshot below (7 files, each with 10 rows of data).

https://i.stack.imgur.com/4p65I.png

Stupid Manual Solution Using the same placeholder # and Join, it's straightforward but annoying to accomplish what I want manually:

MANUALjoin = 
 Join[files2[[1]], files2[[2]], files2[[3]], files2[[4]], files2[[5]], 
files2[[6]], files2[[7]]]; (*the dumb, manual way I want to avoid*)

The screenshot below shows this working as expected, i.e.: it simply concatenates the files while retaining the structure of the individual files.

enter image description here

The Problem When I try to automate the Join operation using code analogous to the &/@ in the first block of code, it jumbles the placement of the rows. This is what I've tried.

AUTOjoin = Join[#] & /@ files2;  (*Fails *)

I see the following jumbled output:

https://i.stack.imgur.com/TNNef.png

It seems that [#] is somehow incompatible with Join but I don't see exactly how. It seems that this code should be identical to the 'manual' version above. Does someone see what the problem is exactly?

$\endgroup$
1
  • 2
    $\begingroup$ Try Join @@ files2. $\endgroup$ – LouisB Mar 5 at 22:58
1
$\begingroup$

Join is expecting a sequence, as you can see in your MANUALjoin. You can convert your list of lists into a sequence of lists as follows:

SetDirectory[NotebookDirectory[]];
headerROWS = 6;
filenames = FileNames["*.csv"];
files = Import[#] & /@ filenames;
files2 = Drop[#, headerROWS] & /@ files;

autojoin = Join[Replace[files2, List -> Sequence, 1, Heads -> True]];
autojoin // TableForm

enter image description here

Incidentally, Join[#] & /@ files2 is equivalent to {Join[files2[[1]]],Join[files2[[2]]],...}, not Join[files2[[1]],files2[[2]],...]

$\endgroup$
2
  • $\begingroup$ We could also write autojoin = Catenate[files2];. $\endgroup$ – WReach Mar 6 at 0:58
  • $\begingroup$ @WReach, yes this is a better solution. Also Flatten[files2, 1] $\endgroup$ – MelaGo Mar 6 at 2:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.