Joining large number of columns

Question

I have 30 individual columns in 30 data files which have around 1000 numbers in each of them. I want to make a grand table by joining all these columns in the form of a single table with 30 columns. I am using following procedure-

  d1=Import["/home/d1.out","Table"];
  .
  .       
  data= Join[d1,d2,d3....d30,2]

Where d1,d2...d30 corresponds to the data files that I am importing. In grand table data, Last column is not complete. It has last 500 or so values missing. What am I doing wrong?

Dummy example: Suppose I have three columns-

        1         5        9
        2         6        10
        3         7        11
        4         8        12

I want my table to look like this after joing above three columns-

    1  5  9
    2  6  10 
    3  7  11
    4  8  12

Instead, I am getting this-

   1  5   9
   2  6   10
   7  11
   8  12 

Dropbox link to original data files: https://www.dropbox.com/sh/n6dz1zz3r65zjhy/AAATH_mX-NeyXolx3YC28f1ca?dl=0

Answer 1

Generally speaking, if the format of the files is the same as those you are working with, you probably want to import as "List", rather than "Table", because the table format will add a spurious level of curly braces.

In the set you currently shared, there are 46 files of varying length. Most of them contain 99 elements, one contains 69 elements, and two contain many thousands of elements.

Let's import them all, as lists:

SetDirectory["fullPath/toFolder/whereYou/savedYour/files"]
data = Import[#, "List"] & /@ FileNames[];

Let's take a look at the sizes of the data imported:

Length /@ data

(*Out: {99, 99, 99, 99, 99, 99, 99, 99, 99, 99, 99, 99, (*SHORT*) 69, 99, 99, 99, 99, 99, 
       (*LONG*) 11579, 99, 99, (*LONG*) 11579, 99, 99, 99, 99, 99, 99, 99, 99, 99, 99, 
       99, 99, 99, 99, 99, 99, 99, 99, 99, 99, 99, 99, 99, 99} *)

As you can see, not all files contain the same number of data points. A quick inspection with a text editor confirmed this.

Let's select a subset of files with the same length, e.g. 99 elements:

ave = Select[data, Length[#] == 99 &];
Dimensions[ave]

(* Out: {43, 99} *)

As you can see, we have 43 sets, each 99 elements long; in other words, we have a list of lists, each row representing a file. Your "grand table" is simply the transpose of that list, in which each column represents a file:

grand = Transpose[ave];
Dimensions[grand]

(* Out: {99, 43} *)

As you can see, grand is a "table" of 99 rows by 43 columns, in which each file is represented by a column.

---

Note that each file's position in the grand table is currently only determined by alphabetical order of the file names. If you need different ordering, you can change to file names to suit (perhaps the easiest way), or associate the origin file name with each data set during the import phase, and sort afterwards. An Association or a Dataset could work well for that.