1
$\begingroup$

I have 30 individual columns in 30 data files which have around 1000 numbers in each of them. I want to make a grand table by joining all these columns in the form of a single table with 30 columns. I am using following procedure-

  d1=Import["/home/d1.out","Table"];
  .
  .       
  data= Join[d1,d2,d3....d30,2]

Where d1,d2...d30 corresponds to the data files that I am importing. In grand table data, Last column is not complete. It has last 500 or so values missing. What am I doing wrong?

Dummy example: Suppose I have three columns-

        1         5        9
        2         6        10
        3         7        11
        4         8        12

I want my table to look like this after joing above three columns-

    1  5  9
    2  6  10 
    3  7  11
    4  8  12

Instead, I am getting this-

   1  5   9
   2  6   10
   7  11
   8  12 

Dropbox link to original data files: https://www.dropbox.com/sh/n6dz1zz3r65zjhy/AAATH_mX-NeyXolx3YC28f1ca?dl=0

$\endgroup$
  • 1
    $\begingroup$ Can you maybe give a smaller example that demonstrates the same problem? $\endgroup$ – J. M. will be back soon Aug 8 '17 at 16:34
  • $\begingroup$ @J.M. I have edited the question. $\endgroup$ – physics101 Aug 8 '17 at 17:00
  • $\begingroup$ @physics101 Thank you for adding an example, but I am afraid that this is still not enough for us to figure out what the problem is. You might want to share a couple of files from the data set you have, and the full code you use to import them and then join them, so that we can try it out on our own, figure out what goes wrong, and help you out. Make sure that the data you give us reproduces your problem. You can use e.g. pastebin or a dropbox link to share data. $\endgroup$ – MarcoB Aug 8 '17 at 17:10
  • $\begingroup$ @MarcoB I have edited the question. $\endgroup$ – physics101 Aug 8 '17 at 17:26
  • $\begingroup$ @physics101 It's probably a problem with the file. Take a look at the result of Import["/home/d30.out","Table"]. $\endgroup$ – Alan Aug 8 '17 at 18:00
1
$\begingroup$

Generally speaking, if the format of the files is the same as those you are working with, you probably want to import as "List", rather than "Table", because the table format will add a spurious level of curly braces.

In the set you currently shared, there are 46 files of varying length. Most of them contain 99 elements, one contains 69 elements, and two contain many thousands of elements.

Let's import them all, as lists:

SetDirectory["fullPath/toFolder/whereYou/savedYour/files"]
data = Import[#, "List"] & /@ FileNames[];

Let's take a look at the sizes of the data imported:

Length /@ data

(*Out: {99, 99, 99, 99, 99, 99, 99, 99, 99, 99, 99, 99, (*SHORT*) 69, 99, 99, 99, 99, 99, 
       (*LONG*) 11579, 99, 99, (*LONG*) 11579, 99, 99, 99, 99, 99, 99, 99, 99, 99, 99, 
       99, 99, 99, 99, 99, 99, 99, 99, 99, 99, 99, 99, 99, 99} *)

As you can see, not all files contain the same number of data points. A quick inspection with a text editor confirmed this.

Let's select a subset of files with the same length, e.g. 99 elements:

ave = Select[data, Length[#] == 99 &];
Dimensions[ave]

(* Out: {43, 99} *)

As you can see, we have 43 sets, each 99 elements long; in other words, we have a list of lists, each row representing a file. Your "grand table" is simply the transpose of that list, in which each column represents a file:

grand = Transpose[ave];
Dimensions[grand]

(* Out: {99, 43} *)

As you can see, grand is a "table" of 99 rows by 43 columns, in which each file is represented by a column.


Note that each file's position in the grand table is currently only determined by alphabetical order of the file names. If you need different ordering, you can change to file names to suit (perhaps the easiest way), or associate the origin file name with each data set during the import phase, and sort afterwards. An Association or a Dataset could work well for that.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.