3
$\begingroup$

I have data in TSV (tab-separated) files. The thing is, columns do not have headers. Importing data is as easy as

ds = Import["foo.tsv", {"Dataset"}];

Import even correctly deduces integer data type for integer-only columns. The problem is, the resulting dataset does not have column names. I know what the columns are, but I cannot find a way to add names during import, short of disassembling and reassembling the dataset (this answer provides very good working recommendations, including benchmarks), but the data size is not negligible). Is what I want possible? I am using MMA 11.3.0 on Windows.

TSV format has an Export option "TableHeadings"; I tried that for Import, but nothing happened, columns are still unnamed:

(* This does NOT work. *)
Import["foo.tsv", {"Dataset"}, "TableHeadings" -> {"a", "b", "c"}]
Improve this question
$\endgroup$
1
  • $\begingroup$ This is a good suggestion. $\endgroup$
    – GenericAccountName
    Feb 2, 2019 at 20:30

2 Answers 2

Reset to default
3
$\begingroup$

A relatively straightforward method is to Import and then name the columns via Dataset syntax like 

ds = Import["foo.tsv", {"Dataset"}][All, <|"col1" -> 1, "col2" -> 2, ...|>]
Improve this answer
$\endgroup$
7
  • $\begingroup$ Thanks, this is what I essentially do (but [Map[AssociationThread[{"col1", "col2", ...} -> #] &]] is about ×3 faster than [All, <|"col1" -> 1, "col2" -> 2, ...|>]). I just thought there could be a way to do that without rebuilding the dataset... $\endgroup$
    – kkm -still wary of SE promises
    Jan 26, 2019 at 3:59
  • $\begingroup$ I didn't know about that speedup, thanks a lot! Why don't you write your headings to the file before import? $\endgroup$
    – Carl Lange
    Jan 26, 2019 at 4:04
  • $\begingroup$ They are large (few million lines each), and I did not generate them. They come as manifests with a huge dataset that I am preparing for an ML task. As for the speedup, the benchmarks are in the answer linked to from my post. it's interesting how different ways to rename columns in an existing set compare! $\endgroup$
    – kkm -still wary of SE promises
    Jan 26, 2019 at 4:10
  • $\begingroup$ Even so, it seems like you could prepend the header to the file and then import them? Seems to me like that would be the easiest thing to do... $\endgroup$
    – Carl Lange
    Jan 26, 2019 at 8:13
  • $\begingroup$ Actually, I imported the files, added headers to datasets and saved them in .mx format, so I can load them faster. That's probably the quickest way (20s extra time at most per file, and I have a lot of RAM to afford restructuring the DS in memory). Adding headers to files is trickier, as they are too large for editing in a text editor. $\endgroup$
    – kkm -still wary of SE promises
    Jan 27, 2019 at 8:06
1
$\begingroup$

First export the data file:

Export["foo.tsv", {{11, 12}, {21, 22}}, "TSV"]

Import and name the columns explicitly:

tsvNamedCol=TableForm[Import["foo.tsv","TSV"],TableHeadings->{None,{"Col1","Col2"}}]

or, you can import and name the columns sequentially and automatically:

tsvAutoCol=TableForm[Import["foo.tsv","TSV"],TableHeadings->{None,Automatic}]

You can do the same with "CSV" files.

Improve this answer
$\endgroup$
4
  • $\begingroup$ There are a few million rows in each though... :( $\endgroup$
    – kkm -still wary of SE promises
    Jan 26, 2019 at 4:01
  • $\begingroup$ The columns will still be numbered correctly even if there are a few million rows. Adding the column headings is done after import, so it will not take time at all. Please remember to suppress output by adding ; to the end of the command line. $\endgroup$
    – Vixillator
    Jan 26, 2019 at 4:06
  • $\begingroup$ Thanks, I'll try that! $\endgroup$
    – kkm -still wary of SE promises
    Jan 26, 2019 at 4:12
  • $\begingroup$ If you get a chance to time the import with and without the TableForm naming of the columns, it will be good to know. Good luck. $\endgroup$
    – Vixillator
    Jan 26, 2019 at 4:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.