3
$\begingroup$

I have data in TSV (tab-separated) files. The thing is, columns do not have headers. Importing data is as easy as

ds = Import["foo.tsv", {"Dataset"}];

Import even correctly deduces integer data type for integer-only columns. The problem is, the resulting dataset does not have column names. I know what the columns are, but I cannot find a way to add names during import, short of disassembling and reassembling the dataset (this answer provides very good working recommendations, including benchmarks), but the data size is not negligible). Is what I want possible? I am using MMA 11.3.0 on Windows.

TSV format has an Export option "TableHeadings"; I tried that for Import, but nothing happened, columns are still unnamed:

(* This does NOT work. *)
Import["foo.tsv", {"Dataset"}, "TableHeadings" -> {"a", "b", "c"}]
$\endgroup$
  • $\begingroup$ This is a good suggestion. $\endgroup$ – GenericAccountName Feb 2 at 20:30
3
$\begingroup$

A relatively straightforward method is to Import and then name the columns via Dataset syntax like

ds = Import["foo.tsv", {"Dataset"}][All, <|"col1" -> 1, "col2" -> 2, ...|>]
$\endgroup$
  • $\begingroup$ Thanks, this is what I essentially do (but [Map[AssociationThread[{"col1", "col2", ...} -> #] &]] is about ×3 faster than [All, <|"col1" -> 1, "col2" -> 2, ...|>]). I just thought there could be a way to do that without rebuilding the dataset... $\endgroup$ – kkm Jan 26 at 3:59
  • $\begingroup$ I didn't know about that speedup, thanks a lot! Why don't you write your headings to the file before import? $\endgroup$ – Carl Lange Jan 26 at 4:04
  • $\begingroup$ They are large (few million lines each), and I did not generate them. They come as manifests with a huge dataset that I am preparing for an ML task. As for the speedup, the benchmarks are in the answer linked to from my post. it's interesting how different ways to rename columns in an existing set compare! $\endgroup$ – kkm Jan 26 at 4:10
  • $\begingroup$ Even so, it seems like you could prepend the header to the file and then import them? Seems to me like that would be the easiest thing to do... $\endgroup$ – Carl Lange Jan 26 at 8:13
  • $\begingroup$ Actually, I imported the files, added headers to datasets and saved them in .mx format, so I can load them faster. That's probably the quickest way (20s extra time at most per file, and I have a lot of RAM to afford restructuring the DS in memory). Adding headers to files is trickier, as they are too large for editing in a text editor. $\endgroup$ – kkm Jan 27 at 8:06
1
$\begingroup$

First export the data file:

Export["foo.tsv", {{11, 12}, {21, 22}}, "TSV"]

Import and name the columns explicitly:

tsvNamedCol=TableForm[Import["foo.tsv","TSV"],TableHeadings->{None,{"Col1","Col2"}}]

or, you can import and name the columns sequentially and automatically:

tsvAutoCol=TableForm[Import["foo.tsv","TSV"],TableHeadings->{None,Automatic}]

You can do the same with "CSV" files.

$\endgroup$
  • $\begingroup$ There are a few million rows in each though... :( $\endgroup$ – kkm Jan 26 at 4:01
  • $\begingroup$ The columns will still be numbered correctly even if there are a few million rows. Adding the column headings is done after import, so it will not take time at all. Please remember to suppress output by adding ; to the end of the command line. $\endgroup$ – Vixillator Jan 26 at 4:06
  • $\begingroup$ Thanks, I'll try that! $\endgroup$ – kkm Jan 26 at 4:12
  • $\begingroup$ If you get a chance to time the import with and without the TableForm naming of the columns, it will be good to know. Good luck. $\endgroup$ – Vixillator Jan 26 at 4:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.