10
$\begingroup$

I'm working with a data format that prescribes separate header and data files, so for instance "file.hdr" and "file.dat". They are identified as belonging together by the fact that their base name is the same ("file" in the previous example), and that they reside in the same directories. In practice I will need to import the header before being able to import the data file, but assume for now that they are both simple text files we can easily import. An example of an import converter is given below. The additional Print statement will be used for illustration later.

ImportExport`RegisterImport["test", importTest]

importTest[file_, OptionsPattern[]] := 
 Module[{base = FileBaseName[file]},

  Print[file];

  If[StringMatchQ[FileExtension[file], "hdr" | "dat", IgnoreCase -> True],
   Check[{Import[base <> ".hdr", "Text"], Import[base <> ".dat", "Text"]}, Return[$Failed]],
       Return[$Failed]
  ]

 ]

Running this on two example files, "file.hdr" (containing the string "hdr") and "file.dat" (containing the string "dat"), gives the expected output:

Import["file.hdr", "test"]

(*Out: "D:\file.hdr"  *)
(*Out: {"hdr", "dat"} *)

Import["file.dat", "test"]

(*Out: "D:\file.dat"  *)
(*Out: {"hdr", "dat"} *)

The problem now lies with the fact that these .hdr and .dat files are often provided in a compressed format, e.g. "file.hdr.gz" and "file.dat.gz". This poses a problem in the example import converter:

Import["file.hdr.gz", "test"]

(*Out: "C:\Documents and Settings\user\Local Settings\Temp\m-xxxx-xxxx_file.hdr" *)
(*Import::nffil: File not found during Import.                                   *)
(*Import::nffil: File not found during Import.                                   *)
(*Out: $Failed                                                                   *)

Obviously the "file.dat.gz" file gives similar results.

Since Mathematica automatically unpacks compressed files (to the temp directory) before supplying them to their corresponding import functions, as one can see in the example the location of the user-provided file is obscured. My question is thus, how might I be able to import such split compressed files?

If somehow I could stop Mathematica from unpacking the compressed files automatically I think the problem would already be solved, but I can't find a way to do this. The only solutions I've come up with now are giving the original file as an option, which will pose an inconvenience for the user, or adding custom downvalues to Import, which gets messy very quickly.

$\endgroup$
3
  • $\begingroup$ I've resolved the issue, but I can't post an answer yet due to my low reputation score. The solution relies on an undocumented option called "OriginalChannel". I'll post an answer as soon as I can. $\endgroup$
    – tom
    Aug 9, 2013 at 16:11
  • $\begingroup$ If I'm not mistaken 1 rep is sufficient to post an answer... $\endgroup$ Aug 9, 2013 at 22:03
  • 1
    $\begingroup$ Yes, it was, but I had to wait eight hours from the posting of the question before I could post an answer. $\endgroup$
    – tom
    Aug 10, 2013 at 9:10

1 Answer 1

9
$\begingroup$

It took some searching, but I've figured out how solve the issue. It turns out that there's an undocumented option to ImportExport`RegisterImport called "OriginalChannel", which if set to True appends the user-given filename or stream to the options supplied to the import converter. An example of a working converter for the described files is thus:

ImportExport`RegisterImport["test", importTest, "OriginalChannel" -> True]

importTest[file_, opts:OptionsPattern[]] := 
 Module[{originalfile = "Channel" /. {opts}},

  Which[

   StringMatchQ[FileExtension[file], "hdr", IgnoreCase -> True],
   Check[{
     Import[file, "Text"], 
     Import[StringReplace[originalfile, ".hdr" -> ".dat", IgnoreCase -> True], "Text"]
   }, Return[$Failed]],

   StringMatchQ[FileExtension[file], "dat", IgnoreCase -> True],
   Check[{
     Import[StringReplace[originalfile, ".dat" -> ".hdr", IgnoreCase -> True], "Text"], 
     Import[file, "Text"]
   }, Return[$Failed]],

   True,
   Return[$Failed]

  ]

 ]

Importing an example file now produces the expected result:

Import["file.hdr.gz", "test"]

(*Out: {"hdr", "dat"} *)

Note that this code is not foolproof, it merely demonstrates a workaround for the problem posted in the question.

$\endgroup$
2
  • $\begingroup$ +1 for sharing your own solution; could you tell us how you found this option? $\endgroup$
    – Mr.Wizard
    Aug 10, 2013 at 9:49
  • 3
    $\begingroup$ @Mr.Wizard Sure. After being stuck on this for over a day I started to look at some of the Import.m files that come with Mathematica (in SystemFiles\Formats\ in $InstallationDirectory) and I found that there were options that I didn't recognize ("AlphaChannel" occurred in some image formats for instance). So in a final attempt I started from the top reading all the files, trying to figure out what these undocumented options meant. So finally I came across the Export.m file for the FLV format, and the "OriginalChannel" option was there, which turned out to do the trick. $\endgroup$
    – tom
    Aug 10, 2013 at 11:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.