5
$\begingroup$

Often Import related questions come up, for which I'm always wanting to inspect the proper import function used to see what Options and features I can get out of it.

For this I generally use this function:

findImportFormatFile[fmt_] :=
 SelectFirst[
  Map[Function @ FileNameJoin @ {#, fmt, "Import.m"},
   System`ConvertersDump`$FormatsDirectory
   ],
  FileExistsQ,
  $Failed
  ]

which for instance can find the Import definitions for "CSV":

findImportFormatFile["CSV"]

"/Applications/Mathematica.app/Contents/SystemFiles/Formats/CSV/Import.m"

from which we can glean that the base import function is:

System`Convert`TableDump`ImportSV

but in a fresh kernel this isn't defined, even after we use System`ConvertersDump`LoadImportFormat.

Looking at the data further we see the issue is the "Sources" for the importer haven't been loaded:

ImportExport`DefaultSources["Table"]

{"/Applications/Mathematica.app/Contents/SystemFiles/Kernel/SystemResources/MacOSX-x86-64/Convert/Table.mx"}

And after we import this we're good.

But I'd like a way to get all this data--import function, options names, "Sources", etc.--automatically.

How can I do this?


Helper code

Here's a partial solution I wrote up before thinking to pose this question. Hopefully it's helpful if someone wants to roll a scrape, if there's no more legit way to do this.

getFormatImportDataRaw[fmt_] :=

  Block[{$Context = 
     "ItsASecretToEverybody`Private`", $ContextPath = {"System`"}},
   Replace[
    SelectFirst[
     Map[Function @ FileNameJoin @ {#, fmt, "Import.m"},
      System`ConvertersDump`$FormatsDirectory
      ],
     FileExistsQ,
     $Failed
     ],
    s_String :> s -> Import[s, "HeldExpressions"]
    ]
   ];
getFormatImportData[fmt_] :=
 Module[{
   data, 
   file
   },
  data = getFormatImportDataRaw[fmt];
  If[data =!= $Failed,
   {file, data} = List @@ data;
   data /. {
     HoldPattern[Set[s_Symbol, v : _List | _Association]] :>
      RuleCondition[Set[s, v], True]
     };
   With[{cont = 
      FirstCase[data, HoldPattern[Begin[c_]] :> c, 
       "System`FakeHo`", ∞]},
    SortBy[
       Switch[#[[1]], 
         "FormatName", 0,
         "BinaryFormat", 2,
         "File", 3,
         _, 1
         ] &
       ]@
      Flatten@{
        "DocumentedElements" ->
         FirstCase[data, 
          HoldPattern[
            Set[_Symbol?(Function[Null, 
                SymbolName[Unevaluated@#] == "$DocumentedElements", 
                HoldFirst]), v_]] :> Hold@v,
          None,
          ∞
          ],
        "HiddenElements" ->
         FirstCase[data, 
          HoldPattern[
            Set[_Symbol?(Function[Null, 
                SymbolName[Unevaluated@#] == "$HiddenElements", 
                HoldFirst]), v_]] :> Hold@v,
          None,
          ∞
          ],
        FirstCase[data,
         HoldPattern[
           ImportExport`RegisterImport[
            fmtName_, 
            raw_,
            post : _List : {},
            ruls___?System`ConvertersDump`RuleQ
            ]
           ] :>
          {
           "FormatName" -> fmtName,
           Replace[
            Flatten@List@raw,
            {
             (r : Rule | RuleDelayed)[k_, h_] :>
              r[k, h],
             f : Except[_Rule | _RuleDelayed] :> "DefaultImport" -> f
             },
            1
            ],
           Replace[
            post,
            {
             (r : Rule | RuleDelayed)[k_, h_] :>
              r[k, h],
             f : Except[_Rule | _RuleDelayed] :> "DefaultImport" -> f
             },
            1
            ],
           Replace[{ruls},
            ("AvailableElements" -> _Symbol) -> Nothing,
            1
            ],
           "File" -> file
           },
         None,
         ∞
         ]
        } /. 
     s_Symbol?(Function[Null, 
         Context[#] == "ItsASecretToEverybody`Private`", 
         HoldFirst]) :> 
      RuleCondition[
       ToExpression[cont <> SymbolName[Unevaluated[s]], StandardForm, 
        Inactivate], True]
    ],
   {}
   ]
  ]

It's not great and if there's a built-in way I'd much, much prefer that.

It does work for some formats, though:

getFormatImportData@"CSV" // 
  Take[#, 3] & // GeneralUtilities`PrettyForm

{
    "FormatName" -> "CSV", 
    "DefaultElement" -> "Data", 
    "DefaultImport" -> (System`Convert`TableDump`ImportSV["CSV", System`Convert`TableDump`elem, All, SlotSequence[1]] &)
 }

Added export scraper

Here's a scraper for export formats:

getFormatExportDataRaw[fmt_] := 
  Block[{$Context = 
     "ItsASecretToEverybody`Private`", $ContextPath = {"System`"}}, 
   Replace[
    SelectFirst[
     Map[Function@FileNameJoin@{#, fmt, "Export.m"}, 
      System`ConvertersDump`$FormatsDirectory],
     FileExistsQ, $Failed], 
    s_String :> s -> Import[s, "HeldExpressions"]
    ]
   ];
getFormatExportData[fmt_] :=
 Module[{data, file},
  data = getFormatExportDataRaw[fmt];
  If[data =!= $Failed,
   {file, data} = List @@ data;
   data /. {
     HoldPattern[Set[s_Symbol, v : _List | _Association]] :>
      RuleCondition[Set[s, v], True]
     };
   With[{cont = 
      FirstCase[data, HoldPattern[Begin[c_]] :> c, 
       "System`FakeHo`", \[Infinity]]},
    SortBy[
       Switch[#[[1]], "FormatName", 0, "BinaryFormat", 2, "File", 
         3, _, 1] &
       ]@
      Flatten@{
        "DocumentedElements" ->
         FirstCase[data,
          HoldPattern[
            Set[
             _Symbol?(
               Function[Null,
                SymbolName[Unevaluated@#] == "$DocumentedElements",
                HoldFirst
                ]),
             v_]
            ] :> Hold@v, None, \[Infinity]],
        "HiddenElements" ->
         FirstCase[data,
          HoldPattern[
            Set[
             _Symbol?(
               Function[Null,
                SymbolName[Unevaluated@#] == "$HiddenElements",
                HoldFirst
                ]),
             v_
             ]
            ] :> Hold@v,
          None,
          \[Infinity]],
        FirstCase[data,
         HoldPattern[
           ImportExport`RegisterExport[fmtName_,
            function_,
            ruls___?System`ConvertersDump`RuleQ
            ]
           ] :> {
           "FormatName" -> fmtName,
           "Function" -> function,
           Replace[{ruls},
            ("AvailableElements" -> _Symbol) -> Nothing,
            1],
           "File" -> file
           },
         None,
         \[Infinity]]
        } /. s_Symbol?(
        Function[Null,
         Context[#] == "ItsASecretToEverybody`Private`",
         HoldFirst
         ]
        ) :>
      RuleCondition[
       ToExpression[cont <> SymbolName[Unevaluated[s]],
        StandardForm,
        Inactivate
        ],
       True]
    ],
   {}
   ]
  ]
$\endgroup$
  • $\begingroup$ This is interesting. I will try to look at this in more detail later, I already commented on that other post that there's no way to look at Options via the register functions and this is true for both Import and Export. $DocumentedElements is not reliable at all, if you look through the various Import.m and Export.m files you'll see they are not completely consistent, which makes these types of approaches rather difficult to implement. $\endgroup$ – GenericAccountName Feb 4 at 19:52
  • $\begingroup$ Finding whether an element is documented or hidden in general means looking at the list of elems, any LHS which is a string should be documented, if it's in AvailableElements but not Import[.., "Elements"] it's hidden. One step in hiding is making sure the LHS is somethign like elem_:"FOO" so it is a Pattern and not a String. $DocumentedElements isn't necessarily named consistently between formats either, and these registrations change often. $\endgroup$ – GenericAccountName Feb 4 at 19:54
  • $\begingroup$ I think we all agree one should be able to figure out which documented Options are available for example, but of course we aren't going to provide methods to look for hidden or undocumented elements etc, they are hidden for a reason and subject to change. The tips I'm giving like how to tell if an element is hidden is to help in your spelunking journey, but also is not guaranteed to be reliable. $\endgroup$ – GenericAccountName Feb 4 at 19:57
  • $\begingroup$ In case you were wondering, the "Options" that shows up in RegisterImport and RegisterExport isn't used as much these days, and is different from what one typically thinks of as options for a particular format. These are options which are also elements, and used with converters that typically have post importers and internal functions that return lists of rules. It's not likely to be of much interest when spelunking. "Options" to RegisterImport and RegisterExport don't even show up in the tutorial/DevelopingAnImportConverter etc $\endgroup$ – GenericAccountName Feb 4 at 20:00
  • $\begingroup$ Some of the time if you find the correct function the converter calls, like System Convert TableDump ImportSV so Options[System Convert TableDump ImportSV] might actually be the thing of interest in this particular case. Can't guarantee this in general. $\endgroup$ – GenericAccountName Feb 4 at 20:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.