Handy / flexible directory tree operations

Question

Is there any shorter / more handy / more flexible way to get directory tree as a nested Rules/Associations list?

I tried to incorporate FileNames + GatherBy and GroupBy, but nothing was as short and handy as old fashioned:

ClearAll @ filesInDir;
filesInDir[dirname_, deep_] :=  If[
    DirectoryQ[dirname],
    FileBaseName[ dirname] -> (
          If[   deep == 1, 
                FileNameTake[#, -1], 
                filesInDir[#, deep - 1]
            ] & /@ FileNames["*", dirname]),
    FileNameTake[dirname, -1]
];


filesInDir[$UserBaseDirectory, 2]

"Mathematica" -> {"ApplicationData" -> {"CCompilerDriver", "CloudObject", "CUDALink", "Parallel"}, "Applications" -> {...}, "Autoload" -> {...}, "FrontEnd" -> {"init.m"}, "Kernel" -> {"init.m"}, "Licensing" -> {}, "Paclets" -> {"Configuration", "Repository", "Temporary"}, "SystemFiles" -> {"CharacterEncodings", "FrontEnd", "Kernel", "SpellingDictionaries"}}

And now, from this nested rules I want to go back to the output of:

FileNameSplit /@ FileNames["*", $UserBaseDirectory, 2]

---

p.s.

I've chosen nested lists/associations to reduce ByteCount and make querying more user friendly. But if you can convince me I should keep a full paths list it would be a valid answer too.

Answer 1

First off, a more compact form of filesInDir:

fileTree[f_, d_] /; DirectoryQ[f] && d > 0 := 
 FileNameTake[f] -> (fileTree[#, d - 1] & /@ FileNames[___, f])
fileTree[f_, _] := FileNameTake[f]

Example usage:

fileTree[$UserBaseDirectory, 2]

(* "Mathematica" -> {"ApplicationData" -> {"CCompilerDriver", 
    "DeviceFramework", "ImageAcquisition", "Parallel"}, 
  "Applications" -> {}, "Autoload" -> {"PacletManager"}, 
  "FrontEnd" -> {"init.m"}, "Kernel" -> {"init.m"}, "Licensing" -> {},
   "Paclets" -> {"Configuration", "Repository", "Temporary"}, 
  "SystemFiles" -> {"CharacterEncodings", "FrontEnd", "Java", 
    "Kernel", "LibraryResources", "SpellingDictionaries"}} *)

Personally I don't like the way that bottom-level directories are treated the same way as files (head String instead of head Rule). Here's an alternate method:

Clear[fileTree]

fileTree[f_, d_] /; DirectoryQ[f] := 
 FileNameTake[f] -> If[d > 0, fileTree[#, d-1]& /@ FileNames[___, f], Missing["TooDeep"]]
fileTree[f_, _] := FileNameTake[f]

Example usage:

fileTree[$UserBaseDirectory, 2]

(* "Mathematica" -> {"ApplicationData" -> {"CCompilerDriver" -> 
     Missing["TooDeep"], "DeviceFramework" -> Missing["TooDeep"], 
    "ImageAcquisition" -> Missing["TooDeep"], 
    "Parallel" -> Missing["TooDeep"]}, "Applications" -> {}, 
  "Autoload" -> {"PacletManager" -> Missing["TooDeep"]}, 
  "FrontEnd" -> {"init.m"}, "Kernel" -> {"init.m"}, "Licensing" -> {},
   "Paclets" -> {"Configuration" -> Missing["TooDeep"], 
    "Repository" -> Missing["TooDeep"], 
    "Temporary" -> Missing["TooDeep"]}, 
  "SystemFiles" -> {"CharacterEncodings" -> Missing["TooDeep"], 
    "FrontEnd" -> Missing["TooDeep"], "Java" -> Missing["TooDeep"], 
    "Kernel" -> Missing["TooDeep"], 
    "LibraryResources" -> Missing["TooDeep"], 
    "SpellingDictionaries" -> Missing["TooDeep"]}} *)

You can replace Missing["TooDeep"] with whatever you want, as long as it's not a List.

---

To convert a tree (either form) to your list format:

treeToList[d_ -> _Missing] := {{d}}
treeToList[s_String] := {{s}}
treeToList[d_ -> l_] := Prepend[Prepend[d] /@ Join @@ (treeToList /@ l), {d}]

Example usage:

tree = fileTree[$UserBaseDirectory, 2];
treeToList[tree] // Column

(* {
 {"Mathematica"},
 {"Mathematica", "ApplicationData"},
 {"Mathematica", "ApplicationData", "CCompilerDriver"},
 {"Mathematica", "ApplicationData", "DeviceFramework"},
 {"Mathematica", "ApplicationData", "ImageAcquisition"},
 {"Mathematica", "ApplicationData", "Parallel"},
 {"Mathematica", "Applications"},
 {"Mathematica", "Autoload"},
 {"Mathematica", "Autoload", "PacletManager"},
 {"Mathematica", "FrontEnd"},
 {"Mathematica", "FrontEnd", "init.m"},
 {"Mathematica", "Kernel"},
 {"Mathematica", "Kernel", "init.m"},
 {"Mathematica", "Licensing"},
 {"Mathematica", "Paclets"},
 {"Mathematica", "Paclets", "Configuration"},
 {"Mathematica", "Paclets", "Repository"},
 {"Mathematica", "Paclets", "Temporary"},
 {"Mathematica", "SystemFiles"},
 {"Mathematica", "SystemFiles", "CharacterEncodings"},
 {"Mathematica", "SystemFiles", "FrontEnd"},
 {"Mathematica", "SystemFiles", "Java"},
 {"Mathematica", "SystemFiles", "Kernel"},
 {"Mathematica", "SystemFiles", "LibraryResources"},
 {"Mathematica", "SystemFiles", "SpellingDictionaries"}
} *)

We can't get back to the full output of FileNameSplit /@ FileNames["*", $UserBaseDirectory, 2] since the base path is lost when converting to a tree.

Answer 2

Leonid Shifrin's implementation of a trie can be used for this purpose. It is available both for lists and for associations. For associations:

ClearAll[makeTreeAssoc];
makeTreeAssoc[wrds : {__String}] := Association@makeTreeAssoc[FileNameSplit /@ wrds];
makeTreeAssoc[wrds_ /; MemberQ[wrds, {}]] := makeTreeAssoc[DeleteCases[wrds, {}]];
makeTreeAssoc[wrds_] := Reap[If[# =!= {}, Sow[Rest[#], First@#]] & /@ wrds, _, #1 -> Association@makeTreeAssoc[#2] &][[2]]

Now you can do

tree[dir_] := makeTreeAssoc[FileNames["*", dir, 2]]
myTree = First@tree[$UserBaseDirectory]

Since the directory/file names are used as keys, I don't think there is a way to differentiate the files from the directories in the structure. You would have to check what it is upon retrieval.

OK so now we have to be able to go back to that other format too... I ended up with this

goBack[tree_, path_] := (
  Sow[path];
  If[Keys[tree] =!= {}, goBack[
      tree[#],
      Append[path, #]
      ] & /@ Keys[tree]]
  )

len = Length@FileNameSplit[$UserBaseDirectory];
Select[
 Reap[goBack[myTree, {}]][[2, 1]],
 Length[#] >= len &
 ]

Answer 3

Update: much simpler now. If this works right I think I would consider it superior.

Update 2: I attempted to add specific handling of files and directories. Please tell me if this appears to work and produce a format that is acceptable to you.

fn[x : {___, {_, __}, ___}] := Normal @ GroupBy[x, First -> Rest, fn]
fn[x_List] := Join @@ x

files =
  Append[FileNameSplit@#, FileType@#] & /@
    FileNames["*", $UserBaseDirectory, 2];

DeleteCases[fn @ files, Directory -> {}, -3] /.
  {{Directory} -> {}, (x_ -> {File}) :> x}

Answer 4

Try this recursive query which is linear-time in input (timing study here)

trie = Query[GroupBy@First, If[First@# == {}, {}, trie@#] &, Rest];

Variations on the above are possible and useful (see Antonov's Trie page here developed in 2013 but without Dataset).

Assume filenames have been split into path component lists, eg "a/b/c" --> {"a","b","c"} :

flatFiles = FileNames["*.gsheet", "~/Google Drive/...", Infinity] // Dataset

Then:

flatFiles[All, StringSplit[#, "/"] &][trie]