# Handy / flexible directory tree operations

Is there any shorter / more handy / more flexible way to get directory tree as a nested Rules/Associations list?

I tried to incorporate FileNames + GatherBy and GroupBy, but nothing was as short and handy as old fashioned:

ClearAll @ filesInDir;
filesInDir[dirname_, deep_] :=  If[
DirectoryQ[dirname],
FileBaseName[ dirname] -> (
If[   deep == 1,
FileNameTake[#, -1],
filesInDir[#, deep - 1]
] & /@ FileNames["*", dirname]),
FileNameTake[dirname, -1]
];

filesInDir[$UserBaseDirectory, 2]  "Mathematica" -> {"ApplicationData" -> {"CCompilerDriver", "CloudObject", "CUDALink", "Parallel"}, "Applications" -> {...}, "Autoload" -> {...}, "FrontEnd" -> {"init.m"}, "Kernel" -> {"init.m"}, "Licensing" -> {}, "Paclets" -> {"Configuration", "Repository", "Temporary"}, "SystemFiles" -> {"CharacterEncodings", "FrontEnd", "Kernel", "SpellingDictionaries"}} And now, from this nested rules I want to go back to the output of: FileNameSplit /@ FileNames["*",$UserBaseDirectory, 2]


p.s.

I've chosen nested lists/associations to reduce ByteCount and make querying more user friendly. But if you can convince me I should keep a full paths list it would be a valid answer too.

• Kuba, sorry about the broken answer. I will try to fix it but I realize I am a bit tired to think clearly so it may not be today. May 15 '15 at 13:28
• @Mr.Wizard no worries :)
– Kuba
May 15 '15 at 13:29
• I am not sure I understand the format that you want. You have e.g. "Licensing" -> {} in your output list. I assume this is OK (desired?) for directories but not files, correct? May 15 '15 at 13:39
• @Mr.Wizard Yep, since it is an empty directory. But At the end the exact format does not matter if I can convert between forms easily and detect what is a file.
– Kuba
May 15 '15 at 13:42
• @Mr.Wizard The point is to have compact form of directory tree for user interface purposes and be able to convert it easily and use to create such directory tree in different directory for example. Or in general for functions like FileDelete, CopyFile etc.
– Kuba
May 15 '15 at 13:51

First off, a more compact form of filesInDir:

fileTree[f_, d_] /; DirectoryQ[f] && d > 0 :=
FileNameTake[f] -> (fileTree[#, d - 1] & /@ FileNames[___, f])
fileTree[f_, _] := FileNameTake[f]


Example usage:

fileTree[$UserBaseDirectory, 2] (* "Mathematica" -> {"ApplicationData" -> {"CCompilerDriver", "DeviceFramework", "ImageAcquisition", "Parallel"}, "Applications" -> {}, "Autoload" -> {"PacletManager"}, "FrontEnd" -> {"init.m"}, "Kernel" -> {"init.m"}, "Licensing" -> {}, "Paclets" -> {"Configuration", "Repository", "Temporary"}, "SystemFiles" -> {"CharacterEncodings", "FrontEnd", "Java", "Kernel", "LibraryResources", "SpellingDictionaries"}} *)  Personally I don't like the way that bottom-level directories are treated the same way as files (head String instead of head Rule). Here's an alternate method: Clear[fileTree] fileTree[f_, d_] /; DirectoryQ[f] := FileNameTake[f] -> If[d > 0, fileTree[#, d-1]& /@ FileNames[___, f], Missing["TooDeep"]] fileTree[f_, _] := FileNameTake[f]  Example usage: fileTree[$UserBaseDirectory, 2]

(* "Mathematica" -> {"ApplicationData" -> {"CCompilerDriver" ->
Missing["TooDeep"], "DeviceFramework" -> Missing["TooDeep"],
"ImageAcquisition" -> Missing["TooDeep"],
"Parallel" -> Missing["TooDeep"]}, "Applications" -> {},
"FrontEnd" -> {"init.m"}, "Kernel" -> {"init.m"}, "Licensing" -> {},
"Paclets" -> {"Configuration" -> Missing["TooDeep"],
"Repository" -> Missing["TooDeep"],
"Temporary" -> Missing["TooDeep"]},
"SystemFiles" -> {"CharacterEncodings" -> Missing["TooDeep"],
"FrontEnd" -> Missing["TooDeep"], "Java" -> Missing["TooDeep"],
"Kernel" -> Missing["TooDeep"],
"LibraryResources" -> Missing["TooDeep"],
"SpellingDictionaries" -> Missing["TooDeep"]}} *)


You can replace Missing["TooDeep"] with whatever you want, as long as it's not a List.

To convert a tree (either form) to your list format:

treeToList[d_ -> _Missing] := {{d}}
treeToList[s_String] := {{s}}
treeToList[d_ -> l_] := Prepend[Prepend[d] /@ Join @@ (treeToList /@ l), {d}]


Example usage:

tree = fileTree[$UserBaseDirectory, 2]; treeToList[tree] // Column (* { {"Mathematica"}, {"Mathematica", "ApplicationData"}, {"Mathematica", "ApplicationData", "CCompilerDriver"}, {"Mathematica", "ApplicationData", "DeviceFramework"}, {"Mathematica", "ApplicationData", "ImageAcquisition"}, {"Mathematica", "ApplicationData", "Parallel"}, {"Mathematica", "Applications"}, {"Mathematica", "Autoload"}, {"Mathematica", "Autoload", "PacletManager"}, {"Mathematica", "FrontEnd"}, {"Mathematica", "FrontEnd", "init.m"}, {"Mathematica", "Kernel"}, {"Mathematica", "Kernel", "init.m"}, {"Mathematica", "Licensing"}, {"Mathematica", "Paclets"}, {"Mathematica", "Paclets", "Configuration"}, {"Mathematica", "Paclets", "Repository"}, {"Mathematica", "Paclets", "Temporary"}, {"Mathematica", "SystemFiles"}, {"Mathematica", "SystemFiles", "CharacterEncodings"}, {"Mathematica", "SystemFiles", "FrontEnd"}, {"Mathematica", "SystemFiles", "Java"}, {"Mathematica", "SystemFiles", "Kernel"}, {"Mathematica", "SystemFiles", "LibraryResources"}, {"Mathematica", "SystemFiles", "SpellingDictionaries"} } *)  We can't get back to the full output of FileNameSplit /@ FileNames["*",$UserBaseDirectory, 2] since the base path is lost when converting to a tree.

Leonid Shifrin's implementation of a trie can be used for this purpose. It is available both for lists and for associations. For associations:

ClearAll[makeTreeAssoc];
makeTreeAssoc[wrds : {__String}] := Association@makeTreeAssoc[FileNameSplit /@ wrds];
makeTreeAssoc[wrds_ /; MemberQ[wrds, {}]] := makeTreeAssoc[DeleteCases[wrds, {}]];
makeTreeAssoc[wrds_] := Reap[If[# =!= {}, Sow[Rest[#], First@#]] & /@ wrds, _, #1 -> Association@makeTreeAssoc[#2] &][[2]]


Now you can do

tree[dir_] := makeTreeAssoc[FileNames["*", dir, 2]]
myTree = First@tree[$UserBaseDirectory]  Since the directory/file names are used as keys, I don't think there is a way to differentiate the files from the directories in the structure. You would have to check what it is upon retrieval. OK so now we have to be able to go back to that other format too... I ended up with this goBack[tree_, path_] := ( Sow[path]; If[Keys[tree] =!= {}, goBack[ tree[#], Append[path, #] ] & /@ Keys[tree]] ) len = Length@FileNameSplit[$UserBaseDirectory];
Select[
Reap[goBack[myTree, {}]][[2, 1]],
Length[#] >= len &
]


Update: much simpler now. If this works right I think I would consider it superior.

Update 2: I attempted to add specific handling of files and directories. Please tell me if this appears to work and produce a format that is acceptable to you.

fn[x : {___, {_, __}, ___}] := Normal @ GroupBy[x, First -> Rest, fn]
fn[x_List] := Join @@ x

files =
Append[FileNameSplit@#, FileType@#] & /@
FileNames["*", \$UserBaseDirectory, 2];

DeleteCases[fn @ files, Directory -> {}, -3] /.
{{Directory} -> {}, (x_ -> {File}) :> x}

• @Kuba I just updated my code to be simpler but I think that problem remains. I guess my file structure is different as I don't think I have files as keys in my test? I'll try to fix that now. May 15 '15 at 13:21
• @Kuba I just tried a different set of files and got errors, so I didn't test this well enough before posting. Sorry. I guess I'll delete this while I work on it. May 15 '15 at 13:23

Try this recursive query which is linear-time in input (timing study here)

trie = Query[GroupBy@First, If[First@# == {}, {}, trie@#] &, Rest];


Variations on the above are possible and useful (see Antonov's Trie page here developed in 2013 but without Dataset).

Assume filenames have been split into path component lists, eg "a/b/c" --> {"a","b","c"} :

flatFiles = FileNames["*.gsheet", "~/Google Drive/...", Infinity] // Dataset

Then:

flatFiles[All, StringSplit[#, "/"] &][trie]