Regular Expressions in FileNameJoin[]

Question

I'm attempting to create a list of folders that match a particular pattern.

For example, consider I have the following files in a directory structure:

a/d.txt
b/f.txt
c/g.txt

Now I want to get a list of all of the files that are in folders a/ and c/. Since I sometimes change work environments, it seemed best to do this with FileNameJoin. I use this in combination with FileNames to match directories in my structure. I'm pleased with the results when I want to take all folders that match a pattern, because I can use a "*" string to encompass all folders. I can use the command

FileNames[FileNameJoin[{"*","*.txt"}]]

However, when I want to pick a few folders the process becomes more complex.

I would expect something like

FileNames[FileNameJoin[{RegularExpression["[a,c]",RegularExpression["*\\.txt"]}]]

to work, but it doesn't. Using StringExpression instead of RegularExpression gives the same error. The issue is that while FileNames does accept regular expressions as arguments, FileNameJoin does not. My understanding is that FileNameJoin must output a string as its result, and StringExpression and RegularExpression are just not strings. I think that the "*" character works because it is a "abbreviated string pattern" as mentioned in the documentation for FileNames, but I can't find further documentation of what other characters might be usable for abbreviated string patterns.

I have workarounds that I've come up with that I'll post as an answer, but I'm curious if anyone has documentation for abbreviated string patterns (is a comprehensive list shown here?) that might be helpful or a more streamlined solution.

Answer 1

Specifying folders for FileNames

You can simply provide the list of directories as the second argument to FileNames:

subFolders = {"subfolder", FileNameJoin[{"subfolder", "subsubfolder"}]};
parentFolders = {ParentDirectory[], ParentDirectory[2]}
filesList1 = FileNames["*", subFolders]
filesList2 = FileNames["*", parentFolders]

You can use patterns in the folder path specification:

patternSubFolders = {"Wolfram*", "d" ~~ __};
filesList1 = FileNames["*", patternSubFolders]

{"data\\Untitled-3.nb", "Wolfram Mathematica\\test.nb", "Wolfram\\Video"}

Note that FileNames and CreateDirectory also allow specifying the parent directory and the current directory using the .. and . syntax respectively (but this functionality is undocumented, so I wouldn't recommend relying on it):

parentFolders = {"..", FileNameJoin[{"..", ".."}]};

They also allow specifying that the path starts from root of the drive by starting the path with either / or \\ (on Windows):

CreateDirectory["/test1"]
CreateDirectory["\\test2"]
FileNames["/test@"]

"C:\\test1"

"C:\\test2"

{"\\test1", "\\test2"}

---

More info on abbreviated string patterns

I'm curious if anyone has documentation for abbreviated string patterns (is a comprehensive list shown here?)

This is a link to the Documentation chapter devoted specifically to abbreviated string patterns. They provide very limited functionality and there is only a limited number of (quite old) built-in functions which support them. There are only two metacharacters that are supported by abbreviated string patterns. Here they are along with their escaped forms:

abbreviated string pattern	meaning
*	zero or more characters
@	one or more characters excluding uppercase letters
\\*	literal *
\\@	literal @

Wrapping a pattern string by Verbatim disables abbreviated string patterns.

It is not mentioned in the Documentation, but worth knowing that if the function supports abbreviated string patterns, they work also inside of StringExpression:

StringMatchQ[{"abc", "bcd", "cde"}, StringExpression[("a" | "b"), "*"]]

{True, True, False}

StringCases["abcd", "a*"]
StringCases["abcd", StringExpression[("a" | "b"), "*"]]

{}

{}

StringReplace[{"abc", "bcd", "cde"}, "a*" -> "X"]

{"abc", "bcd", "cde"}

StringReplace[{"abc", "bcd", "cde"}, StringExpression[("a" | "b"), "*"] -> "X"]

{"abc", "bcd", "cde"}

(StringCases and StringReplace don't support abbreviated string patterns.)

Also, you should realize that inside of RegularExpression these metacharacters have an entirely different meaning.

Note also that \\ is used to represent literal backslash , and \" is used to represent literal double quote " inside of a general String.

As you can notice from the above, there are some subtleties in interpreting the meaning of \\ in abbreviated string patterns that you need to pay attention to:

With[{sm = HoldForm[StringMatchQ[##]] -> StringMatchQ[##] &}, Style[Column[{sm["@", "\\@"],
    sm["@", Verbatim["\\@"]],
    sm["\\@", Verbatim["\\@"]],
    sm["\\@", "\\@"],
    sm["\\@", "\\\\@"]}], "Input", ShowStringCharacters -> True]]

---

FileNameJoin is not supposed to be used with patterns

My understanding is that FileNameJoin must output a string as its result, and StringExpression and RegularExpression are just not strings. I think that the "*" character works because it is a "abbreviated string pattern" as mentioned in the documentation for FileNames, but I can't find further documentation of what other characters might be usable for abbreviated string patterns.

Actually, your use of FileNameJoin for composing an abbreviated string pattern is a kind of a hack because FileNameJoin is not designed to work with patterns.

Perhaps a better solution would be to select all folders with "*", then use StringCases to pick all structures that match the desired pattern.

Yes, some experiments of mine show that this way is the most reliable when working with FileNames, see:

• Efficient way to list only files (but not directories) in subdirectories

You can first split the file name with FileNameSplit, then apply Cases with Condition (/;) containing StringMatchQ, or use SringCases with StringExpression to select the paths you are interested in.

Answer 2

Some Examples:

All files with extensions txt or mp3 in folder a or b:

folders = {"a", "c"};
files= "*.txt" | "*.mp3";
FileNames[files,folders]

All files whose names start with a in all subfolders of dir:

folders = "dir/*";
files= "a*";
FileNames[files,folders]

All files whose name constains "abc":

folders = "dir";
files= "*abc*";
FileNames[files,folders]

Answer 3

My solution is to make a list of the folders I want to include, then Map the FileNames[FileNameJoin[]] function onto those folders:

folders = {"a", "c"};
filesList = 
  FileNames[
     FileNameJoin[{#, "*.txt"}]] & /@ folders;
files = Catenate[filesList]

This doesn't utilize RegularExpressions though. Perhaps a better solution would be to select all folders with "*", then use StringCases to pick all structures that match the desired pattern.