Cleanest way to copy part of one directory to another

Question

I often want to copy a directory and clean it up for deployment purposes. This may involve pruning out large datasets, cleaning up private junk, etc. Doesn't really matter.

The issue I often run into is that this is very, very slow, and very, very wasteful when the things I want to prune out contribute the vast majority of my directory size (e.g. a .git folder or test data).

A better approach, therefore, would be to only copy the relevant parts of the directory over, but I'm not sure what the best way to do this would be. The naive way would just be to call CopyFile on every single file I want to copy, but that seems rather wasteful as CopyFile can't operate in batch.

My thought was to basically do the following:

• Figure out my total set files in the dir (can be slow, but only like .2s slow on every very large dirs)
• Determine which of these files I truly want to copy (if the method can be done without step one, jump here)
• Figure out the total byte size of all the complement of these. If huge, use a method based on CopyFile and CopyDirectory. If small, use one CopyDirectory and prune with DeleteFile and DeleteDirectory.

There's also a question of how to optimize the number of CopyFile/DeleteFile and CopyDirectory/DeleteDirectory calls to make, but I think I can answer that one by determining which folders are copied/deleted wholesale and which ones have only portions copied/deleted.

Honestly, though, it seems like this should be easier that it appears to be. Does it have to be so annoying?

Answer 1

Here's how I've been doing it:

(* ::Section:: *)
(*FileTools*)
BeginPackage["FileSystemTools`"]
PartialDirectoryCopy::usage="";
CopyDirectoryFiles::usage="";
PruneDirectoryFiles::usage="";
Begin["`Private`"];
(* ::Subsubsection::Closed:: *)
(*PartialDirectoryCopy*)
Options[PartialDirectoryCopy]=
  {
    "RemovePaths"->{},
    "RemovePatterns"->{},
    "ModeSwitchByteCount"->5*10^6
    };
PartialDirectoryCopy[src_, targ_, ops:OptionsPattern[]]:=
  Module[
    {
      rempaths=
        Select[
          Flatten@{OptionValue["RemovePaths"]},
          StringPattern`StringPatternQ
          ],
      rempatts=
        Select[
          Flatten@{OptionValue["RemovePatterns"]},
          StringPattern`StringPatternQ
          ],
      fullFNames,
      fileBytesTotal,
      remFiles,
      restFiles
      },
    If[Length@Join[rempaths, rempatts]>0,
      fullFNames=FileNames["*", src, \[Infinity]];
      remFiles=
        Join[
          FileNames[rempaths, src],
          FileNames[rempatts, src, \[Infinity]]
          ];
      restFiles=
        Select[
          Complement[fullFNames, remFiles],
          Not@*StringStartsQ[Alternatives@@remFiles]
          ];
      fileBytesTotal=
        Total[FileByteCount/@Select[remFiles, Not@*DirectoryQ]];
      Quiet@DeleteDirectory[targ, DeleteContents->True];
      If[TrueQ[fileBytesTotal>OptionValue["ModeSwitchByteCount"]],
        CopyDirectoryFiles[src, targ, 
          getMinimalFileModSpec[restFiles, fullFNames]
          ],
        CopyDirectory[src, targ];
        PruneDirectoryFiles[targ, 
          StringTrim[
            getMinimalFileModSpec[remFiles, fullFNames, False],
            src
            ]
          ]
        ],
      If[True(*OptionValue@OverwriteTarget//TrueQ*),
        Quiet@DeleteDirectory[targ, DeleteContents->True];
        ];
      CopyDirectory[src, targ]
      ]
    ];
(* ::Subsubsection::Closed:: *)
(*getMinimalFileModSpec*)
getMinimalFileModSpec//Clear
getMinimalFileModSpec[
  restFiles_, 
  files_,
  pruneEmpties:True|False:True
  ]:=
  Module[
    {
      g1,
      g2,
      unchangedReduction,
      changedReduction,
      containedReduction,
      keys,
      changedKeys,
      missingDirs,
      baseSpec,
      deadDirs
      },
    (* 
            build associations mapping directories to contained files
            all the directories are keys in the Associations optimally 
            *)
    g1=Select[Not@*DirectoryQ]/@GroupBy[restFiles, DirectoryName];
    deadDirs=Complement[Select[restFiles, DirectoryQ], Keys@g1];
    g2=Select[Not@*DirectoryQ]/@GroupBy[files, DirectoryName];
    (* figures out which directories may be copied across wholesale *)
    unchangedReduction=
      AssociationMap[
        #[[1]]->
          If[
            !ListQ@g2[#[[1]]]||
              Length@Complement[Flatten@{g2[#[[1]]]}, #[[2]]]==0,
            #[[1]],
            #[[2]]
            ]&,
        g1
        ];
    (* find the directories that are in the full set but missing in the reduced set to prevent over reduction *)
    missingDirs=
      AssociationThread[
        Complement[Keys@g2, Keys@g1],
        0
        ];
    containedReduction=
      FixedPoint[
        KeySelect[
          (* 
                    checks if both the child *and* the parent are unchanged *and* 
                        if there's nothing missing vis-a-vis the original 
                    *)
          !StringQ@unchangedReduction[#]||
            !StringQ@unchangedReduction[DirectoryName[#]]||
            KeyExistsQ[missingDirs, DirectoryName[#]]||
            AnyTrue[Keys@missingDirs, StringStartsQ[DirectoryName[#]]]&
          ],
        unchangedReduction
        ];
    (* figure out which parent directories have changed *)
    keys=Keys@containedReduction;
    changedKeys=
      Select[keys, !AnyTrue[keys, StringMatchQ[#~~__]]&];
    baseSpec=
      Flatten@Values@
          KeyDrop[containedReduction, changedKeys];
    If[pruneEmpties,
      (* makes sure we're not pulling directories with no stuff to copy *)
      Select[
        !DirectoryQ[#]||
          Length@g1[#]>0||
          AnyTrue[
            Flatten@Values@KeySelect[g1, StringStartsQ[#]], 
            !DirectoryQ
            ]&
        ],
      Union[#, deadDirs]&
      ]@baseSpec
    ]
(* ::Subsubsection::Closed:: *)
(*CopyDirectoryFiles*)
CopyDirectoryFiles[src_, targ_, files_]:=
  (
    MapThread[
      Which[
        DirectoryQ@#, 
          If[!DirectoryQ@DirectoryName[#2], 
            CreateDirectory[#2, CreateIntermediateDirectories->True]
            ];
          CopyDirectory[#, #2],
        FileExistsQ@#,
          If[!DirectoryQ@DirectoryName[#2], 
            CreateDirectory[DirectoryName[#2], CreateIntermediateDirectories->True]
            ];
          CopyFile[#, #2, OverwriteTarget->True]
        ]&,
      {
        files,
        Map[FileNameJoin@{targ, #}&, StringTrim[files, src]]
        }
      ];
      )
(* ::Subsubsection::Closed:: *)
(*PruneDirectoryFiles*)
PruneDirectoryFiles[targ_, files_]:=
  (
    Which[
      DirectoryQ@#, 
        DeleteDirectory[#, DeleteContents->True],
      FileExistsQ@#,
        DeleteFile[#]
      ]&/@Map[FileNameJoin@{targ, #}&, files];
    )
End[];
EndPackage[]

The real meat of all this is getMinimalFileModSpec which figures out what the minimal number of operations I need to do is.

It also performs pretty well:

PartialDirectoryCopy[
   FileNameJoin@{$InstallationDirectory, "SystemFiles", "FrontEnd"},
   "~/Desktop/test",
   "RemovePatterns" -> "*.nb"
   ] // AbsoluteTiming // First

1.59635

FileNames["*.nb", "~/Desktop/test", ∞] // Length

0

CopyDirectory[
   FileNameJoin@{$InstallationDirectory, "SystemFiles", "FrontEnd"},
   "~/Desktop/test2"
   ] // AbsoluteTiming // First

45.2356

FileNames["*.nb", "~/Desktop/test2", ∞] // Length

220

Anecdotally, this cut the build time for my paclets down from ~30 seconds to ~1 second.