Take the 2-minute tour ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

Is there a way to split an external file using Mathematica? (Other than importing it, dividing it internally and exporting the parts)

For instance like some File Split utility do.

share|improve this question
1  
You can use Run to run another program from inside Mathematica. –  b.gatessucks Jan 26 '13 at 9:14
1  
Do you imagine that file utilities don't do the equivalent of importing the file data and then exporting it? –  m_goldberg Jan 26 '13 at 13:07

2 Answers 2

Sure, but Mathematica is probably the wrong tool for the job. I'll assume that by "Other than importing it, dividing it internally and exporting the parts" you mean importing the entire file, as of course something must be imported and exported.

For example, to split the first 100,000 bytes to one file and the next to another you could write:

in = "bigfile.dat";
out1 = "split1.bin";
out2 = "split2.bin";

stream = OpenRead[in, BinaryFormat -> True];

Do[
 BinaryWrite[out1, BinaryRead@stream],
 {100000}
]

Do[
 BinaryWrite[out2, BinaryRead@stream],
 {100000}
]

Don't forget to Close your files when you're done:

Close /@ {in, out1, out2};

It will be faster if you work with chunks rather than single bytes, for example, writing 500,000 bytes in chunks of 100,000:

stream = OpenRead[in, BinaryFormat -> True];

Do[
 BinaryWrite[out1,
  BinaryReadList[stream, "Byte", 100000]],
 {5}
]

If your files are not binary you could use the standard types of Read to get the parts you want in the chunk size that works for your system.

share|improve this answer
    
OK. . .I think the BinaryRead and BinaryWrite cant take the number as in your second code unit. . .But anyway, I think it cant be done at all in a reasonable time for say a 10GB file. –  Hp Radojewski Schäfer Von Jan 27 '13 at 11:35
    
@HpRad Sorry, that was an error: it should have been BinaryReadList. As I said Mathematica probably isn't your best choice for this, but nevertheless with a larger block (and BinaryReadList of course) should still be serviceable for limited use. –  Mr.Wizard Jan 27 '13 at 18:03

From your other "question" about the same topic which was closed it becomes clear that you expect Mathematica here to do something at a very low level which it simply doesn't provide (manipulate file-system entries without moving data).

I think it's also not something that anyone would expect from a high level and multi-platform tool that Mathematica is. While you might be correct that this operation could in principle be done on the filesystem level almost immediatly with almost no actual moving of data that would mean to implement something that would not only depend on the platform but also on what the filesystem in use does offer. I'm not an expert in file system implementations but suspect that some of them could well be more complicated than your simplified abstraction implies and the whole procedure isn't even possible in many cases. If such functionality does exist at all it certainly would mean to make use of file system specific programs or APIs and that's what you should search for. If you find something for the OS/filesystem you're using, then use Run to call such a program or one of the existing connection methods (NETLink, JLink, LibraryLink) to access such an API.

share|improve this answer
    
Here is what your Windows operation system offers: . SetEndOfFile, SetFileValidData to truncate and extend a file. . This would be a good start. . .You see in times where all data grows larger and larger, even Mathematica needs the tools to take hinside to this fact. –  Hp Radojewski Schäfer Von Feb 1 '13 at 8:07
    
@HpRadojewskiSchäferVon: if that's your opinion then try and make that a feature request for future versions. I doubt that WRI will implement that -- the answer to larger and larger datasets seems not to be better handling of huge sequential accessed files, IMHO... This is just a question/answer site, I can only suggest to use functionality that's there. If you are on windows and need that functionality, try whether you can make use of SetEndOfFile and SetFileValidData via NETLink. –  Albert Retey Feb 1 '13 at 9:08

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.