# Slow import of multigigabyte TIF image stacks

Update 6: After contacting Wolfram Technical Support, I was advised that the frame-to-frame import timing variability can be eliminated by reading in the frame stack from frame to frame with the commands:

data[[k]] = Import["multiframe.tiff", {"ImageList", k}]


or

data[[k]] = Import["multiframe.tiff", {"GraphicsList", k}]


My response was:

If we write a simple script like this:

StackDepth = Import["...\\data.tif", "ImageCount"]

data = Array[0 &, StackDepth];
TimingArray = Array[0 &, StackDepth];

For[i = 1, i <= StackDepth, i++,
FrameStartTime = AbsoluteTime[];
data[[i]] = Import["...\\data.tif", {"GraphicsList", i}];

FrameEndTime = AbsoluteTime[];
TimeArray[[i]] = FrameEndTime - FrameStartTime;

Print[i, "   ", TimeArray[[i]]];
];


For a 10,000 frame stack, where individual frames have dimensions {256,256}, we have a ~0.85 second per frame import time (implying that it will take 10^4*0.85 seconds ~ 2.5 hours to load in all of the frames. So the loading times are actually longer than just using data[[i]] = Import["...\data.tif"];.

Also, if we try a 5,000 frame stack, we have a ~0.5 second / frame loading time.

Is there a way around this or a patch?

I noticed that it takes half an hour or so for me to import a $\approx 1.86$ GB TIF file consisting of exactly 15,000 images with 256 x 256 pixels (via the Import[] command), whereas it takes only a few seconds to load the same file in a program like ImageJ. My machine has 128 gigs of ram, and a fairly modern Xeon processor, so I doubt that's the issue. I'm hoping to understand why this is happening, and what, if anything, I can do about it.

Because of the device I'm using to capture the images, it would be very helpful to import files in this TIF format. That said, it doesn't seem to help to try other image formats like PNG.

For an example, I've uploaded the exact TIF file I'm using, which is again a stack with exactly 15,000 images, each consisting of a {256, 256} pixel lattice, to the following dropbox folder: https://www.dropbox.com/s/1xdztnk4iykjabb/ExampleTIF.tif (to retrieve the file, click "download" in the upper-right-hand corner).

The first {256, 256} frame of the TIF file ($\approx 128$ kb in size) can be found here: https://www.dropbox.com/s/o85s74rdi33hq76/frame1.tif (unfortunately, you have to download the image to see it, as the preview appears to be broken).

I have clocked that it takes 1528.1894075 seconds (about 25 minutes and 28 seconds) to load the full TIF stack with the command:

data = Import["E:\\data\\ExampleTIF.tif"];


The loading time I observe for the single frame appears to be $\approx 24$ milliseconds, which implies that the full stack should load in $24$ milliseconds $\times 15,000$ frames $\approx 6$ minutes. The $24$ millisecond loading time per frame seems excessive in and of itself without accounting for the factor of 4-5x per-frame slowdown when loading the full stack. Why is ImageJ, LabVIEW, etc. so much faster than Mathematica? I can load the full stack in less than 30 seconds with either of these programs.

I previously asked this question here: Speeding up TIF import without an explicit example. Unfortunately I lost my account log-in information, and was unable to update the old question.

Update 5: In the comments section, Simon Woods wrote:

"To see what Import is actually doing with those functions you could do a trace like this: With[{syms = Alternatives @@ Symbol /@ Names["*TIFF"]}, Trace[Import["frame1.tif"], syms, TraceInternal -> True]]"

Here's the result of that trace analysis:

{{{{{{{{{{{{{{{{{{{{{{{{{{{{{ImageImportExportDumpImageFrameCountTIFF["E:\\data\\ExampleTIF.tif"], 15000}}},{{{ImageImportExportDumpImageReadTIFF["E:\\data\\ExampleTIF.tif"], {IMAGES FROM TIFF STACK HERE}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}


As can be clearly seen, ImageImportExportDumpImageReadTIFF, which is the same function I've analyzed the behavior of below, is being used for the TIFF stack import. ImageImportExportDumpImageReadTiledTIFF[] & ImageImportExportDumpImageReadTileTIFF[] remain mystery functions.

Update 4: There appears to be some metadata for my TIFF stack. Running the command:

Import["E:\\data\\ExampleTIF.tif", "Table"]


Returns:

{{"II*\.00ü", "\.02\.00Andor",
"SOLIS\.00\.00\.00\.00\.00\.00\.00\.00\.00\.00\.00\.00\.00\.00\.00\
\.00\.00\.00\.00\.00\.00t\.0e\.0fB"}, {""}, {"l"}, {"i"}, {"å\.0eÕ
\.08"}, {"P\.10%\.0e0\.11\.10I\.0f\.06\.11L\[RawEscape]42O<\
\[Divide])»\.19Â\.0f
Í\.10"}, {"æ
½"}, {""}, {"T"}, {"õ\.11g"}, {"\.0e1\.0f\.02\.11Þ\.10Y\.10â\.11\
\.1c\.11]\.15l\.1dG44É%s\.15
\.12L\.0f"}, {"\.10õ"}, {""}, {"­\.0f>\.0e\.1d\.12Ç
Å\.10ç\.0e\.1c\.0fç\.0fý\.0eR\.0f\.0ea\.0e\.0b\.0b\.06\.0e"}, {}, \
{"3\.0eÆ\.0f\.0f
î\.0b\[Paragraph]\.0eb\.0e¹
%\.0fÏ\.0eû\.0fA\.0e\.b4

»\.0f\.10\.12\.11X\.0ei\.10ñ\.10ò\.0eÊ\.0f\.15\.0fk\.10=\.10\.0e¡\
\.11R
A\.0bä\.0eL\.0f¹\.0fØ\.10â\.0e\[PlusMinus]\.0e\.13\.0f~
æ
"}, {"¢
Ü\.0f9"}, {"W\.0e|
t"}, {"Æ
\.14\.0f"}, {"ê\.11{\.11º\.10\.0e¨\.0fõ
\.0fT"}, {"\.07"}, {"É"}, {"\.10¼
\.1f\.0f\.0eì\.13R\.10Å\.10Q\.11B\.0fÁ"}, {"\.10\.08"}, {"l\.10Ô\
\.12j"}, {".\.0eü\.0eÓ\.0fð\.0f\.18\.0f"}, {"\.0f­\.10Þ"}, {"é"}, \
{"I"}, {"0
\.0ea\.0b\.0bÍ\.0bj"}, {"%\.0eB\.10=\.0b[
\[Paragraph]\.0e\.16\.0b3"}, {}, {"\.0b\[Micro]"}, {"Ö\.0f\""}, {"p\
\.0fú"}, {"w
\.0fB\.12E"}, {"Å
Þ"}, {"\.04\.0e¨\.0fÃ\.0b)\.0fÁ\.0eÆ"}, {"Ñ"}, {"m\.0f \.0fû\.10:\
\.0f("}, {"x\.0f\.06\.0f¾\.10¨\.10(\.10½"}, {"ø\.0bW\.0b<

U"}, {"I"}, {"T\.10\.1c\.0el\.11ò

¿
\.10ä\.0e\[Paragraph]"}, {"N\.0f\.02\.0f\.13\.0eÛ\.0eã
\[Degree]
\.05"}, {"H\.0e\.0fô
"}, {"\.0f5
0\.0b«\.0e«\.0f.
§\.0f7\.10+\.0e"}}


Most of this looks like junk, but you can see the name of the camera used to collect the data as well as some optical setup specifications. Perhaps there's a way to strip this data out and prevent whatever trigger is causing Mathematica to repeatedly scan through the image stack?

Update 3: I've updated the second figure to show that the import time for a fixed set of $N = 100$ frames exhibits reliable linear scaling with the depth of the first frame, $k$, in the image stack.

ImageImportExportDumpImageReadTIFF[]


Is clearly the wrong tool to use to import a large stack of TIF files. I also think that, after some handwringing (leading to a delay in importing a single frame), this is probably the function that Import[] calls when passed the ExampleTIF.tif stack.

However, I haven't been able to get:

ImageImportExportDumpImageReadTiledTIFF[]
ImageImportExportDumpImageReadTileTIFF[]
ImageImportExportDumpImageReadStripedTIFF[] (striped?)


To work at all. Perhaps one of these functions represents a way forward? That said, I can't even set of arguments that these functions need.

Update 2:

Ok, let's do some experiments to help elucidate the cause underlying slow loading times for large TIF stacks. First, based on Simon Wood's observation, notice that we can use:

ImageImportExportDumpImageFrameCountTIFF["E:\\data\\ExampleTIF.tif"]


To return an integer value corresponding to the number of frames in the TIF image stack. We can then use:

ImageImportExportDumpImageReadTIFF["E:\\data\\ExampleTIF.tif", Table[i, {i, StartFrame, EndFrame}]]


To return the set of frames including and between specifiable StartFrame and EndFrame integer variables.

First we notice that, setting StartFrame = 1, linearly increasing the number of frames we want to import (by linearly increasing EndFrame) requires what appears to be a quadratically scaling amount of time.

Secondly, we can notice that attempting to import a fixed length stack of 100 frames has a time-cost that scales with the depth of the 100 frames in the stack. In other words, for some {StartFrame, EndFrame} pair, the frame set including and between {StartFrame+k,EndFrame+k} appears to take a longer time to load, and this delay appears to linearly scale with $k$.

So the trouble, I'm guessing, lies with properly indexing the TIF image stack. Is there any clever way to fix this?

Update 1: Following up on Simon Wood's comment, entering ?*TIFF in Mathematica v9.0 gives a set of functions, one of them being ImageReadTIFF. However:

image = ImageImageReadTIFF["E:\\data\\frame1.tif", All];


just returns:

ImageImageReadTIFF["E:\\data\\frame1.tif", All]


Is there any obvious mistake on my part?

-
If I was in your situation, I'd upload a smaller file as well - say 3 or 4 images - so that people could see whether it was a file/format issue or just a scaling issue. (I'd like to help but, like many here, I have less than 10% of the RAM that you have...) – cormullion Jul 13 '13 at 9:28
@cormullion That makes a lot of sense. I've provided an additional link to the first {256, 256} frame of the TIF stack. The file size is $\approx 128$ kb. – Sparse Pine Jul 13 '13 at 9:36
Yes, DropBox only previews smaller images, I seem to recall. Your image imports OK (just one frame?). I think the problem might be that your other software apps probably load pointers to individual frames rather than loading all the data at once. Mathematica on the other hand likes to have everything in memory (and only recently started introducing file-based image processing such as ImageFileScan). But I might be completely wrong...:) – cormullion Jul 13 '13 at 9:48
On version 8 I can use ImageImageReadTIFF[filename, All] to load the single frame file about 120 times faster than Import does. I don't know if there are any nasty side effects or if it will work on the large file, but it might be worth trying. – Simon Woods Jul 13 '13 at 11:30
@halirutan provided this answer to integrating imagej with Mathematica. If imagej is indeed faster, perhaps letting it do the work for M is the right way to go. – bobthechemist Jul 13 '13 at 12:41