15
$\begingroup$

I have several thousands of small .jpg files (30kb each) which I need to import into Mathematica. (I plan to use ImageDistance to investigate their differences.) I tried several ways to import these files, but all of them seem very slow. So I wonder if I have been doing something wrong?

Here is what I tried:

files = FileNames["*.jpg", path];
images= (Import[#,"JPEG"]&)/@files;

The above takes several minutes to import about 30mb worth of files (about 2 thousand images). Alternatively, I tried:

Dynamic[{i, Length[files]}]
Do[
 imgs[i] = Import[files[[i]], "JPEG"];
 , {i, 1, Length[files]}]

This has the advantage that it shows the progress during the import process. Still, it imports about 10 files per second. This means about 300kb worth of data per second. Clearly, this rate is ridiculously low. That is why I am pretty sure that I am doing something wrong here. What should I do to import the files faster?

Improve this question
$\endgroup$
6
  • 1
    $\begingroup$ the import itself is the bottleneck, the way you structure the code wont do much. All you can do is try parallel operations, (ParallelMap , ParallelDo ) $\endgroup$
    – george2079
    Mar 25, 2017 at 16:12
  • 2
    $\begingroup$ There is another thing you can do: implement your own JPEG importer through LibraryLink. Start here. This will be a lot of work, and only worth it if you plan to keep importing large image datasets several times every day for the forthcoming weeks. $\endgroup$
    – Szabolcs
    Mar 25, 2017 at 16:15
  • $\begingroup$ @george2079 I notice that depending on the number of Kernels I launch, the ParallelDo function works faster or slower. Maybe you have a suggestion for some other option to set in order to improve parallel computing performance in this case? $\endgroup$
    – Kagaratsch
    Mar 25, 2017 at 16:37
  • 1
    $\begingroup$ Please see my update. $\endgroup$
    – Szabolcs
    Mar 25, 2017 at 16:59
  • 1
    $\begingroup$ ultimately you are limited by your disk access performance, so you will loose if you try to do too many parallel read operations. $\endgroup$
    – george2079
    Mar 26, 2017 at 15:39

1 Answer 1

Reset to default
22
$\begingroup$

Reading the file as

Import["file.jpeg", "ImageNoExif"]

is noticeably faster on my machine than the default

Import["file.jpeg", "Image"]

Benchmark in 11.1:

Import["~/Desktop/Untitled.jpg", "Image"]; // RepeatedTiming
(* {0.054, Null} *)

Import["~/Desktop/Untitled.jpg", "ImageNoExif"]; // RepeatedTiming
(* {0.025, Null} *)

This is not a full solution, but hopefully it will make things a bit better.

Update

My guess was that the slow part of Import is the one written in Mathematica. With enough spelunking, we might be able to find an underlying fast function that we can use.

Start by importing one image to trigger auto-loading the functions that we need.

file = "~/Desktop/Untitled.jpg";
Import[file]

Now you can do this:

First@Image`ImportExportDump`ImageReadJPEG[
    AbsoluteFileName[file]
]; // RepeatedTiming

(* {0.00070, Null} *)

This is very fast. But it seems to be necessary to pass an absolute file name (or at least one not containing ~). I believe this should be fast enough for your purposes.

Warning: This solution uses undocumented functions, and may not work in future versions. It does seem to work in 9.0.1–11.1.0, which suggests that it is a relatively safe to use, stable function. You should also know that such low-level undocumented functions will sometimes crash the kernel when they are called with incorrect arguments. Since we have to guess at the correct arguments, the risk is high. However, I have not encoutered any crashes or bad effects while experimenting with ImageReadJPEG.

Improve this answer
$\endgroup$
9
  • 2
    $\begingroup$ @Kagaratsch I told you to implement your own reader with LibraryLink, and it will be fast. But it is a reasonable guess that Wolfram did the same, and the slow part of the import is the evaluation of Mathematica code. So we must peel pack Import layer by layer until we find a fast low-level function. Go to SystemFiles/Formats/JPEG/Import.m and find the name of the function JPEG import uses. Then spelunk it, and follow the breadcrumbs until you find what you need. $\endgroup$
    – Szabolcs
    Mar 25, 2017 at 17:08
  • 2
    $\begingroup$ @Kagaratsch Another approach would be to read one image (to trigger auto-loading, as usual), then evaluate ?*`*JPEG* and try to guess which function from the output is the one you need. Then guess at what arguments we can pass to it. With the step-by-step spelunking approach we can see how the function is called by other code, so there is less guesswork involved. But detailed spelunking takes more time. $\endgroup$
    – Szabolcs
    Mar 25, 2017 at 17:11
  • 1
    $\begingroup$ Cool approach! I gave you a thumbs up mot only for your unique way of finding the method, but also for the usage of the verb 'to spelunk'. $\endgroup$
    – Ben
    Mar 25, 2017 at 19:17
  • 4
    $\begingroup$ Image`ImportExportDump`ImageReadJPEG exists and works as far back as version 9(.0.1, that I tested). So I think it is a reasonable bet that it will stay around for a few versions more. $\endgroup$
    – Oleksandr R.
    Mar 25, 2017 at 19:31
  • 1
    $\begingroup$ FWIW the dramatic speedup with ImageReadJPEG seems to go away if you import large images ( Still worthwhile, but only a factor of two for a ~1Mb image) $\endgroup$
    – george2079
    Mar 27, 2017 at 14:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.