4
$\begingroup$

Revised newly posed question: Now, After some "investigation" I came to some harder evidence of unexpected memory "leaks" when processing, importing and exporting images, than I previously thought in first instance of question (below). So, I made test by code:

qN = 50;
q1 = q2 = q3 = q4 = q5 = q6 = q7 = q8 = q9 = ConstantArray[0, qN];
Print["------------- GIF -> JPG  NO extra processing -------------"];
Do[
  Export["test.ext", Import["http://i.imgur.com/u5OaSfs.gif"], "JPG"];
  q1[[i]] = MemoryInUse[], {i, 1, qN}
  ];
q1 = (# - Min[#]) &@q1;
Print[N[Max[q1]/qN/2^20], " MB/element \n", "Total= ", 
 N[Max[q1]/2^20], "  MB"]
Print["------------- GIF -> JPG  WITH extra processing -------------"];
Do[
  Export["test.ext", 
   ColorNegate@
    LocalAdaptiveBinarize[Import["http://i.imgur.com/u5OaSfs.gif"], 
     25, {0.8, 0., 0.034}, PerformanceGoal -> "Speed"], "JPG"];
  q2[[i]] = MemoryInUse[], {i, 1, qN}
  ];
q2 = (# - Min[#]) &@q2;
Print[N[Max[q2]/qN/2^20], " MB/element \n", "Total= ", 
 N[Max[q2]/2^20], "  MB"]
Print["------------- GIF -> GIF  NO extra processing -------------"];
Do[
  Export["test.ext", Import["http://i.imgur.com/u5OaSfs.gif"], "GIF"];
  q3[[i]] = MemoryInUse[], {i, 1, qN}
  ];
q3 = (# - Min[#]) &@q3;
Print[N[Max[q3]/qN/2^20], " MB/element \n", "Total= ", 
 N[Max[q3]/2^20], "  MB"]
Print["------------- GIF -> GIF  WITH extra processing -------------"];
Do[
  Export["test.ext", 
   ColorNegate@
    LocalAdaptiveBinarize[Import["http://i.imgur.com/u5OaSfs.gif"], 
     25, {0.8, 0., 0.034}, PerformanceGoal -> "Speed"], "GIF"];
  q4[[i]] = MemoryInUse[], {i, 1, qN}
  ];
q4 = (# - Min[#]) &@q4;
Print[N[Max[q4]/qN/2^20], " MB/element \n", "Total= ", 
 N[Max[q4]/2^20], "  MB"]
Print["------------- JPG -> JPG  NO extra processing -------------"];
Do[
  Export["test.ext", Import["http://i.imgur.com/ZyUf71d.jpg"], "JPG"];
  q5[[i]] = MemoryInUse[], {i, 1, qN}
  ];
q5 = (# - Min[#]) &@q5;
Print[N[Max[q5]/qN/2^20], " MB/element \n", "Total= ", 
 N[Max[q5]/2^20], "  MB"]
Print["------------- JPG -> JPG  WITH extra processing -------------"];
Do[
  Export["test.ext", 
   ColorNegate@
    LocalAdaptiveBinarize[Import["http://i.imgur.com/ZyUf71d.jpg"], 
     25, {0.8, 0., 0.034}, PerformanceGoal -> "Speed"], "JPG"];
  q6[[i]] = MemoryInUse[], {i, 1, qN}
  ];
q6 = (# - Min[#]) &@q6;
Print[N[Max[q6]/qN/2^20], " MB/element \n", "Total= ", 
 N[Max[q6]/2^20], "  MB"]
Print["------------- JPG -> GIF  NO extra processing -------------"];
Do[
  Export["test.ext", Import["http://i.imgur.com/ZyUf71d.jpg"], "GIF"];
  q7[[i]] = MemoryInUse[], {i, 1, qN}
  ];
q7 = (# - Min[#]) &@q7;
Print[N[Max[q7]/qN/2^20], " MB/element \n", "Total= ", 
 N[Max[q7]/2^20], "  MB"]
Print["------------- JPG -> GIF  WITH extra processing -------------"];
Do[
  Export["test.ext", 
   ColorNegate@
    LocalAdaptiveBinarize[Import["http://i.imgur.com/ZyUf71d.jpg"], 
     25, {0.8, 0., 0.034}, PerformanceGoal -> "Speed"], "GIF"];
  q8[[i]] = MemoryInUse[], {i, 1, qN}
  ];
q8 = (# - Min[#]) &@q8;
Print[N[Max[q8]/qN/2^20], " MB/element \n", "Total= ", 
 N[Max[q8]/2^20], "  MB"]
Print["------------- IMPORT JPG ONLY -------------"];
Do[
  Import["http://i.imgur.com/ZyUf71d.jpg"];
  q9[[i]] = MemoryInUse[], {i, 1, qN}
  ];
q9 = (# - Min[#]) &@q9;
Print[N[Max[q9]/qN/2^20], " MB/element \n", "Total= ", 
 N[Max[q9]/2^20], "  MB"]

And result is:

------------- GIF -> JPG  NO extra processing -------------
0.0000547791 MB/element 
Total= 0.00273895  MB
------------- GIF -> JPG  WITH extra processing -------------
2.44656 MB/element 
Total= 122.328  MB
------------- GIF -> GIF  NO extra processing -------------
9.91821*10^-6 MB/element 
Total= 0.000495911  MB
------------- GIF -> GIF  WITH extra processing -------------
9.91821*10^-6 MB/element 
Total= 0.000495911  MB
------------- JPG -> JPG  NO extra processing -------------
0.00268433 MB/element 
Total= 0.134216  MB
------------- JPG -> JPG  WITH extra processing -------------
2.44919 MB/element 
Total= 122.459  MB
------------- JPG -> GIF  NO extra processing -------------
0.00264175 MB/element 
Total= 0.132088  MB
------------- JPG -> GIF  WITH extra processing -------------
0.00264175 MB/element 
Total= 0.132088  MB
------------- IMPORT JPG ONLY -------------
0.00264175 MB/element 
Total= 0.132088  MB

And now, I can conclude that there is memoy leak in folowing cases:

  • negligible leak: GIF -> GIF with and wihthout extra processing (10 bytes per item), GIF -> JPG wihthout extra processing (57 bytes per item)
  • non-negligible leak : JPG -> GIF with and wihthout extra processing (2.7 KB per item)
  • non-negligible leak : JPG -> JPG wihthout extra processing (2.7 KB per item)
  • non-negligible leak : import JPG only (2.7 KB per item)
  • extreme unacceptable leak : GIG and JPG -> JPG with extra processing (2.4 MB per item)

source imported images are of size: GIF 434 KB, JPG 150 KB.

Please for explanation of this leaks, and how to cope with tasks of processing of about 50000-100000 images in a run.

1ST question instance: I would like that I am wrong, but it seems there is memory leak, or another problem with cache inside Module when repeatedly importing and exporting images, with assignment to the same variable.

Module[{im, m1, m2},
 m1 = MemoryInUse[];
 While[True,
  im = Import["http://i.imgur.com/ZyUf71d.jpg"];
  m2 = MemoryInUse[];
  Print[m2 - m1];
  m1 = m2;
  ]
 ]

This example is with Import only, but similar problem is when Export is included. In every loop it eats about 2.5 KB.

Please help about this.

$\endgroup$
3
$\begingroup$

I don't think this has anything to do with Module. Consider importing the same image repeatedly using Table:

q = Table[Import["http://i.imgur.com/ZyUf71d.jpg"];
   MemoryInUse[], {i, 1, 50}];
Differences[q]

Here you can see the same slow but steady size increase in the memory footprint. Here is a version using Do and a single predefined array:

q = ConstantArray[0, 50];
Do[Import["http://i.imgur.com/ZyUf71d.jpg"];
  q[[i]] = MemoryInUse[], {i, 1, 50}];
Differences[q]

It also experiences the same increase. Interestingly, Exporting does not incur the increase:

q = ConstantArray[0, 50];
Do[Export["test.jpg", "test"]; q[[i]] = MemoryInUse[], {i, 1, 50}];
Differences[q]

So this seems to be something to do with the Importing of images. In fact, we can narrow it down: importing a .gif does not incur the increase while importing a jpg does.

Export["test.gif", "test"]
q = ConstantArray[0, 50];
Do[Import["test.gif"]; q[[i]] = MemoryInUse[], {i, 1, 50}];
Differences[q]

{1584, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0}
$\endgroup$
  • $\begingroup$ Thank you for explanation. Which procedure can you suggest for processing huge amount of images, in some loop without this side effect. There must be a way. Eg to ColorConvert images and saving them to disk in a loop, without this "memory problem". Images are jpegs. $\endgroup$ – Dragutin Oct 30 '16 at 22:15
  • 1
    $\begingroup$ It could have to do with meta-data imported from the jpeg which could be handled separately form the main import. But you could save the image in another from than jpeg (tiff or png come to mind) which may not have the same issue. Unless someone comes up with a workaround or explanation, you should probably report it to Wolfram. $\endgroup$ – bill s Oct 30 '16 at 22:31
3
$\begingroup$

I am wondering if this is actually some sort of misrepresentation of how much memory is used. After modifying the code above to store the MemoryInUse values and to import 100 images from file in a table with 30 rows to make the effect more salient, we can see a continuous increase in memory used:

x = {};
Module[{im, m1, m2, i},
m1 = MemoryInUse[];
i = 0;
While[i < 30,        
  im = Table[Import["p.jpg"], 100];
  m2 = MemoryInUse[];
  m1 = m2;
  x = Append[x, m2];
  i++;
  ]
]

enter image description here

However, the actual memory in use is going up and down as profiled by Windows, so some memory gets freed up even before the module is complete and there is no actual leak. Apparently, the kernel is doing a routine memory sweep. Not sure what the MemoryInUse function is showing us then. If anything, the Wolfram team needs to improve this functionality.

enter image description here

$\endgroup$
  • $\begingroup$ What is surprisingly for me is that when we put im = Import["p.jpg"]; only, instead same in Table, then we get the same result. So, im variable is stored in memory for every iteration, what is not expected. I expect that im will be redefined in place of old one, so no old instance leave in memory. $\endgroup$ – Dragutin Oct 31 '16 at 7:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.