When I start Mathematica with a fresh kernel and load a program I am developing to analyze biology data, the amount of used system memory is 1.9 GB (free memory is 14.5 GB). These memory values are reported by an application external to Mathematica.
After I run the code I have developed, the amount of used system memory is 13.2 GB (free memory is 3.2 GB). So, this means that running my code has consumed 13.2-1.9 = 11.3 GB of memory. After running the code, MaxMemoryUsed[]
reports a value of 2.7 GB.
Even though Mathematica is reporting
MaxMemoryUsed
at 2.7Gb, my system is still reporting 13.2 GB of used memory. If this is true, why hasn't Mathematica cleared the 11.3 - 2.7 = 8.6 GB of memory it says it hasn't used and returned it to the system?I have tried setting
$HistoryLength = 0
and it has not made a difference in the reported memory usage.My code (posted below) does use several
ParallelTable
statements. Do these statements send a copy of the memory content to each parallel kernel?Do you have suggestions to solve this mystery of where is my memory?
Thank you for your thoughts. Todd
--------------- Mathematica Code ------------
(* load cel data *)
time0 = AbsoluteTime[];
SetDirectory[cellocation];
celfilenames = FileNames[]; (* names of data files to import *)
celvarnames =
Table[StringSplit[celfilenames[[i]], {"."}][[1]], {i, 1,
Length[celfilenames]}] ;(* names of datafiles, with extensions \
removed, to be used downstream *)
Table[microarray[celvarnames[[i]]] = Import[celfilenames[[i]]], {i, 1,
Length[celfilenames]}];(* assign data to microarray variable \
using celvarnames as indexes to microarray; each chip's data is \
assigned to a different index to microarray *)
chipsize =
chipdimensions[
microarray[celvarnames[[1]]]] (* determine chip dimensions *)
(* load affy cdf data *)
SetDirectory[affycdflocation];
cdffilenames = FileNames[];
cdffile =
Import[Flatten[StringCases[cdffilenames, ___ ~~ ".cdf" ~~ ___]][[
1]]];
(* convert hybidization symbol names to string names *)
experimentchips = Map[ToString, experimentchips];
controlchips = Map[ToString, controlchips];
Print["Experimental condition chips: ", experimentchips]
Print[]
Print["Control condition chips: ", controlchips]
pmindexes =
Table[Select[cdffile[[i, 3, 2]], #[[5]] != #[[6]] &][[All, 4]], {i,
1, Length[
cdffile]}]; (* perfect match probe indexes by probeset; same \
order as affy cdf file *)
mmapmindices =
Map[Transpose,
Thread[affyindextoMMAindices[pmindexes,
chipsize]]]; (* create Mathematica indices equivalent to Affy pm \
index positions by probeset *)
pmtemp = ParallelTable[
Extract[microarray[celvarnames[[i]]], mmapmindices[[j]]], {i, 1,
Length[celvarnames]}, {j, 1,
Length[mmapmindices]}]; (* get pm data from all the chips; data \
grouped by chip; within each chip,data is organized in same order as \
probes listed in cdf file *)
Table[pmsignal[celvarnames[[i]]] = pmtemp[[i]], {i, 1,
Length[celvarnames]}] ;(* put the pm data into the pmsignal \
variable, using celvarnames as indexes referring to the different \
chips *)
Clear[pmtemp,
microarray]; (* no need to retain duplicate information in memory, \
once data is assigned to specific chips *)
Table[pmstdev[celvarnames[[i]]] =
Map[StandardDeviation, pmsignal[celvarnames[[i]]]], {i, 1,
Length[celvarnames]}];
(* compute standard deviations by pm probeset,for each chip, and \
store it in pmstdev using celvarnames as indexes *)
subsetsize =
0.10*Length[
pmstdev[celvarnames[[
1]]]]; (* Use 10% of probesets to calculate clusters below *)
Table[pmstdevsubset[celvarnames[[i]]] =
BlockRandom[SeedRandom[1234];
RandomChoice[pmstdev[celvarnames[[i]]], Floor[subsetsize]]] , {i,
1, Length[celvarnames]}];
(* select a random subset of pm probset stdev for establishing \
"normal" pm probeset standard deviations for each chip *)
pmtemp = ParallelTable[
Sort[FindClusters[pmstdevsubset[celvarnames[[i]]], 3,
DistanceFunction -> EuclideanDistance,
Method -> "Agglomerate"]][[-1]], {i, 1,
Length[celvarnames]}]; (* Find the largest cluster for each pm \
standard deviation subset by chip; the mean of these clusters will \
set the "normal" pm variation for each chip; only find 3 clusters to \
make calculation quicker *)
Table[pmstdevcluster[celvarnames[[i]]] = pmtemp[[i]], {i, 1,
Length[pmtemp]}] ;(* assign pmtemp results to pmstdevcluster using \
celvarnames as keys *)
Clear[pmtemp]; (* no need to store same data twice *)
Table[pmpostocheck[celvarnames[[i]]] =
Flatten[Position[pmstdev[celvarnames[[i]]],
x_ /; x > Mean[pmstdevcluster[celvarnames[[i]]]]]], {i, 1,
Length[celvarnames]}];
(* determine each chips pm probeset positions whose st devs are \
greater than the "normal" threshold, which is calculated by taking \
the mean of the largest clusters for each chip - stored in \
pmstdevcluster *)
pmexperimentemp =
ParallelTable[
Flatten[Drop[
errorcorrection[pmsignal, experimentchips[[i]], pmpostocheck],
1], 2], {i, 1,
Length[experimentchips]}];(*calculate improved pmsignal in \
experimental chip probesets that have unusually high signal variation \
using the same probesets in comparable experimental condition chips \
as a surrogate*)
pmcontroltemp =
ParallelTable[
Flatten[Drop[
errorcorrection[pmsignal, controlchips[[i]], pmpostocheck], 1],
2], {i, 1,
Length[controlchips]}];(*calculate improved pmsignal in control \
chip probesets that have unusually high signal variation using the \
same probesets in comparable control condition chips as a surrogate*)
Table[pmsignal[experimentchips[[i]]] =
ReplacePart[pmsignal[experimentchips[[i]]],
Thread[pmpostocheck[experimentchips[[i]]] ->
pmexperimentemp[[i]]]], {i, 1,
Length[experimentchips]}]; (* replace spurious pmsignal with \
replacement values in experimental condition chips *)
Table[pmsignal[controlchips[[i]]] =
ReplacePart[pmsignal[controlchips[[i]]],
Thread[pmpostocheck[controlchips[[i]]] ->
pmcontroltemp[[i]]]], {i, 1,
Length[controlchips]}]; (* replace spurious pmsignal with \
replacement values in control condition chips *)
Clear[pmexperimentemp,
pmcontroltemp]; (* don't store data longer than needed *)
Table[experimentchippmmean[experimentchips[[i]]] =
Mean[Flatten[pmsignal[experimentchips[[i]]]]], {i, 1,
Length[experimentchips]}];
(* calculate each experimental chips pmsignal mean to use in \
standardization *)
Table[controlchippmmean[controlchips[[i]]] =
Mean[Flatten[pmsignal[controlchips[[i]]]]], {i, 1,
Length[controlchips]}]; (* calculate each control chips pmsignal \
mean to use in standardization *)
Table[experimentchippmstdev[experimentchips[[i]]] =
StandardDeviation[Flatten[pmsignal[experimentchips[[i]]]]], {i, 1,
Length[experimentchips]}]; (* calculate each experimental chips \
pmsignal standard deviation to use in standardization *)
Table[controlchippmstdev[controlchips[[i]]] =
StandardDeviation[Flatten[pmsignal[controlchips[[i]]]]], {i, 1,
Length[controlchips]}]; (* calculate each control chips pmsignal \
standard deviation to use in standardization *)
pmexpstandtemp =
ParallelTable[
Map[(# - experimentchippmmean[experimentchips[[i]]])/
experimentchippmstdev[experimentchips[[i]]] &,
pmsignal[experimentchips[[i]]]], {i, 1,
Length[experimentchips]}];(* standardize each experimental chips \
pmsignal *)
pmcontrolstandtemp =
ParallelTable[
Map[(# - controlchippmmean[controlchips[[i]]])/
controlchippmstdev[controlchips[[i]]] &,
pmsignal[controlchips[[i]]]], {i, 1,
Length[controlchips]}];(* standardize each control chips pmsignal \
*)
Table[experimentstandard[experimentchips[[i]]] =
pmexpstandtemp[[i]], {i, 1, Length[experimentchips]}];
(* reassign data from pmexpstandtemp to proper indexed variable *)
Table[controlstandard[controlchips[[i]]] =
pmcontrolstandtemp[[i]], {i, 1,
Length[controlchips]}];(* reassign data from pmcontrolstandtemp to \
proper indexed variable *)
Clear[pmexpstandtemp, pmcontrolstandtemp]; (* free up memory *)
time1 = AbsoluteTime[] - time0