non-linear trend reduction using thresholds

Doing some mappings of the earth magnetic field results in the left-hand figure below (which consists of 181 profiles):

To improve this I tried a non-linear trend reduction using, for example,

lm = LinearModelFit[data, {a, a^2,a^3,a^4,a^5,a^6,a^7,a^8,a^9},a];


(middle figure) and

lm = LinearModelFit[data, {a, a^2,a^3}, a];


(right-hand figure), respectively. Data was defined as data = Import[#1, "Table"]; within a Scan environment.

Because of the occurrence of artifacts I feel that I have to consider something like thresholds, so that the reduction will be only considered within a threshold (for example -20 <= t <= 20) and the data outside of this range should not be affected by these calculations.

I have no idea how to handle this effectively. My idea first was to use a lot of IF's and to separate each profile into several parts. But this will be ineffective and time-consuming...

I would like to handle all those 181 profiles separately before combining them.

@R.M.: This is the code I used so far:

SetDirectory[SystemDialogInput["Directory"]]
listOfFileNames = ReadList["listing.txt", Word];

Scan[
(data = Import[#1, "Table"];
newfile = StringReplace[#1, ".dat" -> "-red.dat"];
wfile = OpenWrite[newfile, FormatType -> StandardForm];
Clear[lm]; Clear[a]; Clear[neu]; Clear[aha];
start = First[First[data]]; mdim = Length[data];
lm = LinearModelFit[
data, {a, a^2, a^3, a^4, a^5, a^6, a^7, a^8, a^9}, a];
Do[aha = Last[Take[data, {m, m}]]; ywert = Last[aha];
xwert = First[aha]; neu[m] = ywert - lm[xwert], {m, mdim}];
Do[WriteString[newfile, start + (m - 1)*0.05  , "  "];
Write[newfile,    neu[m]], {m, mdim}];
Close[wfile];
Clear;
) &,
listOfFileNames
]


With listing.txt:

09500.dat 09525.dat 09550.dat 09575.dat 09600.dat 09625.dat 09650.dat 09675.dat etc

@belisarius:

This shows the quality of data outside of the pipeline I want to achieve

@Mr.Wizard: With VSNR there is no effect on the data:

-
"To improve this" -- what does that mean in this context? – Mr.Wizard Jul 29 '12 at 19:04
It is hard to think about this because it is muddled with a lot of domain specific details. Could you strip all of it to a basic minimal toy example (working)? – rm -rf Jul 29 '12 at 19:06
there are small stripes (parallel to the direction of measurement, i.e. from bottom to the top) in the figure, which should be removed. The "background" (all the area outside of the strong anomaly) should be smoothed by this way, so when enhancing the contrast of the picture no small artefacts are disturbing the background. I have to enhance the contrast to be able to detect weak archaeological features, if there are any... – Harald Jul 29 '12 at 19:12
Does this appear helpful? math.univ-toulouse.fr/~weiss/Codes/VSNR/… – Mr.Wizard Jul 29 '12 at 19:42
Could you post an example of your expected result? – belisarius Jul 29 '12 at 19:58
show 5 more comments

protected by rm -rf♦Oct 30 '12 at 14:45

This question is protected to prevent "thanks!", "me too!", or spam answers by new users. To answer it, you must have earned at least 10 reputation on this site.