Doing some mappings of the earth magnetic field results in the left-hand figure below (which consists of 181 profiles):
To improve this I tried a non-linear trend reduction using, for example,
lm = LinearModelFit[data, {a, a^2,a^3,a^4,a^5,a^6,a^7,a^8,a^9},a];
(middle figure) and
lm = LinearModelFit[data, {a, a^2,a^3}, a];
(right-hand figure), respectively. Data was defined as data = Import[#1, "Table"]; within a Scan environment.
Because of the occurrence of artifacts I feel that I have to consider something like thresholds, so that the reduction will be only considered within a threshold (for example -20 <= t <= 20) and the data outside of this range should not be affected by these calculations.
I have no idea how to handle this effectively. My idea first was to use a lot of IF's and to separate each profile into several parts. But this will be ineffective and time-consuming...
I would like to handle all those 181 profiles separately before combining them.
@R.M.: This is the code I used so far:
SetDirectory[SystemDialogInput["Directory"]]
listOfFileNames = ReadList["listing.txt", Word];
Scan[
(data = Import[#1, "Table"];
newfile = StringReplace[#1, ".dat" -> "-red.dat"];
wfile = OpenWrite[newfile, FormatType -> StandardForm];
Clear[lm]; Clear[a]; Clear[neu]; Clear[aha];
start = First[First[data]]; mdim = Length[data];
lm = LinearModelFit[
data, {a, a^2, a^3, a^4, a^5, a^6, a^7, a^8, a^9}, a];
Do[aha = Last[Take[data, {m, m}]]; ywert = Last[aha];
xwert = First[aha]; neu[m] = ywert - lm[xwert], {m, mdim}];
Do[WriteString[newfile, start + (m - 1)*0.05 , " "];
Write[newfile, neu[m]], {m, mdim}];
Close[wfile];
Clear;
) &,
listOfFileNames
]
With listing.txt:
09500.dat 09525.dat 09550.dat 09575.dat 09600.dat 09625.dat 09650.dat 09675.dat etc
@belisarius:
This shows the quality of data outside of the pipeline I want to achieve
@Mr.Wizard: With VSNR there is no effect on the data:
