The x-y data that is being analyzed generates a linear relationship (y=mx+b) when in steady-state flow. NON-REGULAR transients are often introduced into the system. These transients prevent the fitting of a linear relationship to the raw data. I am looking for advice on how to improve the identification of these transients automatically. Using FindAnomalies appears to select the steady state data more often the transient data (see plot down further below) - particulary in the case of the Multinormal method.

Ideally, the following criteria must be met:

  • The analyst does not want to manually remove the transient data
  • If we can identify (therefore remove) the transient points, we can easily generate our own linear regression (or other procedure) for generating the linear relationship
  • The steady state condition is always a linear relationship with a negative slope
  • The analyst can adjust the tolerance, nothing more

As indicated, I am using the FindAnomalies function experimenting with the Method detect the transient spikes and then remove them using AnomalyPositions property.

A review of the plot below will shows the points which are being detected by FindAnomalies using both Multinormal and KernelDensityEstimation methods. Even with small tolerances, the function tends to identify on trend data as an anomaly.

Visualization of detected anomalies

The complete code for the demonstration is below. Any advice or suggestions is appreciated

 TR = FindAnomalies[data[[All, 2]], AcceptanceThreshold -> ACT, 
   Method -> "Multinormal"];
 TR2 = FindAnomalies[data[[All, 2]], "AnomalyPositions", 
    AcceptanceThreshold -> ACT, Method -> "Multinormal"] // Flatten;
 XR = FindAnomalies[data[[All, 2]], AcceptanceThreshold -> ACT, 
   Method -> "KernelDensityEstimation"];
 XR2 = FindAnomalies[data[[All, 2]], "AnomalyPositions", 
    AcceptanceThreshold -> ACT, Method -> "KernelDensityEstimation"] //
 AnomData = Table[
                data[[TR2[[t]], 1]], TR[[t]]
            }, {t, 1, Length[TR]}
 XAnomData = Table[
                data[[XR2[[t]], 1]], XR[[t]]
            }, {t, 1, Min[Length[XR], Length[XR2]]}
 ListPlot[{AnomData, XAnomData, data},
            PlotMarkers -> {{"\[Alpha]", 11}, {"\[Beta]", 10}, {"\[Gamma]", 
            Joined -> {False, False, True},
            GridLines -> Automatic,
            PlotLegends -> {"Multinormal", "KernelDensityEstimation", 
    "Raw Data"}
 {{ACT, 0.2, "Tolerance"}, 0.01, 1, 0.01}

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.