60
Update: This is now available as the BoolEval package.
I recommend using
L UnitStep[15 - T]
for good performance.
To answer your question about boolean indexing:
When you write L[T > 15] = 0 in MATLAB, T > 15 evaluates to a boolean matrix of 0s and 1s, which can be used as a special selector index in assignments, as you showed (I am writing this ...
57
In my view, Cases and Position are in one camp (pattern-based functions used for general expression destructuring), while Select is in another: (more) special-purpose functions optimized to work on certain efficient data structures.
As was mentioned already, both Cases and Select do generally unpack when used with packed arrays. What wasn't mentioned is ...
49
Using a little Mathematica pattern matching, I think you can get similar performance as @Szabolcs's answer while having nice Matlab-style syntax:
replaceWhere[cond_, selectTrue_, selectFalse_] :=
With[{evaluatedCondition = evaluateTensorCondition[cond]},
selectTrue*evaluatedCondition + selectFalse*(1 - evaluatedCondition)]
replaceWhere[cond_, ...
42
Another useful thing to do when testing such things is to determine whether packed arrays are unpacking. For all of your cases there is a lot of unpacking going on (I've only shown the first of such messages...)
In[1]:= On["Packing"]
In[2]:= test = RandomInteger[{-25, 25}, {10^6, 2}];
In[3]:= (res1 = Cases[test, {_, _?Positive}]); // AbsoluteTiming
...
39
L=ReplacePart[L, Position[T, i_/;i>15]->0]
Go with @Szabolcs answer whenever you can
34
You could try BilateralFilter:
ListLinePlot[{data,
BilateralFilter[data, 2, .5, MaxIterations -> 25]},
PlotStyle -> {Thin, Red}]
Or alternatively, MeanShiftFilter can produce similar results:
ListLinePlot[{data,
MeanShiftFilter[data, 5, .5, MaxIterations -> 10]},
PlotStyle -> {Thin, Red}]
Third alternative, as mentioned by @Xavier ...
27
With:
dat = {{10, b, 30}, {100, a, 40}, {1000, b, 10}, {1000, b, 70}, {100, b, 20}, {10, b, 70}};
Perhaps most directly:
Cases[dat, {_, _, Max@dat[[All, 3]]}]
More approaches:
Last @ SplitBy[SortBy[dat, {#[[3]] &}], #[[3]] &]
Pick[dat, #, Max@#] &@dat[[All, 3]]
Reap[Fold[(If[#2[[3]] >= #, Sow@#2]; #2[[3]]) &, dat]][[2, 1]]
Of these ...
27
Perhaps one of the simplest ways is to use Tally:
p = {1, 2, 4, 4, 6, 7, 8, 8};
Cases[Tally @ p, {x_, n_ /; n > 1} :> x]
{4, 8}
A somewhat faster formulation:
Pick[#, Unitize[#2 - 1], 1] & @@ Transpose[Tally @ p]
Taking the optimization to a rather excessive degree:
#[[SparseArray[#2, Automatic, 1]["AdjacencyLists"]]] & @@ Transpose[...
27
Please look up Part and Span.
You can use
data[[ ;; ;; n]]
27
res1 = Select[coordinates, #[[1]] > 6 && #[[1]] < 7 &]; //
AbsoluteTiming // First
6.997629
res2 = Select[coordinates, 6 < #[[1]] < 7 &]; // AbsoluteTiming // First
4.676356
res3 = Pick[coordinates, 6 < # < 7 & /@ coordinates[[All, 1]]]; //
AbsoluteTiming // First
5.266651
res4 = Pick[coordinates, (1 - ...
23
New proposal
I was thinking about this problem today and came up with a new approach. In testing it appears to be competitively fast, often notably faster than any other method yet posted. It is also quite clean.
A limitation shared with rasher's uc: all elements in the drop list must be present in the main list.
fastRF[a_List, b_List] :=
Module[{c, o,...
23
(Edited with a slight tweak for a tiny bit more speed)
For the non-duplicate version I am finding this quite fast:
f = If[+## > 0, ##] & @@ {Max[#], Min[#]} &
f /@ list
(* {-5, 6, 4, -9} *)
23
The following code will filter noisy data…
SGKernel[left_?NonNegative, right_?NonNegative, degree_?NonNegative, derivative_? NonNegative] :=
Module[{i, j, k, l, matrix, vector},
matrix = Table[ (* matrix is symmetric *)
l = i + j;
If[l == 0,
left + right + 1,
(*Else*)
Sum[k^l, {k, -left, right}]
],...
22
You can use the usual UnitStep + Total tricks:
r1 = Table[Total[UnitStep[m-s]], {m,10000}]; //AbsoluteTiming
r2 = Table[Length@Select[s,LessEqualThan[m]],{m,10000}];//AbsoluteTiming
r1 === r2
{0.435358, Null}
{41.4357, Null}
True
Update
As @J42161217 points out, you can take advantage of the fact that the data is sorted to speed things up. ...
21
Please see my second answer; the method therein is far more efficient than the ones below.
removeFrom[b_List, a_List] := Module[{f},
f[_] = 0;
(f[#] = -#2) & @@@ Tally[a];
Pick[b, UnitStep[f[#]++ & /@ b], 1]
]
removeFrom[{a, b, c, a, d, a, e}, {a, c, a}]
{b, d, a, e}
Here somewhat longer but also a bit more efficient:
removeFrom2[b_List, ...
21
edit: this doesn't really answer the question but merely provides some other alternatives, you should probably up-vote other more useful answers.
There are also faster ways to do this using Pick or by compiling Select. Timing comparison done on a Macbook Air OS X 10.8.3 w/ 1.7 GHz Intel Core i5 with Mathematica 9.0.0.0.
t = RandomInteger[100, 10^7];
...
21
Here is a test list:
lst = {{1, 2}, {1, 2}, {3, 4}, {5, 6}, {5, 6}, {7, 8}}
Here is one way then:
GroupBy[Tally[lst], Last][1][[All, 1]]
(* {{3, 4}, {7, 8}} *)
The same idea using purely associations:
Keys[GroupBy[Counts[lst], Identity][1]]
(* {{3, 4}, {7, 8}} *)
A somewhat more efficient method can be this:
Pick[lst, Lookup[Counts[lst], lst], 1]
(*...
20
I will give you two similar methods. But, I will rewrite one of the comments above just to make sure it is read.
You've been given some fine answers, but be absolutely sure that removing the outliers is Doing The Right Thing™. You might want to consider "robust" methods that can deal with the presence of outliers. – Guess who it is.
Simple ...
20
The question title poses a good question, although the question formulation is somewhat specialized and misplaced (as mentioned in a comment).
This answer provides data and a method description answering:
How to find outliers in 3D numerical data?
Data
It order to provide a good answer it would be better to use "real life" data. Not spending much time ...
20
ReplaceRepeated is fine for short lists but it will get very slow if the list is long, because it starts over from the beginning of the list after each replacement. A better approach is to start the next replacement after the point of the previous one. One implementation of that:
fn1 = # /.
{a___, aa_Symbol, _Integer, bb_Symbol, b___} :>
...
19
Szabolcs's answer using UnitStep is the right one for your specific case. It is also the most efficient by far: see the timing results below. Explicit Map operations and size comparisons do seem to be particularly slow.
However there is a Boole function that replicates the Matlab approach of returning 1 and 0 instead of True and False, which might be useful ...
19
You may use Tally to finish the task as follows:
Cases[Tally[list], {x_, 4} :> x]
the result will be {a,b}.
19
Try
testList //. {a___, aa_Symbol, _Integer, bb_Symbol, b___} :> {a, aa, bb, b}
{a, b, c, 4, 5, d, e, f, g, 4}
% == resultList
True
Note that I replaced __ (BlankSequence) with ___ (BlankNullSequence), so this will work for an integer which appears as the second element in the list.
19
Slightly faster than @kglr's solution is to use Clip:
SeedRandom[1];
coordinates = RandomReal[10, {4000000, 3}];
r1 = Pick[
coordinates,
Unitize @ Clip[coordinates[[All,1]], {6, 7}, {0, 0}],
1
];//RepeatedTiming
r2 = Pick[
coordinates,
(1-UnitStep[#-7]) (1-UnitStep[6-#])&@coordinates[[All,1]],
1
];//RepeatedTiming
r1 === r2
...
18
Mathematica does not do well with code that relies on mutable state (i.e. an explicit variable whose value is changing during the run of the program). Let's look at your For code:
For[i=0, i < Length[t], i++,
If[ t[[i]] > 50, r=Append[r, t[[i]]]]
]
Notice that for every iteration, it needs to evaluate the following by interpreting high level ...
18
A fast uncompiled alternative without pattern matching is to use the NonzeroPositions property of SparseArray, as long as you're dealing with numerical data.
list = RandomInteger[{1, 100}, 10^7];
SparseArray[Unitize[list - Max[list]] - 1]["NonzeroPositions"]; // RepeatedTiming
(* 0.0855 *)
Position[list, Max[list]] // RepeatedTiming
(* 0.509 *)
compPos[...
18
Mr.Wizard inspired me to improve. While his recursive approach is elegant, the problem clearly can be done linearly. Indeed:
jjc2 = (
Sow@First@#;
BlockMap[
If[ ! MatchQ[#, {_Symbol, _Integer, _Symbol}], Sow@#[[2]] ]; &,
#, 3, 1 ];
Sow@Last@big;
// Reap
// Last@*Last
) &
(r1=jjc2@big); // AbsoluteTiming
{0....
17
I would use either
Cases[data, {x_, y_} /; Abs[x - y] > 4]
or
With[{diff = Abs[data[[All, 1]] - data[[All, 2]]] - 4},
Pick[data, UnitStep[diff]*Unitize[diff], 1]
]
The first clearly demonstrates what you are trying to do, the second is much faster...
data = RandomInteger[{0, 100}, {10^6, 2}];
(m1 = Cases[data, {x_, y_} /; Abs[x - y] > 4]); //...
17
You can also use FilterRules:
rule = {beta -> 4, alpha -> 2, x -> 4, z -> 2, w -> 0.8};
FilterRules[rule, beta]
(* {beta -> 4} *)
FilterRules[rule, {beta, alpha}]
(* {beta -> 4, alpha -> 2} *)
Update: additional alternatives if you have V10:
KeyTake[rule,{alpha, x}]
(* or *) KeyTake[{alpha,x}][rule]
(* <|alpha->2,x->4|&...
17
For a 1D list you can also use
Pick[Range@Length@list, list, Max@list]
Only top voted, non community-wiki answers of a minimum length are eligible
