# More elegant way

I read ReplaceOrRemoveInvalidOrMissingData Mathematica documentation, and I tried to apply the same approach to my dataset.

I've got two problems, but one is only a style problem!

1. There is one more elegant way to select numeric data in a large dataset? I mean that I've got a dataset o 29 possible numeric value and 2 columns that are not numerical. So I tried this way.

baddata[entry_] :=  Not[MatchQ[ entry,
{_?NumberQ, _?NumberQ, _?NumberQ, _?NumberQ, _?NumberQ, _?NumberQ, _?NumberQ,
_?NumberQ, _?NumberQ, _?NumberQ, _?NumberQ, _?NumberQ, _?NumberQ, _?NumberQ,
_?NumberQ, _?NumberQ, _?NumberQ, _?NumberQ, _?NumberQ, _?NumberQ, _?NumberQ,
_?NumberQ, _?NumberQ, _?NumberQ, _?NumberQ, _?NumberQ, _?NumberQ, _?NumberQ,
_?NumberQ, __}]]



It works, but it is not so elegant!

2 . I would accept some non numerical (e.g. blank values) values in some columns (e.g. the first, the twelfth and the twentieth). How can I modify this algorithm? Thanks in advance.

EDIT

Your answers are very useful for me, and I'm wondering how all of you use Mathematica in a very smart way! For my first question it's very clear all your solutions, but, it is not clear for me (surely I didn't explain in a right way my problem) if I've got this kind of dataset

mydataset={{1, 2, 3, 4}, {3, 4, 5, 4}, {1, "", 3, 4}, {10, "", 12, 13}, {20, 30,
40, ""}, {100, 200, 300, ""}};


I want to select the row in this way: I can admit blank entries for the second column, but not for the fourth column. So the result would be:

result={{1, 2, 3, 4}, {3, 4, 5, 4}, {1, "", 3, 4}, {10, "", 12, 13}}


Can I modify your algorithm in this way? Thanks again, and sorry!

• See function "Repeated". Mar 18 '14 at 22:02
• Try, e.g., Cases[data, _?NumberQ] for a list.
– ciao
Mar 18 '14 at 22:15
• Please see the third section of my updated answer. Mar 19 '14 at 14:45

Shorter at least...

data = Cases[data, {Repeated[_?NumberQ, {29}], _,_}];


For the first example Repeated, as already shown by Ymareth, is the simplest method:

MatchQ[
{1, 2/3, 3.14, 4, 5.0, "x", "y"},
{Repeated[_?NumericQ, {5}], _, _}
]

True


For the case of arbitrary positions it seems to me that there are two natural approaches:

## Method 1: Build a pattern

You can programmatically build a pattern like the one shown in you question. There are many possible methods, but since building the pattern itself will be very fast I'll use the first one that comes to mind: SparseArray:

pat = Normal @ SparseArray[1 | 3 | 7 -> _, 10, _?NumericQ]

{_, _?NumericQ, _, _?NumericQ, _?NumericQ, _?NumericQ,
_, _?NumericQ, _?NumericQ, _?NumericQ}


This can easily be turned into a function, e.g.:

fn[non_, len_] := Normal @ SparseArray[non -> _, len, _?NumericQ]

fn[2 | 4, 6]

{_?NumericQ, _, _?NumericQ, _, _?NumericQ, _?NumericQ}


This kind of pattern has early exit behavior: as soon as one element fails a test the pattern fails to match, and no additional time is spent on needlessly testing the other elements.

## Method 2: Extract the elements to test

The other approach is to extract (or delete) only the elements that are to be handled specially. For the first example you could write a test function like this:

test = Length[#] == 31 && MatchQ[Take[#, 29], {__?NumericQ}] &;

Join[Range, {"foo", "bar"}] // test

True


For your second example you could include Delete:

test2[non_, len_][x_] :=
Length[x] == len && MatchQ[Delete[x, List /@ non], {__?NumericQ}]

{"a", 2, "c", 4, 5, 6, "g", 8, 9, 10} // test2[{1, 3, 7}, 10]

True


This also has early exit behavior without extraneous testing.

## Application examples

mydataset = {{1, 2, 3, 4}, {3, 4, 5, 4}, {1, "", 3, 4}, {10, "", 12, 13}, {20, 30,
40, ""}, {100, 200, 300, ""}};


First method:

Cases[mydataset, fn[2, 4]]

{{1, 2, 3, 4}, {3, 4, 5, 4}, {1, "", 3, 4}, {10, "", 12, 13}}


Second method:

Select[mydataset, test2[2, 4]]

{{1, 2, 3, 4}, {3, 4, 5, 4}, {1, "", 3, 4}, {10, "", 12, 13}}


Cases is used for the first method because it builds a pattern, while Select is used for the second because it builds a test function.

You say you want to be able to decide which columns have to be numeric and which ones don't. There are many approaches that aren't based on patterns and this is probably not the best one, but it will do the job. Let's define

checkNumericColumns[entry_, blanks_] :=
Module[{shouldBeNumeric, areNumeric},
shouldBeNumeric = Complement[Range@Length@entry, blanks];
areNumeric = Union @@ Position[entry, _?NumberQ];
shouldBeNumeric == areNumeric
]


so that

checkNumericColumns[{1, 2, 3, "a", 4, 5, "b"}, {7}]
(* Out: False *)
checkNumericColumns[{1, 2, 3, "a", 4, 5, "b"}, {4, 7}]
(* Out: True *)


Now you can select the elements you want to keep from your original data set like this:

Select[data, checkNumericColumns[#, blanks] &]


where blanks is a list of columns that you represented with blanks.

• Very nice solution. Thanks!
– Mary
Mar 19 '14 at 8:38

For the case with non numerical columns that are allowed.

sel[dat_, nonNum_] := With[{n = Length@dat[]},
Pick[
dat,
And @@@ Map[NumericQ, dat[[;; , DeleteCases[Range[n], Alternatives @@ nonNum]]], {2}]
]
]


so for the edit test:

sel[mydataset, {2}]

{{1, 2, 3, 4}, {3, 4, 5, 4}, {1, "", 3, 4}, {10, "", 12, 13}}

• +1 for taking out the columns that are required to be numerical first. In case the numbers are all integers for example, this has the additional advantage that you might restructure the data as a packedarray. Also I think mapping a function like this should be faster than testing for the pattern _?NumberQ. You could map the following function on level 1 instead, to make it have early exit behaviour. Catch[Scan[If[! NumberQ[#], Throw[False]] &, #]; Throw[True]] &. Maybe it will be faster. Mar 19 '14 at 14:44
• @JacobAkkerboom Thanks for your attention :) Well, you are probably right. I don't know, I just have no motivation for this answer for shaving timings ;P
– Kuba
Mar 19 '14 at 16:52
• yeah and probably there is hardly any difference and it is less clear :P. Wise men speak because they have something to say, fools speak because they have to say something. Screw that, wise men are lonely :P Mar 19 '14 at 20:33
• @JacobAkkerboom In fact the opposite ;) My performance tunning is usually walking randomly. Maybe one idea is better from another in terms of optimal algorithm but when it comes to built-in functions I'm very often suprised :) Have a good night ;)
– Kuba
Mar 19 '14 at 20:50