Take the 2-minute tour ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

I read ReplaceOrRemoveInvalidOrMissingData Mathematica documentation, and I tried to apply the same approach to my dataset.

I've got two problems, but one is only a style problem!

  1. There is one more elegant way to select numeric data in a large dataset? I mean that I've got a dataset o 29 possible numeric value and 2 columns that are not numerical. So I tried this way.

    baddata[entry_] :=  Not[MatchQ[ entry, 
      {_?NumberQ, _?NumberQ, _?NumberQ, _?NumberQ, _?NumberQ, _?NumberQ, _?NumberQ,
       _?NumberQ, _?NumberQ, _?NumberQ, _?NumberQ, _?NumberQ, _?NumberQ, _?NumberQ, 
       _?NumberQ, _?NumberQ, _?NumberQ, _?NumberQ, _?NumberQ, _?NumberQ, _?NumberQ,
       _?NumberQ, _?NumberQ, _?NumberQ, _?NumberQ, _?NumberQ, _?NumberQ, _?NumberQ,
       _?NumberQ, __}]]
    
    DeleteCases[data, _?baddata];
    

It works, but it is not so elegant!

2 . I would accept some non numerical (e.g. blank values) values in some columns (e.g. the first, the twelfth and the twentieth). How can I modify this algorithm? Thanks in advance.


EDIT

Your answers are very useful for me, and I'm wondering how all of you use Mathematica in a very smart way! For my first question it's very clear all your solutions, but, it is not clear for me (surely I didn't explain in a right way my problem) if I've got this kind of dataset

mydataset={{1, 2, 3, 4}, {3, 4, 5, 4}, {1, "", 3, 4}, {10, "", 12, 13}, {20, 30,
40, ""}, {100, 200, 300, ""}};

I want to select the row in this way: I can admit blank entries for the second column, but not for the fourth column. So the result would be:

result={{1, 2, 3, 4}, {3, 4, 5, 4}, {1, "", 3, 4}, {10, "", 12, 13}}

Can I modify your algorithm in this way? Thanks again, and sorry!

share|improve this question
1  
See function "Repeated". –  Ymareth Mar 18 at 22:02
1  
Try, e.g., Cases[data, _?NumberQ] for a list. –  rasher Mar 18 at 22:15
1  
Please see the third section of my updated answer. –  Mr.Wizard Mar 19 at 14:45

4 Answers 4

Shorter at least...

data = Cases[data, {Repeated[_?NumberQ, {29}], _,_}];
share|improve this answer

For the first example Repeated, as already shown by Ymareth, is the simplest method:

MatchQ[
 {1, 2/3, 3.14, 4, 5.0, "x", "y"},
 {Repeated[_?NumericQ, {5}], _, _}
]
True

For the case of arbitrary positions it seems to me that there are two natural approaches:

Method 1: Build a pattern

You can programmatically build a pattern like the one shown in you question. There are many possible methods, but since building the pattern itself will be very fast I'll use the first one that comes to mind: SparseArray:

pat = Normal @ SparseArray[1 | 3 | 7 -> _, 10, _?NumericQ]
{_, _?NumericQ, _, _?NumericQ, _?NumericQ, _?NumericQ,
 _, _?NumericQ, _?NumericQ, _?NumericQ}

This can easily be turned into a function, e.g.:

fn[non_, len_] := Normal @ SparseArray[non -> _, len, _?NumericQ]

fn[2 | 4, 6]
{_?NumericQ, _, _?NumericQ, _, _?NumericQ, _?NumericQ}

This kind of pattern has early exit behavior: as soon as one element fails a test the pattern fails to match, and no additional time is spent on needlessly testing the other elements.

Method 2: Extract the elements to test

The other approach is to extract (or delete) only the elements that are to be handled specially. For the first example you could write a test function like this:

test = Length[#] == 31 && MatchQ[Take[#, 29], {__?NumericQ}] &;

Join[Range[29], {"foo", "bar"}] // test
True

For your second example you could include Delete:

test2[non_, len_][x_] := 
  Length[x] == len && MatchQ[Delete[x, List /@ non], {__?NumericQ}]

{"a", 2, "c", 4, 5, 6, "g", 8, 9, 10} // test2[{1, 3, 7}, 10]
True

This also has early exit behavior without extraneous testing.


Application examples

Addressing the addendum to your question using this data:

mydataset = {{1, 2, 3, 4}, {3, 4, 5, 4}, {1, "", 3, 4}, {10, "", 12, 13}, {20, 30,
40, ""}, {100, 200, 300, ""}};

First method:

Cases[mydataset, fn[2, 4]]
{{1, 2, 3, 4}, {3, 4, 5, 4}, {1, "", 3, 4}, {10, "", 12, 13}}

Second method:

Select[mydataset, test2[2, 4]]
{{1, 2, 3, 4}, {3, 4, 5, 4}, {1, "", 3, 4}, {10, "", 12, 13}}

Cases is used for the first method because it builds a pattern, while Select is used for the second because it builds a test function.

share|improve this answer

You say you want to be able to decide which columns have to be numeric and which ones don't. There are many approaches that aren't based on patterns and this is probably not the best one, but it will do the job. Let's define

checkNumericColumns[entry_, blanks_] := 
 Module[{shouldBeNumeric, areNumeric},
  shouldBeNumeric = Complement[Range@Length@entry, blanks];
  areNumeric = Union @@ Position[entry, _?NumberQ];
  shouldBeNumeric == areNumeric
  ]

so that

checkNumericColumns[{1, 2, 3, "a", 4, 5, "b"}, {7}]
(* Out: False *)
checkNumericColumns[{1, 2, 3, "a", 4, 5, "b"}, {4, 7}]
(* Out: True *)

Now you can select the elements you want to keep from your original data set like this:

Select[data, checkNumericColumns[#, blanks] &]

where blanks is a list of columns that you represented with blanks.

share|improve this answer
    
Very nice solution. Thanks! –  Mary Mar 19 at 8:38

For the case with non numerical columns that are allowed.

sel[dat_, nonNum_] := With[{n = Length@dat[[1]]}, 
  Pick[
   dat, 
   And @@@ Map[NumericQ, dat[[;; , DeleteCases[Range[n], Alternatives @@ nonNum]]], {2}]
   ]
  ]

so for the edit test:

sel[mydataset, {2}]
{{1, 2, 3, 4}, {3, 4, 5, 4}, {1, "", 3, 4}, {10, "", 12, 13}}
share|improve this answer
    
+1 for taking out the columns that are required to be numerical first. In case the numbers are all integers for example, this has the additional advantage that you might restructure the data as a packedarray. Also I think mapping a function like this should be faster than testing for the pattern _?NumberQ. You could map the following function on level 1 instead, to make it have early exit behaviour. Catch[Scan[If[! NumberQ[#], Throw[False]] &, #]; Throw[True]] &. Maybe it will be faster. –  Jacob Akkerboom Mar 19 at 14:44
    
@JacobAkkerboom Thanks for your attention :) Well, you are probably right. I don't know, I just have no motivation for this answer for shaving timings ;P –  Kuba Mar 19 at 16:52
    
yeah and probably there is hardly any difference and it is less clear :P. Wise men speak because they have something to say, fools speak because they have to say something. Screw that, wise men are lonely :P –  Jacob Akkerboom Mar 19 at 20:33
    
@JacobAkkerboom In fact the opposite ;) My performance tunning is usually walking randomly. Maybe one idea is better from another in terms of optimal algorithm but when it comes to built-in functions I'm very often suprised :) Have a good night ;) –  Kuba Mar 19 at 20:50

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.