4
$\begingroup$

As the question states, I'm trying to import a .csv file but I would like to exclude certain rows (points which I know are bad data).

This is easy when the points are at the end I simply use something like:

  Data = Import["\\\\MyLocation\\MyFolder\\Myfile.csv"][[Range[N],{1,2}]];

Where Range[N] is where I want to stop importing and {1,2} are the columns I want. However I am unsure how to exclude points which are in the middle of the data set.

Thanks in advance!

|improve this question
$\endgroup$
  • 1
    $\begingroup$ In fact Import has nothing to do with your approch, as you Import the whole file and then take a Part of it... take a look at Part,Span and Drop which will come in very handy for such tasks. $\endgroup$ – Yves Klett Apr 8 '16 at 11:26
  • 1
    $\begingroup$ You should avoid using capitalized variables because they might conflict with built-ins, such as N,E,D,.... N - which you used - is already defined. $\endgroup$ – Lukas Apr 8 '16 at 11:39
  • $\begingroup$ Thanks, I do try to! It is a habbit from more conventional coding where I use Camel-Casing (e.g. MyVariableName) $\endgroup$ – QuantumPenguin Apr 8 '16 at 12:34
  • $\begingroup$ Is there a way to escape reserved characters for use in such names? $\endgroup$ – QuantumPenguin Apr 8 '16 at 12:35
  • $\begingroup$ Just start with lowercase letters, e.g. myVariableName, myFun[x]:=.... For one letters variables, just use the uncapitalized version. $\endgroup$ – Lukas Apr 8 '16 at 12:47
3
$\begingroup$

As mentioned in the comments to your question, you just Import the whole data and then manipulate it afterwards. This is (I believe) a very transparent way to achieve the end result that you want and also illustrates why you should not use an "advanced" Import:

Let's produce some sample data

dim = 100;
data = RandomReal[{-1, 1}, {dim, 3}];
Export[NotebookDirectory[] <> "dummy.dat", data, "Table"];
ClearAll[data];

of which we want to exclude certain rows:

exclude = RandomInteger[{1, dim}, 8];
wanted = DeleteCases[Range[dim], Alternatives @@ exclude];

Of course, if you do just Import your data, say into the variable import, then dim=First@Dimensions@import. Now let's compare the two approaches:

Import everything and manipulate afterwards:

First@AbsoluteTiming[
  data = Import[NotebookDirectory[] <> "dummy.dat", "Table"];
  correctedData = data[[wanted]];]
(* 0.023810 *)

Import only the wanted rows:

First@AbsoluteTiming[correctedImport = Import[NotebookDirectory[] <> "dummy.dat", {"Data", wanted}];]
(* 0.050041 *)

Note that the latter uses the built-in functionality of Import

Import["file",elements]: imports the specified elements from a file.

Now, we verify that both are the same:

correctedData == correctedImport
(* True *)

So, concluding, it is probably only useful to import specified elements if you would run into memory issues otherwise. If there are no limitations due to ressources, importing everything and manipulating afterwards is faster.

|improve this answer
$\endgroup$
  • $\begingroup$ Thank you very much Lukas! The explanation was also very helpful! $\endgroup$ – QuantumPenguin Apr 8 '16 at 12:31
1
$\begingroup$

What follows is a variation on a previous answer

To exclude row 4 from comma-delimited data:

dataGood = Flatten[#, 1] & @(Import["/path/to/myfile.txt",{"Data", #, {All}}]
  & /@ {Range[3], Range[5, 6]});

and

dataGood // TableForm

Good Data - point excluded

For comparison:

dataAll = Import["/path/to/myfile.txt", {"Data", {All}, {All}}];
dataAll // TableForm

All data points with no exclusions

|improve this answer
$\endgroup$
  • $\begingroup$ I have just updated my answer. It should be mentioned that timings of this selective approach are pretty bad compared to just importing everything and manipulating afterwards. $\endgroup$ – Lukas Apr 8 '16 at 12:23
  • 1
    $\begingroup$ @Lukas. The OP specifically asks to exclude data which are known to be bad. By excluding a row or rows, you may be excluding MB of data. $\endgroup$ – user1066 Apr 8 '16 at 12:30
  • $\begingroup$ Hi TomD, thanks for answering, I like the simplicity of this solution. It also builds/extends upon what I already know. Thanks! $\endgroup$ – QuantumPenguin Apr 8 '16 at 12:32
  • $\begingroup$ @TomD That is of course true. However, unless one works with data files on the order of a few GB, I do not believe there is any advantage to preselect rows (just because it appears to be way slower). But you're right that this is for sure important, if operating at the limits of one's ressources. $\endgroup$ – Lukas Apr 8 '16 at 12:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.