I have a csv file, size around 50Mb. The data is 986132 by 3 matrix of the following form.
The third column is always 1 and there is no pattern between a's and b's.
In the matrix form of
data, I focus on the second column (b's) and want to remove all the rows that has b$i$ that appears less than a certain number. Say it's 10.
So, for example, if b10, b31, b412 each appears less than 10 times in
data, I want remove all the rows that contain b11, b31 and b412.
How can I do this? I've tried to convert it to a pivot table and sort it by the number of each b, but my computer can't process it maybe because of the memory limitation..