I have a csv file, size around 50Mb. The data is 986132 by 3 matrix of the following form.
data={{a1,b1,1},{a1,b2,1},{a1,b3,1},{a1,b4,1},{a2,b1,1},{a2,b5,1},{a2,b6,1},{a3,b1,1},{a3,b2,1},{a3,b6,1},{a3,b7,1},...{a3012,b1423,1}}
The third column is always 1 and there is no pattern between a's and b's.
In the matrix form of data
, I focus on the second column (b's) and want to remove all the rows that has b$i$ that appears less than a certain number. Say it's 10.
So, for example, if b10, b31, b412 each appears less than 10 times in data
, I want remove all the rows that contain b11, b31 and b412.
How can I do this? I've tried to convert it to a pivot table and sort it by the number of each b, but my computer can't process it maybe because of the memory limitation..