This is neither elegant nor smart nor memory efficient. It is a brute force method to get all solutions of a given size

isGood[m_] := Sort@m === reye@Length@m;
i : reye[l_] := i = Reverse@IdentityMatrix@l;

getAllSolutions[n_?PrimeQ] := With[{id = IdentityMatrix@n},
     Pick[id, #, 1] & /@ Boole@PrimeQ@Partition[Range[n^2], n] // 
     Tuples]~Select~isGood;

So

Row[MatrixForm /@ #] & /@ 
  Composition[getAllSolutions, Prime]~Array~4 // 
 Column@Riffle[#, "New prime"] &

Gives

This is neither elegant nor smart nor memory efficient. It is a brute force method to get all solutions of a given size

isGood[m_] := Sort@m === reye@Length@m;
i : reye[l_] := i = Reverse@IdentityMatrix@l;

getAllSolutions[n_?PrimeQ] := With[{id = IdentityMatrix@n},
     Pick[id, #, 1] & /@ Boole@PrimeQ@Partition[Range[n^2], n] // 
     Tuples]~Select~isGood;

So

Row[MatrixForm /@ #] & /@ 
  Composition[getAllSolutions, Prime]~Array~4 // 
 Column@Riffle[#, "New prime"] &

Gives

EDIT

I imagined that a solution along the lines of @bgatessucks 's great answer, but with booleans, would be more efficient and appropriate. However, while this is true for sizes below 13 (an order of magnitude faster in my tests), for some reason it suddenly becomes terribly slow afterwards.

v2[n_?PrimeQ, nsols_Integer: 1] := Module[{mat, vars},
  {mat, {vars}} = 
   Reap[PrimeQ@Partition[Range[n^2], n] /. True :> Sow@Unique["p"]];
  SatisfiabilityInstances[
    And @@ BooleanCountingFunction[{1}, n] @@@ 
      Join[mat, Transpose@mat], vars, nsols] /. 
   res_ :> (mat /. (Thread[vars -> #] & /@ res) /. {False -> ".", 
       True -> "P"})
  ]

Now

MatrixForm /@ v2[Prime@6, 3]

Gives

This is neither elegant nor smart nor memory efficient. It is a brute force method to get all solutions of a given size

isGood[m_] := Sort@m === reye@Length@m;
i : reye[l_] := i = Reverse@IdentityMatrix@l;

getAllSolutions[n_?PrimeQ] := Select[With[{id = IdentityMatrix@n},
     Pick[id, #, 1] & /@ Boole@PrimeQ@Partition[Range[n^2], n] // 
     Tuples], isGood];

So

Row[MatrixForm /@ #] & /@ 
  Composition[getAllSolutions, Prime]~Array~4 // 
 Column@Riffle[#, "New prime"] &

Gives

This is neither elegant nor smart nor memory efficient. It is a brute force method to get all solutions of a given size

isGood[m_] := Sort@m === reye@Length@m;
i : reye[l_] := i = Reverse@IdentityMatrix@l;

getAllSolutions[n_?PrimeQ] := With[{id = IdentityMatrix@n},
     Pick[id, #, 1] & /@ Boole@PrimeQ@Partition[Range[n^2], n] // 
     Tuples]~Select~isGood;

So

Row[MatrixForm /@ #] & /@ 
  Composition[getAllSolutions, Prime]~Array~4 // 
 Column@Riffle[#, "New prime"] &

Gives