An Ulam Spiral is quite an interesting construction, revealing unexpected features in the distribution of primes.
Here is a related topic with one answer by Pinguin Dirk, who has provided one approach.
I've created a faster implementation, but I believe it can surely be speeded up:
ClearAll[spiral]
spiral[n_?OddQ] := Nest[
With[{d = Length@#, l = #[[-1, -1]]},
Composition[
Insert[#, l + 3 d + 2 + Range[d + 2], -1] &,
Insert[#\[Transpose], l + 2 d + 1 + Range[d + 1], 1]\[Transpose] &,
Insert[#, l + d + Range[d + 1, 1, -1], 1] &,
Insert[#\[Transpose], l + Range[d, 1, -1], -1]\[Transpose] &
][#]] &,
{{1}},
(n - 1)/2];
spiral[5] // MatrixForm
spiral[201] // PrimeQ // Boole // ArrayPlot
This is good enough for my purposes, and I don't have time to focus on this now, However, it might be interesting for future visitors, so the question is how to make it faster by improving/rewriting.
Note: There may be confusion concerning whether I need a matrix populated with primes or one populated only with 0 and 1, 1 indicating the prime positions. Let's assume it doesn't matter. Maybe someone has a neat idea for the first representation that does not look good in the second one, so I'm leaving it open.