Not a solution but too big for a comment. There seems to be a catastrophic failure in Eigenvalues happening that is not due to the matrix being crazy. As a diagnostic, let's calculate the smallest (by absolute value) eigenvalue of the upper-left $n\times n$ part of the matrix

M = mat[xlist[[3]]];

For odd $n$ the answer is zero, so let's only do this for even $n$. We do this in two ways

1. Calculate all eigenvalues and pick the one with the smallest absolute value:

    e1[n_?EvenQ] := M[[;; n, ;; n]] // Eigenvalues // Abs // Min

2. Calculate only the smallest eigenvalue (by absolute value) with the Arnoldi algorithm:

    e2[n_?EvenQ] := Eigenvalues[M[[;; n, ;; n]], 1, 
      Method -> {"Arnoldi", "Criteria" -> "Magnitude", "Shift" -> 0}] // First // Abs

Method (2) is very reliable, whereas method (1) breaks down for $n=358$ and above:

Considering that the Arnoldi algorithm has no problems with this matrix, there seems to be something really strange going on in method 1.

Not a solution but too big for a comment. There seems to be a catastrophic failure in Eigenvalues happening that is not due to the matrix being crazy. As a diagnostic, let's calculate the smallest (by absolute value) eigenvalue of the upper-left $n\times n$ part of the matrix

M = mat[xlist[[3]]];

For odd $n$ the answer is zero, so let's only do this for even $n$. We do this in two ways

1. Calculate all eigenvalues and pick the one with the smallest absolute value:

    e1[n_?EvenQ] := M[[;; n, ;; n]] // Eigenvalues // Abs // Min

2. Calculate only the smallest eigenvalue (by absolute value) with the Arnoldi algorithm:

    e2[n_?EvenQ] := Eigenvalues[M[[;; n, ;; n]], 1, 
      Method -> {"Arnoldi", "Criteria" -> "Magnitude", "Shift" -> 0}] // First // Abs

Method (2) is very reliable, whereas method (1) breaks down for $n=358$ and above:

Considering that the Arnoldi algorithm has no problems with this matrix, there seems to be something really strange going on in method 1.

$Version
(* 12.0.0 for Mac OS X x86 (64-bit) (April 7, 2019) *)