First, precompute RegionMemberFunction
for better performance. Second, increase MaxPlotPoints
.
regMem = RegionMember[Polygon[...]];
Show[ListContourPlot[...,
RegionFunction -> Function[{x, y, z}, regMem[{x, y}]],
MaxPlotPoints -> 50]
]

Alternatively, if you don't want all of your points to get additionally resampled at a grid (which is what increasing MaxPlotPoints
does and can be observed with Mesh -> All
), you can trick Mathematica by providing RegionFunction
of a region which is very slightly smaller than your actual region of datapoints. This will trigger finer resampling only at the boundary (observe with mesh). I do not know whether this can also be achieved in any other way ...
reg = Polygon[...];
smallerReg =
GeometricTransformation[reg,
ScalingTransform[{1 - 1/100000, 1 - 1/100000}, RegionCentroid[reg]]];
regMem = RegionMember[smallerReg];
Show[ListContourPlot[...,
RegionFunction -> Function[{x, y, z}, regMem[{x, y}]]
]
