Just work in local coordinates for the polygon's plane. This requires finding a change of basis matrix and then applying it (which is just matrix multiplication).
One way to obtain a basis is to use any three points on the polygon, assuming they are not collinear. Thus:
basis[{x_, y_, z_, ___}] := Orthogonalize[{y - x, z - x}] // Transpose;
(Orthogonalize
is not really necessary but it assures the transformation preserves distances and angles for applications that might need that.)
Right-multiplication by basis[...]
converts 3D coordinates to 2D coordinates (ignoring any component of the 3D coordinate orthogonal to the polygon's plane). Then apply whatever 2D algorithm you prefer.
Example
Let's generate a random 3D polygon and another random point:
{vertices, p} = Through[{Most, Last}[RandomReal[NormalDistribution[0, 1], {4, 3}]]];
To illustrate the use of basis
, here are the 3D and local 2D renderings of this configuration:
Graphics3D[{Polygon[vertices], PointSize[0.02], Darker[Red], Point[p]}]
With[{a = basis[vertices]},
Graphics[{Lighter[Gray], Polygon[ vertices . a], Darker[Red], PointSize[0.02], Point[p . a]}]]

One check that the point actually is in the plane of the polygon is achieved this way:
inPlane[p_, basis_, origin_] := Abs[Det[Append[Transpose[basis], p - origin]] ] < 10.^(-12)
The origin
must be a point known to be in the plane (such as one of the polygon's vertices). E.g.,
inPlane[#, basis[vertices], First[vertices]] & /@ Append[vertices, p]
{True, True, True, False}
verifies that the triangle does lie in its plane and, as it happens, this particular point does not.
FindGeometricTransform
. $\endgroup$Cross
product of two vectors in the plane, and then useRotationMatrix[{n,{0,0,1}}]
to get a rotation matrix which will rotate everything into the xy plane. $\endgroup$