Been a while since I did any serious graphics work, but how about:
1. Calculate (not draw) lines between all points.
2. Drop any lines which cross (at some point other than original vertices). That is, if two lines cross, drop both of them.
3. Calc points from remaining lines.
If there's a concave point, however, that won't work. One possibility for *that* problem would be to calc your triangles and drop any vertices which are contained within a triangle.
These are just ideas off the top of my head.