Check the value of $DISPLAY. If it's not set, you're safe to assume a non-GUI environment, though you might want to check to see if you're attached to a terminal or not.

If $DISPLAY is set, you'd then do an appropriate GUIfied response. If that fails, say because $LUSER is an idiot and isn't running an actual X11 session, then you would abort miserably, complaining loudly to stdout with a nonzero exit status.

Of several programs I've looked at lately, testing $DISPLAY to see that there is/isn't a GUI session attached is occasionally omitted. It's a seriously brain-dead mistake. SAS does it in its v8/v9 incarnations. The user has to provide an extra flag to the program to tell it it's not in an X11 session. This is Bad Code™