Psychological thriller is a popular subgenre of thriller in which the battle between protagonist and antagonist is primarily played out in a battle of wits, rather than a physical confrontation. These books are often dark and disturbing, and explore the unstable emotional state of the characters.
Well-known psychological thriller writers include Thomas Harris and James Patterson. A recent, more literary example is Gyllian Flynn's Gone Girl.
If you write psychological thriller, explore the subgenre to connect with other like-minded writers, workshop, and publish your book.