For decades, Tom McCarthy has attempted to blow up the conventions of the realist novel. With his latest book, “The Rhyl Poster,” it seems that his approach may have reached its limit.