A source told the outlet that the pro golfer's close friends voiced their "concerns" that his relationship with Vanessa would be bad for his public image.