During lunch, Tom knew there is something between Daisy and Gatsby from the way they look at each other.
Oh no! It seems like Daisy and Gatsby are having an affair.
As Gatsby was explaining to Tom that his house is directly across the bay from theirs, Daisy sensed that Gatsby was going to admit to Tom about his affair with Daisy, so she suddenly insists that they go to Manhattan immediately.
How about we go to Manhattan now?!
What's wrong with Daisy?
On the way to Manhattan, Tom explains to Nick and Jordan that he's been investigating Gatsby, and knows a lot about him.