The change in phrasing is what he picks up on, because 'ultimately' feels like Bruce feels that he knows what will happen. And Clark is well aware that Bruce is a damn pessimist. And when it involves the two of them, he... can't really understand why. He wonders if Bruce knows how much hope, how much... how much drive comes from what they share? From his relationship with the other man? He's reasonably certain that if he has an inkling, he probably gives him, Clark, too much credit.

"What does 'ultimately' mean here? Why do you think you'll 'ultimately leave anyway'?"

Do you think I'd ever kick you out? Do you think there's anything you could ever do to make me turn my back on you?