Under the circumstances, Jim hasn't been abandoned as much as he might assume. Look, Felix knows he's a smart man but Jim seems to end up in trouble with powerful entities more often than not. So he reappears at about this point, having acquired drinks for them both only to find Jim's already being plied with... something.

"And here I was about to recruit you as my test subject, Ji- oh!" He's just spotted who the captain's with. And now, despite the smile he gives Loki, his eyebrows are drawn together in the faint frown of a man who's pretty certain he's just been posed a trick question. "I beg your pardon, have we met?"