So what to make of a guy who seems quite interested after 1st date and then distracted on second date?

I would not try so hard to read into this guy. It was only the second time you got together and it is a Holiday weekend. I am sure he has a lot on his mind. See how it goes on the next "date" if there is one. Take it slow.