The Doctor made an uncharacteristic statement (assumption) about a group of people in the past implying they are sub-human savages.  He’d only just met them and they were leading them to a place to stay, being kind, and trying to protect them from danger.  Nope.