Chris Rock has a pretty hilarious stand-up routine about high allergy foods, and how allergy foods are something that can happen only in the western world where we have such abundance that our bodies can actually reject certain sources of nutrition. As he put it, "Do you think anyone's running through Ethiopia saying 'I can't have dairy, I'm lactose intolerant!'?" Unlikely, and keenly observed on his part.
However, with apologies to Mr. Rock, it's a simple fact that the human body has evolved to process certain foods more efficiently than others. Lactose - milk and milk-based products - are a good example. If you look in nature, you'll see that mammals generally stop weaning by the time they're the human equivalent of four years old. Generally we stop weaning our kids well before that, yet as a society we still drink milk and have milk products (generally cow's milk) much later in life. This in spite of the fact that we are not really evolved to do so.
The side effects of high allergy foods like dairy are easy to see. Humans simply absorb more fats from these foods than we would from other foods of similar nutritional value - vegetable oils come immmediately to mind in that regard. Both cow's milk are high in fat, but your body extracts the good fats from olive oil much more easily and readily than it does from cow's milk. Your body also burns those calories easier, and generates more energy from vegetable oils than it would from a similar amount of dairy calories.
Now, to be clear, this is actually more of a food "intolerance" than a food allergy. We are evolved to process certain foods better than others, and the "others" can cause an adverse reaction that's more of a nuisance than a legitimate medical problem.
An allergy is a serious medical problem, but one that effects a far smaller number of people than food intolerances. While as many as 45% of adults have some form of lactose intolerance, it's estimated that only one to three percent of the population suffers from a legitimate food allergy foods.
Most common among high allergy foods are wheat and gluten, as well as various allergies to various types of nuts. My niece is among those with a peanut allergy, for example, and in her case even though it's a relatively minor allergy, it's still good policy to bring an epi-pen just in case she's exposed to something that triggers a reaction.