To be any good magically yourself means studying well, constant practice to improve and hone the skills naturally inherited in the blood as well as taking the time and making the effort in developing new ones. Each generation of a species is expected to be better than the one before. And there is rivalry to be the best of your own kind too.
And when your world has been at conflict with other worlds, as my Fairy Kingdom has been, there are other disadvantages. Those who have been beaten by it are likely to resent magic and those who practice it and such would consider any magical being to be fair game.
And if a magical species is tired, ill or run down, their powers will wane, making them more vulnerable to attack from those without such powers. So being magical is not all a question of waving wands about at will. Nor it is always much fun.