How can my opponent cast so many spells on my units?

You have a pool of 500 favor, but it can be raised to 550 if you win a world (i.e. you were a member of the first alliance that owned four completed Wonders on a world).
You can also save up on items like "Blessing of Favor", which instantly fills your favor pool.
In addition, you can also obtain the ability to steal favor from other Players through the research "Temple looting", which is also instantly added to your favor pool.