why does my magic level go down for no apparent reason and then builds back up?

Where are you what you doing?

I can tell you for sure that it builds back up because all your stats that are not maximum, excluding Prayer and Summoning, recover 1 point each minute.

 

 

 

Why your magic level is dropping, I don't know. Some in-game activities cause you to lose magic levels, like being hit by a Seercull Bow's special during player-to-player combat.

You've probably been hit by a Dark Wizard. They can reduce your mage level. It's common for lower levels, but I'm only making an assumption. May other factors can contribute at well, ex. if you drank a Sara Brew.