So I'm trying to go down the trapdoor behind the chaos temple altar in the wildy and it says "it is not safe to go here alone". WTH?

I believe they made Wilderness quests different as of the Wilderness & Free Trade update. This is from the news post, hope it helps :P

 

If you're doing Defender of Varrock or Curse of Arrav, speak to Captain Rovin or Hartwin upstairs in Varrock Palace to be escorted safely through the Wilderness.

 

That should get you down there hopefully :)

Figured it out. You have to get an escort now to get you safely through the wilderness thanks to new update. So I'm going to talk to Captain Rovin in Varrock.

Thanks Omnitec :)