Bah...I finally got it off. I just needed a 5 minute break a a cold beer. For those that search and find this, you need to remove the battery (for working space), and fuse box (3 bolts). You then push the tab next to the 80a fuse AWAY from the fuse. I thought I was going to snap the tab I was pushing so hard, but it finally released the fuse and fuse holder. The holder for the 80a and 60a fuses comes out.

Hopefully Autozone will have a replacement