Very simple; I want to deny access to part of a layout unless a specific checkbox set is marked as "Yes". How would I go about doing this?

Since you have vs. 11, create a self-join table occurrence on x (cartesian product). Then create a one-row portal of the self-join table and make it large enough to hold your 'part of the layout'. Filter the portal to boolean calculation of:

checkbox = "Yes"

This is known as visibility and is easy in vs. 11. BTW, make the portal transparent with no lines. Then the fields will pop onto the layout when the checkbox is checked and disappear when it isn't.

The object that I'm actually trying to hide is a portal. Does FMP 11 allow a portal to display inside of a portal?

You can't have a portal inside a portal.

But you can put your portal in a tab control with no visible tabs and use an OnLayoutLoad or OnRecordLoad script trigger to go to the correct tab panel.

Yup, the tab trick worked great.