Maybe ask them to provide you with a Linux cli only bastion? Then you’ve got a lot of options, it costs almost nothing, and it’s even better security wise.
My plan is to find a solution that complies with their security standards (i.e. through AWS’s authentication spec)
I think SSO is your best bet, if you use identity center.
I’m not sure what you use by workspaces, I haven’t touched windows in a while.
Wouldn’t a bastion with SSO do the same thing? In both cases OP needs to pass AWS based security checks in order to ssh from the bastion instance. And both options can be locked down by enterprise standards.