LDAP Groups can give permissions to different sites.
When a user is in the Skyrim Group, they have access to the Skyrim Site, but not to the other sites.
Phil Cordier commented
Strong support for this idea. You should be able to configure an LDAP group to hold only those members that you wish to have access to a specific site. The functionality of the current group mechanism is not of much use, at least to me, otherwise.
Simon Welling commented
I would suggest a separate group access settings page on blog level. So all the connection settings on network level, all extra (custom) settings on blog level.