Hands shouldn’t go through objects. Grasping and interaction should feel natural like real world objects. Things should seem like they have weight with some requiring two hands to move. This kind of interaction is important and creates a sense of presence.
Hurricane VR is a physics interactions framework that has everything to achieve this. There is also a companion physics based rigid body controller that integrates with it called HexaBody VR. Together they form a starting point for BONEWORKS or Blade & Sorcery levels of interaction.
I have been working with both frameworks and highly recommend them. There is a significant learning curve with them but they are well written and actively being developed with a great community on discord.
These assets are definitely bringing next level interaction to VR developers!