My compass also drops if i inventory drop a weapon in to my hands, this is not realism its frustrating, If you want "realism" allow me to paracord it to my neck so it will never drop on swap.

This is a silly design choice it is just to frustrate and not to improve anything.

You need to take some responsibility its your credibility or find a better solution.