That sound like a process keeping alive on your mountpoint directory. Can be a lot of process like dirmngr, haveged and so on.
Try lsof command to find the process like
# lsof /mountpoint
or
# lsof +D /mountpoint
find the process and kill it before unmounting your directory