it happened to me twice, i just fall through a hole and return to my home. I wasnt pressing action (for those who gonna say “you probably was pressing action and fell in front of a statue”).
Sometimes i fall through a hole and appear uppon a wall, i have to walk (its really dangerous sometimes u cant move and have to break the wall, imagine if you are surrounded by mind flayers)