Anything that disrupts or vectors the escaping gasses asymmetrically at the muzzle will probably have a POI shift, albeit very small. Hence the reason for careful machining on crowns of precision rigs. It would make plenty of sense to notice a POI shift in your scenario. I haven't tried shooting my XCR without a muzzle device, but it wouldn't surprise me if this happened to me as well.