centos uninstall

Hi , Let's say I want to uninstall a package in centos. I used yum remove command and it shows the package are removed . However, I still see the files are there. So I understand that uninstalling does not remove the package and files together the way we uninstall in windows. Is that correct...