Goats, as in Leviticus 16:21, 22, signify faith. Because man is regenerated by the Lord by the truth of faith, and consequently, his sins are removed and cast into hell, therefore it is said that "Aaron made the goat bear upon him all the iniquities of the children of Israel unto the land of separation, or into the wilderness." From correspondences, a goat signifies the natural man. The goat which was sacrificed, as in Leviticus 16:5-10, signifies the natural man regarding a part purified, and the goat which was sent into the wilderness regarding the natural man not purified.

(Reference: Apocalypse Explained 730)