Remove purity-related unsafe blocks

When pure is gone (#5384), the libraries should be reviewed for uses of unsafe that were only there for purity reasons. #1913 might make this easier.

One instance I just came across is in core::vec::map.