The evacuations are in Cudahy, which is about six miles from the temple in Oak Creek.
Police have roped off four blocks in a neighborhood with a mix of duplexes and single-family homes, as well as one house.
FBI agents are there with an armored truck, a trailer and other vehicles. Other law enforcement officers are there too, along with a police dog.
Fifty-eight-year-old Ken Kacner lives about a block from the house that police have targeted.
Kacner says neighborhood has changed over the years and has become more dangerous.