Brown has been outgunned by the personal wealth of both candidates, who are dueling it out in TV ads.
Brown wants a prime-time, three-way debate before the primaries on June 8.
Poizner accepted, but Whitman turned it down.
She leads in GOP polls and she already debated Poizner last month.