Several justices, including some liberals who seemed open to gay marriage, raised doubts during a riveting 80-minute argument that the case should even be before them. And Justice Anthony Kennedy, the potentially decisive vote on a closely divided court, suggested that the court could dismiss it with no ruling at all.
Such an outcome would almost certainly allow gay marriages to resume in California but would have no impact elsewhere.
Kennedy said he feared the court would go into "uncharted waters" if it embraced arguments advanced by gay marriage supporters. But lawyer Theodore Olson, representing two same-sex couples, said that the court similarly ventured into the unknown in 1967 when it struck down bans on interracial marriage in 16 states.
Kennedy challenged the accuracy of that comment by noting that other countries had had interracial marriages for hundreds of years.
There was no majority apparent for any particular outcome and many doubts expressed about the arguments advanced by lawyers for the opponents of gay marriage in California, by the supporters and by the Obama administration, which is in favor of same-sex marriage rights.
Kennedy made clear he did not like the rationale of the federal appeals court that struck down Proposition 8, the California ban, even though it cited earlier opinions in favor of gay rights that Kennedy wrote.
That appeals court ruling applied only to California, where same-sex couples briefly had the right to marry before voters adopted a constitutional amendment in November 2008 that defined marriage as the union of a man and a woman.
Several members of the court also were troubled by the Obama administration's main point that when states offer same-sex couples civil union rights of marriage, as California and eight other states do, they also must allow marriage. The other states are: Colorado, Delaware, Hawaii, Illinois, Nevada, New Jersey, Oregon and Rhode Island.
Justice Samuel Alito described gay marriage as newer than such rapidly changing technological advances as cellphones and the Internet, and appeared to advocate a more cautious approach to the issue.
"You want us to assess the effect of same-sex marriage," Alito said to Solicitor General Donald Verrilli. "It may turn out to be a good thing. It may turn out to be not a good thing."
Charles Cooper, representing the people who helped get Proposition 8 on the ballot, ran into similar resistance over his argument that the court should uphold the ban as a valid expression of the people's will and let the vigorous political debate over gay marriage continue.
Here, Kennedy suggested that Cooper's argument did not take account of the estimated 40,000 children who have same-sex parents. "The voices of these children are important, don't you think?" Kennedy said.
If the court is to find the exit without making a decision about gay marriage, it has two basic options.
It could rule that the gay marriage opponents have no right, or legal standing, to defend Proposition 8 in court. Such an outcome also would leave in place the trial court decision in favor of the two same-sex couples who sued for the right to marry. On a practical level, California officials probably would order county clerks across the state to begin issuing marriage licenses to gay and lesbian couples, although some more conservative counties might object.
Chief Justice John Roberts and Justice Ruth Bader Ginsburg had the sharpest questions for Cooper on the issue of standing.
The justices also could determine that they should not have agreed to hear the case in the first place, as happens a couple of times a term on average. In that situation, the court issues a one-sentence order dismissing the case "as improvidently granted." The effect is to leave in place the appeals court ruling, which in the case of Proposition 8, applies only to California. The appeals court also voted to strike down the ban, but on somewhat different grounds than the trial court.
The Supreme Court waded into the fight over same-sex marriage at a time when public opinion is shifting rapidly in favor of permitting gay and lesbian couples to wed, but 40 states don't allow it.
The court's first major examination of gay rights in 10 years continues Wednesday, when the justices will consider the federal law that prevents legally married gay couples from receiving a range of benefits afforded straight married Americans.
The courtroom was packed on Tuesday and the crowd included actor-director Rob Reiner, who helped lead the fight against California's Proposition 8. Some people waited since Thursday - even through light snow - for coveted seats for the argument.
Both sides of the case were represented outside the courthouse. Supporters of gay marriage came with homemade signs including ones that read "a more perfect union" and "love is love."
Among the opponents was retired metal worker Mike Krzywonos, 57, of Pawtucket, R.I. He wore a button that read "marriage 1 man + 1 woman" and said his group represents the "silent majority."
Same-sex marriage is legal in nine states and the District of Columbia. The states are Connecticut, Iowa, Maine, Maryland, Massachusetts, New Hampshire, New York, Vermont and Washington.
Thirty states ban same-sex marriage in their state constitutions, while ten states bar them under state laws. New Mexico law is silent on the issue.
Polls have shown increasing support in the country for gay marriage. According to a Pew Research Center poll conducted in mid-March, 49 percent of Americans now favor allowing gays and lesbians to marry legally, with 44 percent opposed.
The California case was argued 10 years to the day after the court took up a challenge to Texas' anti-sodomy statute. That case ended with a forceful ruling prohibiting states from criminalizing sexual relations between consenting adults.
Kennedy was the author of the decision in Lawrence v. Texas in 2003, and he is being closely watched for how he might vote on the California ban. He cautioned in the Lawrence case that it had nothing to do with gay marriage, but dissenting Justice Antonin Scalia predicted the decision would lead to the invalidation of state laws against same-sex marriage.
Kennedy's decision is widely cited in the briefs in support of same-sex unions.
The California couples, Kris Perry and Sandy Stier of Berkeley and Paul Katami and Jeff Zarrillo of Burbank, filed their federal lawsuit in May 2009 to overturn the same-sex marriage ban that voters approved the previous November. The ballot measure halted same-sex unions in California, which began in June 2008 after a ruling from the California Supreme Court.
Roughly 18,000 couples were wed in the nearly five months that same-sex marriage was legal and those marriages remain valid in California.
The high-profile case has brought together onetime Supreme Court opponents. Olson, a Republican, and Democrat David Boies are leading the legal team representing the same-sex couples. They argued against each other in the Bush v. Gore case that settled the disputed 2000 presidential election in favor of George W. Bush.
On the other side Tuesday was Cooper, Olson's onetime colleague at the Justice Department in the Reagan administration.
The case is Hollingsworth v. Perry, 12-144.
Associated Press writer Jessica Gresko contributed to this report.