But a polygamist, or a polyandrous, person could say, "Why am I being denied a right to marry in my choice?" Where is the compellling intererestin restraining the rights of the polygamist? I can't see one, if there is no compelling interest in restraining the rights of homosexuals.
The way you've phrased that is odd. Either you see a compelling interest in restraining the rights of the polygamist or you don't. Whether or not there's such an interest in restraining the rights of homosexuals shouldn't factor into that.
For my own part, I wouldn't be the least bit bothered by legal polygamy, but I can see several arguments against it that don't rely on anything like a religious agenda.
The biggest of those is simply the difficulty of making the change, as we discussed. We'd be talking about a vast undertaking in revising all sorts of public and private rules and laws, to accommodate a very tiny portion of the population, and a long period of turmoil as we sorted through those issues, most of which couldn't be anticipated until they popped up. Just to give another example, picture a situation where three people are married, and one decides that she doesn't want to be married, any more, to one of the other two, but she does to the other. Would that dissolve the entire plural marriage, just oust her from it while leaving the other two, or would it create a "chain marriage" (where two people are married to the central link in the chain, but not to each other -- creating yet another form of fundamentally different marriage for the law to cope with). The changes in divorce property laws, alone, would become a full time job for an army of lawyers for years.
But, even if we could rub a magic lamp and have a genie rewrite all our laws, legal precedents, and private policies to accommodate the change, there'd still be serious social concerns. The population of potential participants in plural marriage is pretty much all of us -- we all have non-monogamous natures. By comparison, the population of potential participants in gay marriage is quite small -- gays and bisexuals, for practical purposes. So, while legalizing gay marriage couldn't realistically be expected to have fundamental impacts on the organization of society, legalizing plural marriage could.
This would most likely take the form of violence stemming from males who are unable to find mates. If historical polygamous societies are any guide to human tendencies, high-status and wealthy males would collect harems of females, creating a shortage of mating opportunities for low-status and poor males, who then become a destabilizing force within society. Polygamous societies often become hyper-violent, with clan wars and street violence beyond anything we see in the US today. Often they become extremely expansionist, as low-status males happily risk life and limb for the chance of raising their status enough to mate (and the powerful happily give them the opportunity, since the wars allow a way of "soaking up" that excess male population.)
We wouldn't necessarily go down that road. But you needn't be someone with a prudish objection to polygamous sex to worry seriously about such things.
there is no reason an adult brother couldn't marry his adult sister
The only non-religious argument I can see against it is based on the potential impact that the possibility has on young people. If society entertains the possibility of marriages between brothers and sisters in adulthood, it might make underage brothers and sisters more likely to head in that direction, which can be a serious problem because typically they'll be different ages, creating the risk of exploitation and abuse of uneven power. That's even a bigger concern with parent/child relationships, where a parent who knows a child is a long-term option as a mate might be more likely to abuse his position of authority, consciously or not, to set the child on a path of thinking about him that way.
So, if I were to allow incestuous marriage, it would be only in very rare situations, where reproduction was impossible AND the couple did not know each other when either of them was still a minor. That would take away any risk that society extending marriage to incestuous couples would encourage the abuse of minors. So, then, if you had siblings who never knew each other until adulthood and fertility wasn't an issue, they'd be welcome to get married if they wanted.
There are a number of crypto-Mormons who would disagree with that statement. Ditto for many Muslims, whose traditional cultures all included polygamy
I'm not singling them out. A vast number of cultures in history have had plural marriage (it's common in the Old Testament, for example). That makes sense, since monogamy is so unnatural, biologically. A rule against polygamy is an equal-opportunity infringer. It's similar to the rule against marrying children.
Anyway, it'll be fun to watch social conservatives change their tune on this, in coming years. For quite a while, they've gotten into the habit of offering up devil's advocate arguments in favor of polygamy, because it was linked, in their mind, with gay marriage, and they thought that by portraying it as a slippery slope from one to the other, they could prevent us from taking that first step. But, they've lost. We're in the process of taking that first step. Before too long, gay marriage will be legal in all the advanced nations, including coast to coast in the US. The rules against gay marriage will go the way of the laws against miscegenation... you'll barely even be able to find anyone who will admit he EVER took the bigoted view on the topic. At that point, conservatives will need to switch gears and start arguing against polygamy.... saying that there's something about it that makes it fundamentally different from gay marriage, so that an argument against outlawing gay marriage doesn't naturally mean we have to legalize polygamy. It's the same way conservatives had to find way to distinguish between the anti-miscegenation laws they once supported (and now oppose) and gay marriage. I look forward to watching that transformation.