A scalable, fair and transparent system for arbitrating user-submitted "fact-checks" would be:

Some subset of the site's users sign up to arbitrate fact-checking submissions (what are referred to as "Community Notes" on Twitter/X).
Whenever a user submits a fact-check/Community Note, a fixed number of other users (say, 10) are selected to vote on whether the fact-check is valid or not. (If enough other users vote that the fact-check is valid -- which could be a simple majority, or some other threshold like 7/10 -- then it is "upheld", and the fact-check is displayed alongside the post.)
Before "earning" the right to submit a fact-check, the user has to participate in 10 votes on fact-checks submitted by other people. By enforcing this ratio, this ensures that there are always enough voters to vote on the fact-checks that each user submits.

From this point on, I'll discuss this hypothetical fact-check system as it would apply to Twitter/X specifically, because: (a) Twitter has already set the precedent of letting users (rather than company employees) submit Community Notes and vote on whether they are valid, but (b) existing evidence shows that this is not effective. But, with the changes described above, it could before.

First, the existing countermeasures against misinformation do not work. Any user can simply post a reply to something that is obviously false or misleading, and I don't know of any systematic studies of this, but anyone who has tried this against a high-profile account has probably seen anecdotal evidence that it's ineffective:

On March 18, 2024, Twitter user @BehizyTweets tweeted, "The DOJ has charged Influencer Isabella De Luca for touching a table that came out of the Capitol during January 6th. I repeat, she got charged for TOUCHING A TABLE." I replied:
"The article at wusa9.com/article/news/n... says: 'According to charging documents, DeLuca can be seen climbing through a broken window into an unoccupied congressional office. Once inside, investigators say DeLuca assisted other rioters in passing a table out of the window.' The article includes pictures of her climbing out of the window. Your post only shows a picture after she’s already standing outside the window."
The reply got 11,000 views and 67 likes, compared to 3 million views and 25,000 likes for the original (incorrect) tweet, and no Community Note was ever attached to the original.
On March 6, 2024, Seattle radio host Ari Hoffman tweeted: "Hamas supporters plan on protesting over the freeway during the funeral procession of fallen WA State Trooper Christopher Gadd." Many right-wing commentators falsely refer to people rallying against the bombing in Gaza as "Hamas supporters", but the more salient falsehood in this case was the implication that protesters were planning to disrupt the cop's funeral. The tweet was quoted hundreds of times, including by Libs of TikTok creator Chaya Raichik.

As I said in a reply:
"This post is obivously intended to deceive people into thinking that this protest was planned to disrupt the cop's funeral. In real life, the Boeing protest was scheduled for March 12th weeks before Christopher Gadd was killed, to coincide with the Boeing conference happening in Seattle; and the protest is 30 miles away from where the funeral is being held in Everett."
However my reply got less than 1% of the views of the original post (1,137 views for the reply vs. 157K for the original).
A tweet from Jonathan Chait linked to his article in New York Magazine which stated: "The group [Students Allied for Freedom and Equality, the University of Michigan chapter of Students for Justice in Palestine] vandalized his [a Jewish regent's] law office." But this statement was linked to an article that only said that the law office had been vandalized; it didn't say police had identified the group responsible.

I posted this reply saying:
You wrote, "The group [Students Allied for Freedom and Equality, the University of Michigan chapter of Students for Justice in Palestine] vandalized his [a Jewish regent's] law office." However that just links to this article:
freep.com/story/news/loc...
which says the vandalism occurred but there's no evidence of who did it. How can you blame a specific student group without evidence?
The reply got 1 like and 154 views, compared to 137 likes and 29,000 views for the original. (Note this example also raises the issue of how to handle the case where a tweet links to an off-site article, and the false statement is contained in the off-site article but not in the original tweet. But this is a moot point for now, since Twitter isn't even handling cases where the falsehood is in the original tweet.)

Since replying to misinformation tweets by itself does not work, what about submissions to Community Notes? I applied for the Community Notes program in October 2023 but never received a response (other users have reported similar experiences). After some searching, I found an article by someone who had been admitted as a Community Notes volunteer (who is remaining anonymous for the purpose of this write-up), so I messaged them and asked them if they could submit a Community Note for three tweets that I had found:

Tweet Proposed Community Note Date Proposed

A tweet from conservative outlet The Post Millenial stating: 'TPM's @thehoffather speaks with a Christian preacher who was arrested at the UW Gaza camp after police let hordes of pro-Palestinian agitators walk free: "I just thought it was completely hypocritical and it was an injustice."' "Street preacher Matthew Meinecke was camping during the same period as the pro-Palestine protesters, and the university offered everyone the same deal: leave the camp by May 20th and you won't get any citations for camping violations. Everyone else left, but Meinecke refused to leave, so he was arrested. He was not treated differently because of his beliefs." May 25, 2024

An image post stating that "Gays make up 1% of the population and 33% of the child molesters". "The percent of LGBT adults in the U.S. exceeds 7%, and academic literature says that researchers have 'failed to find a connection between homosexuality and child molestation'." [Source 1, Source2] June 13, 2024

The COVID vaccine hasn't saved a single life." "COVID vaccinations saved an estimated 14.4 million lives from 2020-2022." [Source] June 21, 2024

None of the proposed Community Notes resulted in any action being taken. The volunteer told me that if a submitted note doesn't happen to get any attention, it "just sits there" without getting any more votes. This echoes the experience of Slate editor Nitish Pahwa as a Community Notes volunteer -- where he reported that the most common response to the Notes he submitted was not that they were rejected, but that they got no engagement at all from other volunteers.

This, in turn, mirrors the "pile-on lottery" effect of social media generally, including the example I go to most often: I post a lot of silly jokes on Reddit, and once in a while one of them blows up to 50,000 upvotes, but most of them "just sit there" because they don't gain "traction", and it has nothing to do with the post quality. And it's easy to see why some of the same fallacies which would cause people to believe that social media "works" meritocratically, would also cause them to believe that Community Notes works meritocratically as well:

survivor bias (you see the times that Community Notes works, not the times it doesn't)
the just-world hypothesis (people want to believe that lies get corrected, because that would be "fair")
"truth-default" (Musk's image has been tarnished a lot, but he's not dumb, and if he says Community Notes works, some people default to believing him)

Solution: Peer-ratio-enforced fact-checking

But these problems (in particular, the problem of a fact-check "just sitting there" with nobody acting on it) go away if you enforce a rule that every user has to participate in 10 votes on other fact-checks, for every fact-check that they submit of their own. As long as the user base is relatively stable (or growing), this ensures that there will be enough people available to vote on the fact check.

(Whether or not this guarantees people actually will vote on the fact-check, depends on the implementation. If all newly submitted fact-checks simply go into a pool of "unadjudicated" fact-checks, and if users are free to browse that list and choose the ones they want to vote on, then this risks users ignoring the more boring-looking fact-check in favor of the ones that look more interesting. My recommendation would be to require voters to handle submitted fact-checks in a First-In-First-Out fashion -- you are automatically assigned to vote on the most-recently-submitted fact-check that doesn't have enough votes yet. Voting on incorrect or badly written fact-checks is part of the job of being a volunteer; that way, the people submitting those fact-checks receive feedback that they have room for improvement.)

Another advantage of peer-ratio-enforced fact-checking is that you can "spend" your fact-checks on any post or comment that you want, even a low-traffic post from a user with very few followers. By contrast, in the existing system, suppose for the sake of argument that you're a Community Notes volunteer that has identified a post from a low-profile user that is clearly wrong. On the other hand, you feel that the time of the other volunteers is a precious and limited resource, and you wouldn't want to "waste" it on a low-profile post that won't do a lot of harm anyway. A peer-ratio-enforced fact-check system mitigates that dilemma, because by participating in 10 votes for other people's fact-checks, you have "earned" the right to submit a fact-check of your own, and you can "spend" it however you want. (That is still generating labor for other people, of course -- they have to read your fact-check and vote on it -- but you can feel less guilty because you've traded your labor for their labor.)

Further, you could implement a system that would provide real-time online fact-checking -- ensuring that when you submit a fact-check, the system pings users who are online at that moment, and requires (or strongly encourages) them to adjudicate the fact-check in the next five minutes. While this sounds at first like a costly imposition on other users, you can "earn" the right to this service by providing it to others, and then "spend" it when you want to. In other words: you can log in and mark yourself as being available for real-time fact-checks. During that time, if the system pings you and asks you to vote on a fact-check, you have to complete it in the next five minutes. After you have done that 10 times, you have earned the right to a "real-time fact-check". Then if you ever submit a real-time fact-check, the system will ping 10 other users who submit their votes within five minutes, and you'll have your result in five minutes as well.

I maintain this would be the first social media fact-checking system that meets all of these criteria:

Anyone* can sign up and submit a fact-check. [*If the site decides to limit signups to people whose accounts have existed for some fixed time period, or who pass some other verification check, then even if this does not include all users, it will include enough users that any false statement published on the site will probably be seen by at least one of them.]
Any submitted fact-check will get voted on properly by other users instead of being ignored.
By having other fact-checkers vote separately and independently on whether the fact-check is valid, we achieve the greatest possible chance of getting the "right" answer.

See also "Beyond fact-checking" - if Community Notes / fact-checking ever does become enforceable at scale, there are ways to guard against people trying to slip through loopholes in the rules.

ADDENDUM: A way to test "scalable fact-checking" in an offsite system, so you can submit fact-checks to both the real Twitter and the offsite system at the same time, and see which system produces better results -- all without changing a single line of code in Twitter itself.

sorv

Solution: Peer-ratio-enforced fact-checking