sorv

 sample of randomized voters

Home

About

Understanding the "pile-on lottery"

Advantages of the sorv system

How to implement scalable fact-checking

Why transparency at the algorithm level is not enough

Using sorv to fight social-media-induced depression

With sorv, government wouldn't need special privileges to report rules violations

Scalable fact-checking: Proof-of-concept using an offsite system

On a separate page I described a proposal for scalable fact-checking (using X/Twitter as an example, since they have already set the precedent of letting regular users submit fact-checks, but their existing system fails to catch most false statements):

  • Users sign up as volunteers to submit fact-checks and vote on other users' fact-checks.
  • In order to earn the right to submit your own fact-check, you have to first vote on 10 other fact-checks that have been submitted by other users. Thereafter, the ratio continues to be enforced that for every 1 fact-check you submit, you have to vote on 10 fact-checks submitted by other users.
  • If some threshold of users (say, 7 out of 10) vote that a fact-check is valid, then it is "upheld" and displayed alongside the original post/comment.
Below is a description of how a proof-of-concept could be implemented completely separately from the existing social media site (again using X/Twitter as an example). A separate site could be set up where users submit proposed "fact-checks" on existing tweets/comments, other users vote on whether they are valid, and then people can review the results of the experiment to see if it achieved better results than X/Twitter's existing system. All of this could be done without changing a line of code in Twitter/X itself.

The offsite system

The experiment could work as follows:

  • Users sign upon on the offsite system as fact-checkers.
  • Since the system is enforcing a rule "you have to vote on 10 other people's fact-checks before you can submit a fact-check of your own", in order to bootstrap the system you have to "seed" it with some initial fact-checks, which can be submitted by a subset of trusted users. (Alternatively, you can simply say that for e.g. the first 24 hours, any user can submit a fact-check even if they haven't voted on anyone else's fact-checks yet, perhaps up to a limit of 5. There is limited benefit to submitting a junk fact-check because it will just get shot down when other people vote on it.)
  • Once the ratio rule is in place, each user has to vote on 10 other fact-checks for every 1 fact-check that they submit themselves. Additionally, to ensure that no fact-check "languishes" without being voted on, we can also enforce a rule that voters are assigned to vote on the most-recently-submitted fact-check that hasn't received enough votes yet.
  • If some threshold of voters (say 7 out of 10) agree with the fact-check, then it is marked in the offsite system as "upheld".

All of this, however, takes place solely within the offsite system, so the fact-checks that are "upheld" do not get displayed alongside the real posts on the X/Twitter site.

At any point in the experiment, we can review the results and compare them against the real X/Twitter site:

  1. For each fact-check that was upheld by the offsite system, was there ever a corresponding Community Note displayed on the real post on X/Twitter?
  2. In cases where the two systems produced different results (i.e. a fact-check was upheld by the offsite experimental system, but there was no corresponding Community Note attached to the real post on Twitter), we can look closer to decide if the fact-check was "correct". This will obviously be controversial in many cases, but the parties running the experiment can leave it to the public to decide. In other words, you can take a list of 1,000 cases where the offsite experiment upheld a fact-check on a post where there was no corresponding Community Note on the real post on X/Twitter. You randomly pick 5 of those cases to take a closer look at. If, in all 5 of those cases, you form the opinion that the fact-check was valid, and if none of those posts had a Community Note on the real X/Twitter, then you'd tend to form the conclusion that the peer-ratio-enforced system was working better than the "real" Community Notes system.

One caveat is that if the experiment is done publicly in real time, then it is possible, of course, that a Community Notes volunteer on the real X/Twitter might notice the fact-checks that are being upheld in the offsite system, and would submit them into the real Community Notes system with a note saying, "Hey, this fact-check was upheld as valid by 8 out of 10 voters in the offsite experiment, so maybe we should vote up this Community Note too." This person may be well-intentioned, but it would skew the results of the experiment, because the point is to determine if the peer-ratio-enforced system works better than X/Twitter's existing system, and it defeats the purpose if the valid fact-checks in the offsite system are also being copied over to the real site. Possible countermeasures would include:

  1. Hiding the data in the offsite experiment from the public until the experiment was concluded, and asking the participants in the experiment not to leak the data or copy the fact-checks into the real Community Notes. However this assumes total cooperation from everyone who has access to the data. It may also mean less interest from the public if they cannot watch the experiment unfold in real time.
  2. Hiding the data from everyone including the participants themselves, other than the fact-checks that they have submitted or the ones that they have assigned to vote on. (This would of course mean that the system would have to assign fact-checks to users; they wouldn't be able to browser a list of all available fact-checks that need votes.) The downside of this is that it might make the whole experiment less interesting to participants, if they can only see the fact-checks that they are required to vote on.
  3. Do nothing -- simply live with the fact that the upheld fact-checks may be copied over to the real X/Twitter site. At the end of the experiment, we can look at the timestamps showing when the fact-check was first submitted in the offsite experiment, and compare to the timestamp showing when the Community Note was first submitted to Twitter. If the vast majority of the time, the fact-check was submitted to the offsite system first, then this strongly suggests that at the very least, the offsite system was faster at detecting falsehoods, and most of the time, the falsehood probably never would have been detected by X/Twitter at all.

If the experiment shows that the peer-ratio-enforced system was better at fact-checking than X/Twitter's own system, that would be a strong argument for implementing the peer-ratio-enforced system in X/Twitter itself.

Alternatively, we could simply leave the "offsite experiment" up and running in perpetuity, and encourage Community Notes volunteers to look for fact-checks that are upheld in the offsite system, and copy them over to be submitted as Community Notes in the real system. If enough Community Notes volunteers were interested in using the results of the offsite experiment, this would also avoid the problem of valid Community Notes "languishing" without getting attention, since an entire subset of Community Notes volunteers would be paying attention to the proposed Community Notes that are copied over from the offsite system.