Reputation systems are a very popular way for people to attempt to address certain very difficult realities of how the internet works. Given a network that puts in contact everyone from the most naive potential victims to the most canny predators, how do you keep safe the one population from the other? Many people think that a reputational system - that is, having various people vouch for or condemn others - is a suitable solution, but unfortunately those systems are vulnerable to a certain class of attacks.
This has become more prominent with recent news of certain apps and other projects that are intended to take crowd-sourced reports of personal behavior and rate them in such a manner that they will shame the badly behaved and encourage the better behaved.
Further, there are certain projects that have been criticized for attempting to do similar reputation-based naming and shaming which also are susceptible to these attacks. Worse, by inviting mass action, they leverage the attack-centric nature of personal information and put people at risk of genuine harm.
Monolith or Market?
There are two general divisions in reputation management in general. The first is a strongly heierarchical system with centralized management - consider TLS certificates as an example. These have a relatively strong architecture, but are fatally prone to single points of failure - the higher, more trusted authorities are prone to constant attacks by threat actors motivated to compromise the system for their own purposes. We see this kind of attack with, for instance, compromised TLS certificates issued ostensibly by those authorities, but used inappropriately by government intelligence or other clandestine surveillance operatives.
The other division is a decentralized one - the PGP trust network is supposed to be of this type, and many blockchain-backed schemes attempt to do things of this type. The decentralized nature does allow these schemes to be resistant to the kinds of attack endemic to monolithic management structures - there's no "top" to attack - but they become weak to Sibyl attacks.
This particular discussion will be focused on decentralized trust networks, as they are most emblematic of the kind of solutions the recent crowd-sourced reputation models are trying to implement.
A Malign Oracle
A "Sibyl attack" is one in which a single person or organization creates a large number of accounts (frequently called "sockpuppets" or "sibyls" in this context) on a decentralized trust network. The goal is to use the large number of centrally-controlled accounts to wield disparate influence over the nature of trust on the network - that is, to either endorse unsuitable accounts as trustworthy, or damn more legitimate accounts as untrustworthy.
A "Sibyl network" is a group of such accounts that is self-reinforcing: the owner of the accounts has used the trust mechanisms of the network to grant a trusted relationship to their other accounts, thus making them appear to have widely accepted backing. Sibyl networks of this sort can sometimes be discovered by analyzing trust networks, looking for a large number of accounts that have a high degree of interconnection with each other, but a low degree of interconnection outside this enclave.
(The name is a reference to the book by Flora Schreiber; not, sadly, to ancient Greek prophets and oracles.)
Who is allowed in?
One of the architectural weaknesses common to decentralized reputational systems is fairly fundamental: how do you determine who is allowed access, and who is not? With too generous an access policy, it's easy to generate enough accounts to spam or otherwise compromise a network - and with too strict a requirement, induction becomes too high a bar for a meaningful population to participate.
Many systems err on the side of generosity - they use relatively simple thresholds of proof, like the ability to respond to mail sent to a given email address, or respond from a social media account. These systems are especially subject to Sibyl attacks, as they have a very low cost to enter: many of the 'verifications' used are capable of being scripted and automated, and even if being orchestrated manually, can be done quickly and with a very low amount of skill required.
Systems that attempt to mandate the use of 'real names' - verifying government identification or through other means - are not immune to this kind of attack, depending on the nature of verification. Simple "scan of gov't issued document" schemes can be fooled with image editing; others, though they raise the bar, can be similarly fooled with some effort. Additionally, these schemes entail retention of identifying documentation - which raises the prospect that the registrar of such will themselves be attacked in order to take the identification database for various purposes.
Being in the system signals trust - which is a problem
The very property of having been validated as a valid user in a trust system signals to other members that you have some baseline level of trust. This causes certain psychological assumptions in other users - someone who is 'part of the system' is, even if relatively new, a 'known quantity' and is someone who can be presumed not to be inherently harmful.
This base level of trust is enough of a niche for a malicious user to start with - they can then use this base interaction and whatever privileges come with it to start manipulating other users into granting them increased privilege.
Attack is easy; defense makes you seem less trustworthy
Early stages of 'attack' are identical, in nature, to good behavior within the system - because the point of such reputational attacks is to be seen by a majority of those other than the intended victim(s) as being reliable and useful to the overall health of the community. Sibyling is highly useful in this context, too - generating interactions between ostensibly separate accounts showcasing helpful behavior and positive interactions.
To act suspicious of such apparently beneficial interactions is to seem anti-social, paranoid, cynical, and un-empathetic - strong signals of untrustworthiness in many people's minds.
The Wounded Gazelle Gambit
Of special note, there is one specific tactic that is very useful to attackers trying to subvert reputational systems during their attempts to build themselves as trustworthy sources within a system. TVTropes calls it the wounded gazelle gambit - the attacker claims that they have been, themselves, attacked, generally by a person within the system whose influence they wish to discredit. The mode of attack described is generally very similar to, or congruent with, the mode of attack that they themselves are attempting to make - e.g. claiming others are running sockpuppets, or are trolling, or what have you.
This is an extremely common tactic, and one which can be seen by many groups.
The proof is in the photoshopped pudding
Given the data-network-supported-platform nature of these systems, reputational "proofs" are often somewhat suspect. Screenshots can be manipulated easily. Those fora that allow editing can be used for bait-and-switch posting. Emails are forgeable.
Administrators, moderators, and other users have little or no capability to verify the veracity of any claims of trustworthiness or nontrustworthiness.
Attackers have endurance
Those who are attacking such reputational systems often have a specific victim or set of victims in mind. The resources that they are willing to bring to bear - their "attack budget" as it were - is significantly higher than many defenders are likely to expect. Those who attack these systems with the end-goal of harassing others frequently do so at a length and intensity far beyond the expectations of those who have not encountered this specific variety of maladjusted behavior. In some cases, this has persisted for - literally - years.
Sibyl accounts may be generated and persist for months or years before use in some cases, expressly to defeat account-aging trust systems. This specific sort of Sibyl has even been used to create false identities for real world use.
Almost by definition, anyone attempting to architect a reputation management system is disqualified from judging the risk from such attackers - they themselves are not likely to have the necessary perspective of years-long obsession in mind during design.
Maintenance is a chore
The inevitability of long-term highly repeated threats building networks of sibyls and sockpuppets for the express purpose of attacking others generates unfortunate maintenance problems for administrators of such systems. When a given account is suspected, detected, or proven to be a sibyl, how is this knowledge promulgated through the system, and what effect does it have on the users of the system?
Users must be prepared, especially in web-of-trust style systems, to actively participate in adjudicating the reality or malice of a given other account - and they must be willing, themselves, to revoke trust on a regular basis.
Without a regular pruning of trust relationships, older ones inevitably stale - and those stale links can then be used by sibyl infiltrators, either by exploiting an older sibyl account that has been acting 'benign' and has become stale-trusted by a significant base, or through direct malicious takeover of low-activity but well-established accounts.
With actively encouraged pruning of trust, on the other hand, the nature of the trust sytem itself becomes an end - the whole purpose of interacting with it ends up becoming a metagame of evaluating trust rather than a useful tool.
Asymmetric information exchange is a vulnerability
Worse, the ruleset by which the community can moderate itself becomes a significant avenue of attack. Those who are most socially or technically adept are more capable of manipulating the rules of the system to put themselves to advantage, whereas those who are ostensibly intended to be protected by such systems often look for these systems because they do not, themselves, have that skillset to survive in a more open network.
Thus, these reputation systems become "watering holes" - locations where highly skilled attackers can prey on lower skilled victims, often manipulating such reputation systems in such a fashion that they can turn victims against each other.
When out-of-band communications are involved, this problem is greatly exacerbated; the social networking inherent in these systems generally allows for networks to be extended to other social networking platforms, which allows for communications that are exempt from the normal trust relationships - and which are susceptible to all manner of manipulations.
The malign oracle subverts trust
Thus the problem with decentralized reputation networks - including "crowd-sourced" attempts at trust: they are relatively easy to subvert to an attacker's aim, and there is no shortage of very dedicated attackers with extremely high budgets to carry out their attacks.
These problems are inherent to the nature of decentralized networks in general, and no properly trustworthy reputation system can be implemented without solving these issues.
If you want to implement a reputation system, therefore, you must first solve these troubling issues surrounding Sibyls.