Sun 22 Nov 2009 |
JED Voting Adventures
Written by Toni Marie
Accurate listings? It's a lot of work!
Voting and Reviews by Users
There are more than 115,000 registered users voting in the Joomla Extensions Directory, and most of them do it the right way. In the past I've talked about reasons we reject reviews from users, but we've never really opened up the process we use to make sure the votes are valid, honest and accurate.
Because the JED is the most complete and accurate listing of Joomla extensions available, its traffic is high and as such listings are valuable. The temptation to influence a JED listing is clearly great, and we spend quite a bit of our time monitoring voting patterns and preventing abuse. Because of the value of that traffic, we believe it is our duty to make sure the listings are as accurate and honest as we possibly can make them.
Voting and Reviews by Extension Owners
The vast majority of the 1600+ extension owners in the JED use the system properly, and for that we're grateful.
When an extension developer decides to influence their listing with fraudulent votes, they are hurting not only the other developers but the users that trust the ratings in the JED. They take our time from working on extension and review approval, and slow the process down for everyone.
In the past six months we've had 24 different incidents of voting abuse discovered. Of those 24 infractions, 12 resulted in a warning, 10 resulted in a 1 month suspension, one in a 2 month suspension and one repeat offender racked up 6 months of JED-free living.
Why not just remove voting for owners? The Joomla community has folks who write extensions and use extensions. We'd prefer not to remove the voting ability for someone who codes a module and submits it to the JED. It's just not fair to lose all those privileges just because you want to contribute. So we've written some simple rules about when and how extension owners can vote.
The Rules are Short and Simple
The voting rules in the JED are available in our Terms of Service. An excerpt of our TOS:
Voting
1. A user account can only vote once per extension. Please do not vote until you've used an extension thoroughly, making sure you have configured it correctly.
2. It is not allowed for any person to solely rate an extension with the purpose of artificially inflating the rating or vote count.
3. It is not allowed for the same person to have more than one account on the Joomla! Extensions Directory.
4. It is not allowed for an extension developer to give negative votes to other extensions who are in competition for rating ranks, or to submit votes to extensions which offer similar functionality.
5. An extension developer, his family and\or colleagues are not allowed to vote on extensions from that developer.
There are varying degrees of damage a developer can do with fraudulent votes. They might decide to give their own extensions good (5 star) votes. Against the rules, but not the worst thing that could happen. Getting a little more bold, some extension owners will create other user accounts using their personal, work, and freebie email addresses to give their extensions good votes. This is clearly fraudulent and a conscious attempt to sway the ratings.
The ones that really bother me are those who jump right to using those additional accounts to also give competitors 1-star votes as well. Not just inflating their own extensions anymore, these folks are actually deliberately bringing down another developer's rating. They do the most damage and the intent is abundantly clear.
We See Everything
We have a tremendous amount of data at our fingertips to help us investigate voting patterns. I won't detail that data here, as publishing our methods would just give violators a road map to circumvent our investigations. I'll be vague instead.
What do we do when we suspect fraudulent voting has occurred? We investigate and compare the accounts, votes, time frames and other information we have available to us and a pattern will emerge. We then discuss among the team the possible reasons for the voting pattern and come to conclusion as to the validity of the votes.
When we've confirmed the fraudulent voting privately, we unpublish the user's extensions and we email the developer to let them know what we've found and ask them to explain themselves. Sometimes, we find mitigating circumstances. Sometimes, the developer is remorseful. Sometimes, they deny it vehemently. We take whatever response we get, discuss it, and determine the appropriate suspension length. In the past we have asked for an explanation before unpublishing a developers extensions, but unfortunately too many developers ignored our emails while their extensions remained published. Importantly, this is the process for clear-cut cases of fraud. If the data is not conclusive, we start by contacting the dev and then proceed as appropriate.
The first infraction may warrant a warning or a one month suspension. A few times the severity has caused us to jump straight to the three month suspension for a first offense. Typically, we settle on one month. If abuse continues, the suspension can be three months, six months or permanent (with yearly review).
Most Developers Do the Right Thing
Most of the time, abuse does not continue. Developers see that each listing is treated equally. Occasionally, the "excuse" for the voting fraud was that "we thought everyone was doing it". When they discovered that listings are as accurate as we could keep them, they have apologized for mucking it up. We've been known to lighten a term for folks who are honest and remorseful.
On the other side of the coin are those who will vehemently deny culpability when we have indisputable evidence. They complain the loudest that they are being treated unfairly. This does not make us feel all warm and fuzzy.
We never discuss individual cases in public unless the extension owner reveals the reason their extension has been unpublished in public. It is not necessary, and we do not think that public shaming is helpful even in the most serious cases.
It's a shame we had to post this to begin with, but we'd like the public to know that we don't take voting abuse lightly, nor do we jump to conclusions and deal with a heavy hand. We hope it will prevent misunderstandings and preempt new issues and keep the JED the most reliable source of information on Joomla extensions.
2009-11-22 17:06:36
Thanks for this posting, it certainly makes things a little clearer and more comprehensible. I do not thinks that it is a shame to make a posting like this, but would go further by suggesting that this be highlighted every so often.
With regard to the following;
Is it not possible to block Developers from voting in categories their extensions are in ? It would cetainly lighten the load of the JED Editors.
This confirmation that there is no "JED anti-vote-rigging software" is welcome and leads to another issue that is discussed from time to time, Who is watching the watchers ? It's unavoidable that we have a situation where JED Editors may also be Developers, and the community understands and accepts this fact, the sacrifice made by the Editors in general etc... Is there any system in place to "Watch the Watchers" ?
I am sure that in addition to the normal category sweeps, the Editors also sweep lists like the "Top Extensions" , 'Most Popular Extensions" etc...
Once again thanks for the Professional and enlightening posting, much appreciated.
2009-11-22 17:22:13
It may happen ( perhaps rarely ) that a Developer has an email address for a Site that is managed (workflow management) , would you advise that under these circumstances anyone with an address under that domain be informed ( by the Developer with such an address ) not to vote ? Perhaps that could be included with Rule 5.
Thanks for the consideration
2009-11-22 17:38:20
Don't worry if someone posts "What is going on?" Or tweets that their Extension was removed due to voting fraud. The JED Team can back up these decisions easily, in private, where it is less embarrassing to the developer, or right out there in the open where everyone can see, if that is how the developer chooses to handle it. The fact that JED initiates these discussions delicately is very much appreciated.
I am encouraged to hear there were only 24 incidents in the past six months. That means your work is paying off. When this system was first implemented, I believe the problem was far bigger.
Keep up the great work!
2009-11-22 18:02:01
2009-11-22 19:54:11
The JED team is a carefully selected, close knit staff. While some members do have extensions, they actually recuse themselves from any contact with extensions or developers that can be considered "competitors".
I consider the integrity of the JED team above reproach. It's the reason we're so far behind! We don't take every offer of help. Every time there's a new addition to the team, it takes weeks, even months, to choose a new member.
However, oversight happens anyway by the nature of the project. The Sites and Infrastructure working group oversees the JED and we interact regularly with members of both the core and OSM. If we have any question as to a change in policy or even a controversial suspension, plenty of outside folks sign off on it first. Rarely do the overseers disagree with our stance, but when they do it is always brought back for review and a fair decision is reached.
2009-11-22 20:59:19
Be very sneaky and make a really good extension. Then perfect strangers with different IP and e-mail addresses will vote and review highly for you....
That's really messing with their heads!
2009-11-22 22:27:35
2009-11-23 07:05:13
2009-11-24 00:58:16
2009-11-26 07:56:20
Great post
recently he spent very strange things on the site JED and voting system.
Why not remove the right to vote for any extension developer.?
These decisions give confidence to any user in his chosen extension before any commitment.
Thank you for all the work you do.