Peer Review: opening discussion

Last edited by on Sat Mar 29, 2008 10:18 pm; edited 1 time in total

Afternoon everyone,

I wanted to talk about our next version of Peer Review. For those who tried out the Alpha in December, we have re-worked the entire application to incorporate many of the great suggestions we got from the community. We’d like to take a moment to talk to you all about the next generation of Peer Review and gather some helpful feedback.

What is Peer Review?

Last year the community requested that we develop a system to ensure that our products were in compliance with the Virtual Goods Policy, while at the same time reducing the risk of a product being taken away due to flagging. In response to this request we’ve been working on a system called Peer Review, which gives the community the ability to decide if a product belongs in the catalog or not.

The system we’ve developed is designed to be very quick and easy to use, and also helps to ensure that as many products as possible can be reviewed. Once the system is turned on, all new products will be required to be peer reviewed, old products can be self-submitted.
The Benefits of Peer Review

There are a number of good reasons to participate in Peer Review. We’ll cover the most important ones in this note:

1) The community gets to say what is and isn’t acceptable according to the Virtual Good Policy. This means you have a chance to voice your opinion on the kinds of products you want to see at IMVU. We will review voting patterns over time in order to understand potential changes to the Virtual Good Policy.

2) Peer Reviews will be plugged into the next generation of Tiers and help contribute to your overall score. Those who do more reviews will earn more points toward higher tier levels.

3) Ensure your products stand-out to users. All products that have been Peer Reviewed will be marked as reviewed on their product pages, this means users will have a higher degree of confidence that buying your good over a non-reviewed will reduce the chance of it ever being taken away.

4) The catalog search and presentation mechanism will return peer-reviewed items first over non-reviewed. This will ensure that the highest quality products with the least chance of being taken-away are presented first to the customers.

5) As more items are peer reviewed the number of products flagged should reduce significantly. This means fewer worries for Creators.
Peer Review Mechanics

Peer Review will operate as outlined below: (we are looking for constructive feedback to refine the mechanics)

1) A product is submitted for review – This can happen by any of the following three ways:
• New products submitted to the catalog.
• Any changes made to an existing product, this includes older created items.
• Customer Service moves the item into Peer review.

2) Creators load up Peer Review - There will be several links on IMVU.com that will let you access the Peer Review tool. Each product is randomly assigned to reduce gaming.

3) Creator reviews the product and answers a few questions – For each product reviewed we are asking the community to evaluate the product, its assigned rating, and product page to ensure compliance with the Virtual Goods Policy. The page is really simple and quick to use.

4) Product passes or fails – At launch all products will receive either 1 Pass or Fail vote based per reviewer. We are using a simple formula to determine if a product will Pass or Fail that consider the number of votes in agreement vs. the total count. We will continue to monitor the performance of the system and adjust the thresholds accordingly.

5) Product owner receives notification – Based on the outcome of the Pass/Fail vote, the product owner will receive a Homepage message telling them if the product passed or failed. Also product owners will have a special page where they can review the status of any products under review.

Timing for this project

The Peer Review system should be online sometime within the next few weeks. We are finalizing V1.0 and will be launching it to all Creators to use. We’ll give you advance notice of when the system will go live, and monitoring it closely to ensure an acceptable level of performance and any bugs are addressed. Right now we wanted to give you all a chance to ask questions.

I wanted to give everyone in the community a chance to discuss this prior to launch and take in everything which is going to happen. Please email me at contentcreatorprogram@imvu.com if you have any questions.

I will also be sending out HP messages to the entire creator community; which I will try to ensure it gets out to everyone, so all are aware of this process. If you do not get this message then send an email to the CCP mailbox with your ID number.
Regards,

Rob
Creator Community Manager
Imvu.com

Posted: Mon Feb 25, 2008 1:17 pm Post subject:

Since music products are on my mind constantly I have a question. Since I've heard IMVU doesn't want us to flag music products we see does this mean we can't find them unacceptable (fail) in peer review as well?

Are there any penalties for giving someone a fail grade incorrectly or is completely a matter of personal opinion?
_________________

Posted: Mon Feb 25, 2008 1:23 pm Post subject:

Posted: Mon Feb 25, 2008 1:26 pm Post subject:

Thanks for clearing that up for me I appreciate it. I wasn't around for the original alpha testing. This looks really promising.
_________________

Goose 76 Joined: 30 Nov 2005 Posts: 3303 Location: USA

In the first review, I was wrong about 18% of the time according to the stats bar even though I know I was right as far as something being AP or UFI was concerned. I was counted wrong when going against the popular vote.

Has this been addressed in this roll out and will it effect my tier level if I do not vote how I think others will vote, even when i know I am right?

Thanks for the heads up on this.

G

Ken 158 Joined: 18 Feb 2007 Posts: 14031 Location: USA

Rob, what happens if say I submit a new product to the catalog, it meets VGCP's rules for AP content, but yet someone fails my product due to it violating "their personal beliefs" and not the actual VGCP? Will I be able to have the fail over-turned by staff?
_________________
In loving memory of Tiggerrific

Posted: Mon Feb 25, 2008 1:33 pm Post subject:

Notorious Joined: 02 Jan 2006 Posts: 13470 Location: USA

So...
let's say we have an item in the Peer Review process that the community decides is a-ok, however the item is actually UfI....
If I , as a responsible developer, rate it UfI...will I be penalized for not voting with the group of peers?
_________________

Goose 76 Joined: 30 Nov 2005 Posts: 3303 Location: USA

Posted: Mon Feb 25, 2008 1:38 pm Post subject:

Ken 158 Joined: 18 Feb 2007 Posts: 14031 Location: USA

yep, thanks Rob
_________________
In loving memory of Tiggerrific

Notorious Joined: 02 Jan 2006 Posts: 13470 Location: USA

Posted: Mon Feb 25, 2008 1:52 pm Post subject:

Notorious Joined: 02 Jan 2006 Posts: 13470 Location: USA

AnakaTarren Joined: 16 Mar 2006 Posts: 2179 Location: USA

Rob,

It's great to hear the Peer Review is coming! Hopefully the submission slowdown and furniture testing issues in the alpha version have been resolved. This can be a good step towards better product rating.

I have some of the same concerns as Goose on how ratings can potentially hurt a developer if we are correct but vote against the majority. Once I got past the learning curve *, I was the ONLY person who flagged some of the alpha test items as out of compliance even though I know my response was correct. This was especially true with items involving inappropriate "merchandising behavior." I'd hate to see my tiers hurt because others don't know TOS as well as I do. Hopefully IMVU will stay on top of these issues as they come up.

* I incorrectly rated certain initial alpha test products due to a misunderstanding of TOS for items that I normally don't buy. But after the feedback mechanism finally started working and I figured out that rule, the rest of my answers were almost totally supportable for the remaining test period.
_________________