PDA

View Full Version : New Vote/Rating System Feedback thread



ricktas
25-09-2011, 7:53pm
With the voting for POTW 274 underway and using a trial of our new entry rating system, this thread is for member feedback

PLEASE DO NOT DISCUSS SPECIFIC COMPETITION ENTRIES

We would like to know how the experience of voting/rating the entries was, for you. Anything you wish to raise will be welcome, as all feedback, good and bad, is going to be of benefit.

Looking forward to reading what you think, at this stage, of the new vote/ranking system.

Kym
25-09-2011, 8:24pm
I'm biased :rolleyes:

Voting was quite easy, I just scored each entry on merit, it really did not take any more effort than the old system as there was less scrolling back to compare

Mods have the advantage of seeing the scoring as it happens, I think people will love the feedback and will be very pleased next Sunday

:th3:

ricktas
25-09-2011, 8:26pm
I was going to wait for some members to provide feedback, but for me, I found the process of voting on each entry faster than I expected. Once I got into the swing of it, it did not take that long to assess a photo and give it a ranking, and move onto the next one.

AmPhot
25-09-2011, 8:33pm
I noticed it took me a bit longer to complete voting, especially for the intermediate level 48 entries, but this is a first go so maybe I just need some more practice. Could be coincidental but when I logged on to vote I noticed some severe lag on the site, it took a bit longer for the voting pages to load. :confused013 I'll be curious to see how the voting results are displayed on Sunday. Overall not a bad experience, but I'm not convinced yet it'll be an easier system to use - especially for a single level comp with 50+ entries - definitely more testing required. ;)

mpb
25-09-2011, 9:39pm
When I saw the new system then 48 entries in intermediate my first thought was "you have got to be kidding", but once I read the points guide-line and got going it really did not take as long as I expected and was probably actually easier because you get to rate all entries rather than trying to pick one shot over another.

I think the system will provide good feeback on entries and it will be interesting to see the results.

Cliff
25-09-2011, 9:50pm
I liked the new system. Previously I looked at all of the entries and then went back over them to make my choices. This time I was able to score each entry at first viewing. I did go back over them and review my scores, altering a couple either up or down by one point. I'm looking forward to seeing the final results.

Ms Monny
25-09-2011, 10:04pm
LOVED IT!

Previously, I would um and ah about each photo, click on what I liked then went back again and re-clicked, not really having a 'system' to rate the photos by. They were only chosen because I liked them and it was sometimes hard to chose between them, trying to get the top 4 (or 6) :umm:

Now, I put the 'rating guide' on a seperate page for easy access and go through each photo, using the guide as a backup. I went back ONCE to check the scores (not 2 or 3 times like before) and I had finished quicker with this system than the old way. :th3: :cool:

Mary Anne
25-09-2011, 10:17pm
It took me a long time and I would not like to do this every week.
With the printed guide beside me I looked at it often, I think that was a bad idea.
Actually it got at me so much I left it alone for a while and went and replied to some threads
Then went back to voting which to me was more of a chore this way.
Though something that I didn't mind doing before the other way.

ricktas
26-09-2011, 5:28am
It took me a long time and I would not like to do this every week.
With the printed guide beside me I looked at it often, I think that was a bad idea.
Actually it got at me so much I left it alone for a while and went and replied to some threads
Then went back to voting which to me was more of a chore this way.
Though something that I didn't mind doing before the other way.

Do you think Mary Anne, that if you did this for a couple of weeks, you would not have to refer to the guide and be able to make an informed judgement without using the guide, and then, with a bit of experience, it would be quicker, and easier? Or is this style of voting/ranking going to become tiresome for you?

ricktas
26-09-2011, 5:47am
To give everyone some early feedback, the rankings are very close (this is a sample from part of the results of one poll)

Some of the rankings look like this:
6.273
6.152
6.091
6.091
6.061
5.970
5.970
5.909
5.879

It shows that entries are actually very closely rated by members. Which for me, tells me a lot more than the old voting with entries getting one or two votes only.

Kym
26-09-2011, 8:04am
Lets face it, voting is generally once a week and then the POTM and the odd special comp.

I think the value of the feedback from the new system will outweigh the little extra effort (which will get easier as people get used to it).

The spread 1st to last is around 4-5 points on the three polls. This is statistically similar to club competitions.

Looking at individual voting (who voted what and how many points) all members are being reasonably consistent,
albeit some score higher on average than others (which is fine) and some use a bigger spread than others (also fine).

Good work everyone :th3:

junqbox
26-09-2011, 8:18am
more tedious than previous method.

stoogest
26-09-2011, 8:31am
The process I used was:

1. Scan all photos for the best; score that one 10 points
2. If I noticed a few other photos that were almost the best then I'd give them 9's straight away
3. Otherwise, go back up to the top and start scoring everything relative to the best photo, only giving less than 5 to those photos that have serious issues (very few to be honest)

Yes, it did take longer to vote. BUT, I actually found it less frustrating than the old system where I would often pick 7 favourites, then have to whittle it down to 4, involving much scrolling up and down to find my selections and make my decision. With the new system it only involves two passes, one for the best photo, and a second one for the rest. If you bear in mind that you're simply scoring relative to the best photo then it's quite simple to dish out scores to the reamaining photos.

It will be extremely interesting to see the results and how they compared to the scores I gave!

Cheers!
Andrew.

Mary Anne
26-09-2011, 9:04am
Do you think Mary Anne, that if you did this for a couple of weeks, you would not have to refer to the guide and be able to make an informed judgement without using the guide, and then, with a bit of experience, it would be quicker, and easier?

I certainly hope so Rick, next vote I will not look at the guide and see how I go. Maybe I will be quicker.



Or is this style of voting/ranking going to become tiresome for you?

Tiresome interesting word that one, no I don't think so, its new and I have to get used to it. Change is hard for me to gasp sometimes.
I did do a lot of manual work in the garden yesterday and I was tired, so perhaps not a good time to vote in the evening.
In future I will vote after lunch the next day when the mind and body are more in tune with each other :D

Kym
26-09-2011, 9:56am
The process I used was:

1. Scan all photos for the best; score that one 10 points
2. If I noticed a few other photos that were almost the best then I'd give them 9's straight away
3. Otherwise, go back up to the top and start scoring everything relative to the best photo, only giving less than 5 to those photos that have serious issues (very few to be honest)

That is a very good process!

JohnB5319
26-09-2011, 12:07pm
I think this is an excellent innovation.

I see some members are talking about the time it takes. Yes, it does, but it makes the competition so much more meaningful and if members are trying to assist their fellow members, they should see the time invested as worthwhile. There are plenty of photos that get low scores each week but they're not that bad. This new method provides a type of critique so that even photos that might have received 2 or 4 votes might actually rate a 6 or 7. This gives those members some encouragement, in my view.

I personally believed that going to an optional number of selections (e.g. pick between 2 & 4) was a retrograde step - I tried to vote for as many as possible so as to give as many people encouragement as possible. Having two photos to vote for most weeks meant that the more competitive amongst us may only have voted for their own photo and one other.

The system may need some tweaks, but at this stage, it's a great step forward in value to members.

Wandapics
26-09-2011, 12:08pm
I like the new voting. I was able to score each without having to leave out some that I thought were good. I to often chose to many with the old system. It didn't take me as long as I thought it would.

Allann
26-09-2011, 12:18pm
I liked the new voting system. But I think this could improve things;

At the confirmation stage, can the images be sorted by the score given (10 to 1) so you get a preview of your rankings before confirming the vote, otherwise the confirmation is quite tedious.
I think this will make people think about the images more, I know it did for me, but once the meaning of the numbers were remembered, I found it easier and quicker to vote for each. I did it slightly differently to stoogest, and voting on each image by it's merit (where it fitted in the scores), which meant that there may have been 2 images with 9, or even 10, many with 7, etc. I assume this is what was desired.

I am quite used to this scoring system from another site where you can vote similarly, but also includes a drop down for additional feedback (quite useful on very good/bad images). Since this site allows voting at any time, you never know when another image is submitted so you can't select a 10, but instead score them on a standard that you (the scorer) hold all images too, these seems fairer to me. Why vote someone a 10 if it is really only a 8? Using that system it's easier to get a running highest ranked image over time, once a member has voted on the image they can't vote again.

If any of the mods want to know the site to check it out, feel free to drop me a line.

Kym
26-09-2011, 12:26pm
I liked the new voting system. But I think this could improve things;

At the confirmation stage, can the images be sorted by the score given (10 to 1) so you get a preview of your rankings before confirming the vote, otherwise the confirmation is quite tedious.

Great idea (one software guru to another :D ) ... I'll implement that later in the week.

CAP
26-09-2011, 12:44pm
Actually found this new system a good exercise in making me look deeper into each photo, and it wasn't any where near as daunting as I first thought it would be.
Found that I use a similar method as "stoogest", find what I consider to be the standout pic, but don't rate it automatically to 10, more to what I consider it's merit is on it's own.
I then rated each of the remaining photos against the top selections.
Only problem is that I feel I may have given some entries either a lower or higher score when judged against the first choice shot then they would have if judged on there own.
It all depends on what score I gave to the "first choice photo. I think I had a fair spread between most liked and least liked photos.....
To my mind it's still a competition so am I correct to compare and judge all other pics against what I perceive to be the best shot put forward for that week?

WhoDo
26-09-2011, 12:56pm
To my mind it's still a competition so am I correct to compare and judge all other pics against what I perceive to be the best shot put forward for that week?
Exactly! In fact the new guidelines make it clear that entries should be judged competitively for that week rather than against an arbitrary criterion for what is a good or bad image in the mind of the voter. It is a comparison scoring system, not an all-time best rating system.

Cage
26-09-2011, 1:00pm
I concur with Allann's comments above.

I didn't necessarily give my top pick a 10 if I didn't think the shot was worth a 10.

I feel each photo should be judged from 1 -10 on it's individual merit, and not picking the best of the bunch as a '10' and down-rating all other entries.

This method should stop the scenario where one not particularly outstanding photo is the best of a rather ordinary bunch of entries for that competition, and receives the maximum 10 score from all voters. When looked at in the context of one monthly winner scoring maximum votes and being overshadowed by eleven other monthly winners with better quality entries and lower scores, it seems rather inequitable.

CAP
26-09-2011, 1:01pm
I quite like the idea of automatically sorting the images on screen as they are scored, it would definitely help the voting process.
While we at at it and Kym is busy punching code like crazy.... would be be worthwhile to impliment this scoring system into the everyday forums like Constructive Critique?
A "Rate My Photo" option that will tally the averaged scores as they happen.
This option could be either permanently enabled or selectable option to enable by the thread starter if there are some that choose not to.
My thinking that if someone puts forward a score out of 10 for a pic in CC then they might be more inclined to say more than "nice shot" (of which I am guilty on numerous occasions)

stoogest
26-09-2011, 1:12pm
The problem with giving the best image an 8 out of 10 (rather than 10 out of 10), is that your definition of 10/10 may be vastly different to someone else. By giving the best image in the competition a score of 10 it helps keep the scoring relative, and a little bit more consistent between voters.

Andrew.

ving
26-09-2011, 1:14pm
i like it. you vote for every entry giving it a score out of 10. top stuff.
I found with the old system id scroll thru and if i liked one i'd tick it and after 5 ticks if there was still one down further i liked id have to scroll back and forth comparing the two and maybe untick one before i could tick another....

in shot, much easier! :th3:

alextdel
26-09-2011, 1:24pm
Thanks for the guide, I found it very useful.

I took longer than normal to consider the photos, this will probably improve with practice but the voting could get cumbersome if there were many entries. I felt that the pictures were relatively 'right' for my judgement - again the guide was handy here.

Forgive me if this the following has been discussed previously. Coming from a tech background (! :eek:), I know that using measures of average (median) are sensitive to relatively few extreme values. That is a few, even one, extreme value has much more effect than many non-extreme values. This effect is potentially more significant the less people are doing the scoring. A major concern here is the case of voting for your own picture. I assume most people would vote high for their own photography despite wanting to be fair-minded for the overall process - after all, we put a photo in the comp because we think it is worthy.

For example, assume 20 Ausphotographers have given their score for a photo and the average of the 20 scores is 5 (total of the scores would be 100). Now another 4 people log in and give the same photo scores of 4, 6, 5 and 5. The result would be that the average score (rank) remains at 5 [(100+4+6+5+5) divided by 24]. There doesn't seem anything wrong with this as the photo looks like it is probably a picture worthy of a 5 score. But now a single new user logs in and gives the photo a 10. The mean now rises to 5.2. So one voter has had more effect (4%) than the previous four voters (0%).

The sensitivity of the median to extreme values is also the case for low votes. Potentially relatively few scores of 1 could jeopardise an otherwise reasonable score average.

I think that 'voting' for yourself is ok but the scoring for yourself could be an issue. I do not suggest for one second that any of our fellow Ausphotographers are deliberately unethical (quite the opposite is why I like the site), but in fact the fairest and most ethical competition user will probably be disadvantaged if they accurately judge their own photo compared to others who automatically give their own photo a 10. Now in the case of a fabulous and obvious winner I think the best photo will still get the best rank, but in close run contests voting for your self and strategically voting low for other close competitors could tip the balance.

How much this all matters, I don't know. I also know the site admins do a good job of moderating the competitions for abnormal practices. I still applaude the effort to increase the value of the competitions to all entrants.

WhoDo
26-09-2011, 1:26pm
I feel each photo should be judged from 1 -10 on it's individual merit, and not picking the best of the bunch as a '10' and down-rating all other entries.
Individual merit against what criteria, Kevin? It's a competition for THAT WEEK ONLY, so you are rating say 40 entries on a scale of worst (1) to best (10) in that company only. Sure there is nothing stopping you from saying "I don't think any of these deserves to win", in which case you won't allocate a 10, but you need to remember that everyone else will rate their winner for that week a 10, and if it isn't the same image you think is best then giving yours less than 10 is robbing it of some comparative value.


This method should stop the scenario where one not particularly outstanding photo is the best of a rather ordinary bunch of entries for that competition, and receives the maximum 10 score from all voters. When looked at in the context of one monthly winner scoring maximum votes and being overshadowed by eleven other monthly winners with better quality entries and lower scores, it seems rather inequitable.
Well first we are talking about POTW Qualifying only at the moment. The final choice is still a pick from 4-6 entries. Secondly, there will always be some POTW or POTM entries that outshine their peers; that's why there is a POTY from all of the winners! POTW is about picking the best photo submitted THIS WEEK, not this month, this year, or compared to whatever else I may have seen previously in comps here, etc. I hope that explains the intent a little better, mate. No disrespect to your opinions intended. :cool:

WhoDo
26-09-2011, 1:33pm
I know that using measures of average (median) are sensitive to relatively few extreme values.
Yes, Alex, we discussed the statistical impact of outliers and decided the effects were too small here to justify a system of discarding high and low votes as they do with diving, skating, etc. The ratings only apply to qualifying so the chances of affecting the outcome are are ~1% and too small to justify the programming effort at the moment. I'm sure Kym and Rick will be monitoring the system for fairness, but it already beats one where you are forced to discard images of equal value that sit at the margins; i.e. vote or not vote.

Kym
26-09-2011, 1:50pm
Thanks Waz. Yes, that is why we are keeping the final single selection vote.

Even if 50 voters (and we usually have 80+) give 7 pts and 10 more give 8 and 10 give 6 (mean of 7) and the entrant gives themselves 10 points....
The total is 7.042 - meh! No drama.

In looking at the actual voting trends I see a good spread and consistency; so it appears to be very fair.
Even at the extremes they seem to balance out in a normal curve sense, i.e. for each low vote there is a high vote.

FYI Rick and I have a tool that shows all the raw voting down to each individual vote in a table,
and we do monitor voting for irregularities, esp. when prizes are involved.

Remember not everyone sees an image the same way - there will be variations in scoring :D
I don't agree with the camera club judging about 1/3 of the time, but don't complain either as the judge has their own views.

Allann
26-09-2011, 2:28pm
would be be worthwhile to impliment this scoring system into the everyday forums like Constructive Critique?
A "Rate My Photo" option that will tally the averaged scores as they happen.
This option could be either permanently enabled or selectable option to enable by the thread starter if there are some that choose not to.


I love this idea too... especially with a opt in/out when creating the thread. That would save the OP asking for CC all the time. And it'll make the 10-1 scoring system more natural for people to use, and we would start to get an understanding of what the scores actually mean.

kiwi
26-09-2011, 3:09pm
I love this idea too... especially with a opt in/out when creating the thread. That would save the OP asking for CC all the time. And it'll make the 10-1 scoring system more natural for people to use, and we would start to get an understanding of what the scores actually mean.

I've suggested this before too, perhaps flippantly. Even a gold/silver/bronze breakdown for fun. Add up the number of points per month and get a monthly merit etc - this would also encourage perhaps more pictures being posted

Kym
26-09-2011, 3:40pm
would be be worthwhile to implement this scoring system into the everyday forums like Constructive Critique?
A "Rate My Photo" option that will tally the averaged scores as they happen.
This option could be either permanently enabled or selectable option to enable by the thread starter if there are some that choose not to.

I love this idea too... especially with a opt in/out when creating the thread. That would save the OP asking for CC all the time. And it'll make the 10-1 scoring system more natural for people to use, and we would start to get an understanding of what the scores actually mean.

Interesting idea!

If I ever got around to it ... It would work with single image threads.
It's too difficult/messy to do this per image due to attachment vs linked image.
It would amount to an optional anonymous thread rating system.

It's not going to happen for some time, if we decide to even implement it; bedding down the new voting is far more important.

Wilky
26-09-2011, 4:53pm
Thumbs up from me guys.

I found it simple and quick because I wasn't scrolling up and down.

I did have trouble with the page timing out when I clicked on the vote button, in both chrome and firefox last night but tonight in firefox it worked fine.

rene52
26-09-2011, 5:14pm
Well I am sorry to be one of the few that don't think this is better. I agree with the problem of only havig 6 votes to decide on previously being difficult but this is time consuming at its peak. I have been critiquing as much as possible which again takes time but I assume that a critique is better than a vote where there is no feedback at all. I for one will now look at only critique not weekly photos (mind you I haven't had time to put photos in anyway so I suppose nothing lost or gained). Will wait to see the next iteration of voting.

ricktas
26-09-2011, 5:53pm
Well I am sorry to be one of the few that don't think this is better. I agree with the problem of only havig 6 votes to decide on previously being difficult but this is time consuming at its peak. I have been critiquing as much as possible which again takes time but I assume that a critique is better than a vote where there is no feedback at all. I for one will now look at only critique not weekly photos (mind you I haven't had time to put photos in anyway so I suppose nothing lost or gained). Will wait to see the next iteration of voting.

I agree that it is a bit more time consuming than picking 4/5/6. Wait till Sunday evening when we reveal the polls and members can see their rating. I think that is where the member benefit will be. Members will then be able to see how their fellow AP members rated their photo. I think the benefit of a 6.182/10 will offer more in the way of valuable feedback, than the current, you got 6 votes, system.

Certainly forum critiquing is also an integral part of AP, and we are trying to improve the value of a vote, in the competitions, to where the rating reflects how good an entry was (for all entrants), rather than a simple, number of votes. By comparing their rating to the guidelines (http://www.ausphotography.net.au/forum/showthread.php?92107-The-Ausphotography-Competition-Voting-Guide-for-rating-entries-1-10) a member can see how other rated their photo and use the guideline description to get a better understanding of how to improve it. The current you got 2, 6, 12, 20 votes doesn't give entrants that feedback in any way.

Next weeks POTW will use the 'old' voting system, and then the week after we will again trial the new vote ranking system. After that members will have had a good chance to evaluate both systems and see the results of how a ranked vote gives more feedback to the entrants, and then we will put it to members as to which system they want, or both (we can still have both systems and apply them to different competitions).

Kym
26-09-2011, 5:56pm
I liked the new voting system. But I think this could improve things;

At the confirmation stage, can the images be sorted by the score given (10 to 1) so you get a preview of your rankings before confirming the vote, otherwise the confirmation is quite tedious.

Great idea (one software guru to another :D ) ... I'll implement that later in the week.

Done!

Dylan & Marianne
26-09-2011, 6:04pm
I found it pretty easy and as someone who usually thumbs images I thought were non contenders, the number of clicks was no more or less
I must admit though, I made the mistake of judging the images on standalone merit and not with regard to each separate competition so I didn't use the entire spectrum of marks (next time I'll read the voting guidelines closer!)
I also like the fact that you get the intermediate feedback - after all, an image that everyone thinks universally scores 7-8 might well have got no votes because it just looked good to everyone but not unique.

Kym
26-09-2011, 6:14pm
I must admit though, I made the mistake of judging the images on standalone merit and not with regard to each separate competition so I didn't use the entire spectrum of marks (next time I'll read the voting guidelines closer!)

You should ideally have one 10 and probably none <5 points, assuming all entries meet the theme and are technically ok.

rene52
26-09-2011, 6:17pm
I agree that it is a bit more time consuming than picking 4/5/6. Wait till Sunday evening when we reveal the polls and members can see their rating. I think that is where the member benefit will be. Members will then be able to see how their fellow AP members rated their photo. I think the benefit of a 6.182/10 will offer more in the way of valuable feedback, than the current, you got 6 votes, system.

Certainly forum critiquing is also an integral part of AP, and we are trying to improve the value of a vote, in the competitions, to where the rating reflects how good an entry was (for all entrants), rather than a simple, number of votes. By comparing their rating to the guidelines (http://www.ausphotography.net.au/forum/showthread.php?92107-The-Ausphotography-Competition-Voting-Guide-for-rating-entries-1-10) a member can see how other rated their photo and use the guideline description to get a better understanding of how to improve it. The current you got 2, 6, 12, 20 votes doesn't give entrants that feedback in any way.

Next weeks POTW will use the 'old' voting system, and then the week after we will again trial the new vote ranking system. After that members will have had a good chance to evaluate both systems and see the results of how a ranked vote gives more feedback to the entrants, and then we will put it to members as to which system they want, or both (we can still have both systems and apply them to different competitions).

Thanks for your comments on my feedback - I just feel that even the rating (well in my opinion ) doesn't really help me to take a better photo. To do so in most cases you need to be in the right place at the right time and also have the right equipment. I am by no means a professional photographer and as such would prefer to have a critique rather than just a rating. I do realise that without comps there isn't much to get excited about but for me this makes me spend more time where I could have spent the time giving feedback to people that post photos.

I find that most of the photos that are presented at the POW are of a high standard anyway and as such would not find much to 'fix'. Now I am not writing this to complain or anything it is just it seems to difficult and time consuming. I am a member of quite a few sites that have photos posted and ask for people to discuss and spending extra time will not be something that would like to do.

ricktas
26-09-2011, 6:21pm
Thanks for your comments on my feedback - I just feel that even the rating (well in my opinion ) doesn't really help me to take a better photo. To do so in most cases you need to be in the right place at the right time and also have the right equipment. I am by no means a professional photographer and as such would prefer to have a critique rather than just a rating. I do realise that without comps there isn't much to get excited about but for me this makes me spend more time where I could have spent the time giving feedback to people that post photos.

I find that most of the photos that are presented at the POW are of a high standard anyway and as such would not find much to 'fix'. Now I am not writing this to complain or anything it is just it seems to difficult and time consuming. I am a member of quite a few sites that have photos posted and ask for people to discuss and spending extra time will not be something that would like to do.

Sure, there are members on AP who enter every comp, but only post to the forums about once every couple of weeks. We know we can't have all of the site, be everything, for each member. In the end we are trialling an idea to improve the current vote system. Each member is free to either join in on the comps, forums etc, or not, as they wish.

Kym
26-09-2011, 6:23pm
Thanks for your comments on my feedback - I just feel that even the rating (well in my opinion ) doesn't really help me to take a better photo. To do so in most cases you need to be in the right place at the right time and also have the right equipment. I am by no means a professional photographer and as such would prefer to have a critique rather than just a rating. I do realise that without comps there isn't much to get excited about but for me this makes me spend more time where I could have spent the time giving feedback to people that post photos.

I agree nothing beats good CC.
But!
If you missed the finals of a comp by 0.5 points (say you got 7.3 and the cutoff was 7.8), but under the old system you may have got 5 votes and the cutoff was 25 votes?
That tells you a lot about where you are at.

CherylB
26-09-2011, 6:50pm
Might have known Kym wouldn't wait until "later in the week" to implement the ranked confirmation view! :D Definitely a good idea and certainly helped to review my ratings of the photos relative to the other entries in the same comp.

Overall, I like the system! :th3: Yes, it takes a bit longer to go through, but I found there wasn't anywhere as much scrolling back and forth as with the old system. Previously, I would scroll through and "thumb" the entries that I felt didn't make the grade, until I was left with at least twice as many entries as I had votes for - hence the scrolling back and forth to try to decide which ones to pick. With this system, I found that all I needed to do was to scroll from top to bottom viewing the entries to get an overall sense of all the photos, and then scroll back up giving each one my rating. Easy as! And with the ranked confirmation view - icing on the cake!

On the "downside", I am now suffering from considerable guilt because I couldn't explain why I ranked certain entries with a lower score. :( Obviously, I cannot discuss the entries, but I can say that in general, in this competition, it was mostly due to the fact that I personally felt that they did not meet the theme. There were a number of very good entries I ranked at the lower end of the scale simply because "ships or boats" were only incidental in the photo.

On a side note, the camera club I am a member of recently introduced a new way of doing their monthly competitions (on FB of all things! :rolleyes:). They also have a scoring system from 1 to 10 ..... however, we are "not allowed" to give any photo a score of less than 5. I'm sorry, but I completely disagree with those kinds of limitations. If your scoring scale is 1 to 10 then there are 10 scores you can choose from .... however we individually determine the score. If you are going to not allow 1, 2, 3 and 4, then the scale is actually 1 to 6!! Needless to say, I haven't participated in this travesty of a competition yet!

Kudos to AP - I think this will be a very beneficial improvement for members. Well done to Rick, Kym and the team! :th3:

All I need to do now is dust the cobwebs off my camera and put in some comp entries again!

Mark L
26-09-2011, 6:53pm
.........A major concern here is the case of voting for your own picture. I assume most people would vote high for their own photography despite wanting to be fair-minded for the overall process - after all, we put a photo in the comp because we think it is worthy.
.................
I think that 'voting' for yourself is ok but the scoring for yourself could be an issue. I do not suggest for one second that any of our fellow Ausphotographers are deliberately unethical (quite the opposite is why I like the site), but in fact the fairest and most ethical competition user will probably be disadvantaged if they accurately judge their own photo compared to others who automatically give their own photo a 10.

I, for some reason, won't vote for anything I've entered in a comp.. Now I have to, in a way.:confused013
Anyway, originally I was a little sceptical of this new voting method. However after due consideration and having now used it, I think it's a good move.
I think it may encourage more entries, though I wonder if it will encourage less people to vote. Judging by the responses in this thread, probably not.

ricktas
26-09-2011, 7:11pm
Just to give a bit more feedback on how the results are stacking up.

The highest ranking across all three competitions at present is 7.986 and the lowest is 3.459.

Geoff79
26-09-2011, 7:25pm
Jeez I love the new system. One of the things that concerned me about the old system was great photos going unacknowledged because there weren't enough photos you could pick, and a photo I thought was excellent was left voteless, even when I wanted to acknowledge it's excellence.

Now, it wasn't always the case and I'll be totally honest - I would not have had an issue this week with the boat comp. Maybe it's just the type of photo I am attracted to, but I wasn't into a lot of shots this week. Not to say they were bad photos... just a lot were doing little or nothing for me.

But I just love being able to leave feedback and being able to acknowledge every single photo - I much prefer it to making a heads or tails decision on which photo to delete from my list because I'd selected 8 and I could only pick 6.

Yeah I'm rambling, but the main point it that I think have each photo acknowledged in some way or other is a good thing. Big thumbs up from me. :)

Kym
26-09-2011, 8:53pm
Might have known Kym wouldn't wait until "later in the week" to implement the ranked confirmation view! :D Definitely a good idea and certainly helped to review my ratings of the photos relative to the other entries in the same comp.

Thanks, and this evening was later in the week. :p I had a quick look when I got home, turns out it was very easy to do.


All I need to do now is dust the cobwebs off my camera and put in some comp entries again!

Yeah!!

Kerry Rainer
27-09-2011, 10:16pm
Even though it will take a little bit to get used to, I like the new voting system very much for two main reasons. Firstly I feel it encourages me to look and judge each entry far more closely than in the old system. Secondly I like the fact the each entry will receive feedback in the form of a score. :th3:

bricat
28-09-2011, 12:09am
You should ideally have one 10 and probably none <5 points, assuming all entries meet the theme and are technically ok.

So by that standard you have taken part of the vote away from me? Isn't the voting supposed to be about my perceptions? And I must humbly apologise as I voted not in the manner that was intended by the management as I was confused on the first instance and on the other I have no excuse. However I did vote so as I picked what in my opinion was the best/better photograph. I think I skipped through the rules somehow. damn! Like all change there are teething problems and adjustments need to be made to the voters mindset also. Change is difficult to acheive sometimes. I can't change my wifes' mind..... damn again.
If by all this change you achieve your objectives then it is a worthwhile pursuit cheers

ricktas
28-09-2011, 5:32am
So by that standard you have taken part of the vote away from me?

Not at all, but based on the guidelines (http://www.ausphotography.net.au/forum/showthread.php?92107-The-Ausphotography-Competition-Voting-Guide-for-rating-entries-1-10), the vast majority of competition entries on AP would be 5 and above. However, if members do not use that, and still vote the best entries higher, followed by the mid-range entries and then the lower entries, the system still works. The only reason we said most entries would be 5 and above, is historically on AP, the photographs entered fit the 5 and above, on the guidelines.

Note that the guidelines were constructed by the mods and myself, using a congestion of judging guidelines from camera clubs, along with some resources on judging we had access to, thanks to google.

Kym
28-09-2011, 7:15am
a) So by that standard you have taken part of the vote away from me? b) Isn't the voting supposed to be about my perceptions?

a) Not at all!
b) Yes, but we encourage the use of a guideline.

As Rick said and by the guidelines published an image scoring in the 1-4 range is going to be quite bad in terms of a comp.
5 is going to be the basic comp acceptable level.

Lets face it most entries in comps should score at least 5 - read my assumption again 'assuming entries meet the theme and are technically ok.'
But if an image is not up to scratch for a comp by all means give it a score < 5; but there are very few comp entries that fall into this category.

In the end it wont matter a lot as long as you are consistent.

Kym
28-09-2011, 7:18am
BTW there are over 100 voters ... thank you all!!

This should give a high confidence in the result (statistically speaking).

When everyone sees the result on Sunday ... I think just about everyone will be pleased!

Ms Monny
28-09-2011, 5:18pm
Cheryl said...."On the "downside", I am now suffering from considerable guilt because I couldn't explain why I ranked certain entries with a lower score. Obviously, I cannot discuss the entries, but I can say that in general, in this competition, it was mostly due to the fact that I personally felt that they did not meet the theme. There were a number of very good entries I ranked at the lower end of the scale simply because "ships or boats" were only incidental in the photo."

My thoughts exactly, Cheryl. I too gave a 3 because the 3rd rating had this as a guideline..."A 'nice' entry a.k.a. a record shot. It may have a fault or two, but it isn't working for you for some reason. It probably doesn't have a range of glaring issues, however it really doesn't quite come up to standard as a competition entry."

If the image, in my opinion, is just a 'record' shot with no creativity or even no thought put into the image a.k.a a snap shot, I would rate it a 3. Blurry images too wouldn't fair well, esp combined with the above, as I wouldn't consider them to be competition entries.

I certainly wouldn't give these a 5, which is... "Middle of the range and has some merit. It is a good entry, with some values that make it worthy of being in this competition. This entry isn't up with the high quality entries above, it simply doesn't stand out. However it also is not a photo that has glaring issues that need resolving."

But at least then, the people can see that they need to work a bit more on their images to bring them up to a 5 or even higher.

I too felt guilty, especially when it was pointed out the there shouldn't be any under 5 points, but I know Kym has said that that really isn't the case.

stoogest
28-09-2011, 5:26pm
I scored a few at around 3 or 4 for those same reasons Ms Monny (with no guilt :) ).

WhoDo
28-09-2011, 5:36pm
I too felt guilty, especially when it was pointed out the there shouldn't be any under 5 points, but I know Kym has said that that really isn't the case.
Well, Monika, you've got plenty of company because I rated a couple of images at '3' and even rated one at '2'! What cheesed me off was that, being a mod, I can see that others have rated that last image at '6'! That just go to show that, even though we have a set of guidelines, the vote is still going to be subjective ... sometimes in the extreme. :rolleyes:

There "shouldn't be any under '5'" really is only going true if people aren't invested in their own images when they submit them. I know I've posted images I thought should have done well and they didn't get a single vote! At least this way my "jaundiced" view of the image I gave a '2' will be balanced by those who obviously have seen something in it that I didn't/couldn't. That absolves me of all guilt in the matter, I'd say! :D

Cliff
28-09-2011, 5:43pm
There shouldn't be any guilt if you honestly rate an entry. Even amongst qualified judges opinions of individual photos can vary greatly. I think that because of the number of voters it will all come out in the wash - so to speak.

Kym
28-09-2011, 5:46pm
There shouldn't be any guilt if you honestly rate an entry. Even amongst qualified judges opinions of individual photos can vary greatly. I think that because of the number of voters it will all come out in the wash - so to speak.

And that is what is happening! Rick and I are monitoring the voting and all appears consistent.

Redback51
29-09-2011, 9:00am
Worked well for me. Cheers :food04:

ricktas
29-09-2011, 11:19am
The next big step will be:

www.estyles.com.au photo of the month voting, starting tomorrow (Friday 30th Sept) at 8.45 will be using the rating system

Sunday night (2nd October) at 8.30 the photo of the week winners will be announced and the poll results made public...so you can see you rating.

We look forward to more feedback soon

coolie21
29-09-2011, 12:29pm
Food for thought Alex, but are you possibly mixing medians with means? A median is relatively unaffected by an extreme value, because the median value is the value of the middle score when all scores are placed in order. So the median score of 4,5,5,6 and 10 would be 5. The mean score (i.e. the sum of all scores divided by the number of scores) would certainly be affected by an extreme outlier.


Thanks for the guide, I found it very useful.

I took longer than normal to consider the photos, this will probably improve with practice but the voting could get cumbersome if there were many entries. I felt that the pictures were relatively 'right' for my judgement - again the guide was handy here.

Forgive me if this the following has been discussed previously. Coming from a tech background (! :eek:), I know that using measures of average (median) are sensitive to relatively few extreme values. That is a few, even one, extreme value has much more effect than many non-extreme values. This effect is potentially more significant the less people are doing the scoring. A major concern here is the case of voting for your own picture. I assume most people would vote high for their own photography despite wanting to be fair-minded for the overall process - after all, we put a photo in the comp because we think it is worthy.

For example, assume 20 Ausphotographers have given their score for a photo and the average of the 20 scores is 5 (total of the scores would be 100). Now another 4 people log in and give the same photo scores of 4, 6, 5 and 5. The result would be that the average score (rank) remains at 5 [(100+4+6+5+5) divided by 24]. There doesn't seem anything wrong with this as the photo looks like it is probably a picture worthy of a 5 score. But now a single new user logs in and gives the photo a 10. The mean now rises to 5.2. So one voter has had more effect (4%) than the previous four voters (0%).

The sensitivity of the median to extreme values is also the case for low votes. Potentially relatively few scores of 1 could jeopardise an otherwise reasonable score average.

I think that 'voting' for yourself is ok but the scoring for yourself could be an issue. I do not suggest for one second that any of our fellow Ausphotographers are deliberately unethical (quite the opposite is why I like the site), but in fact the fairest and most ethical competition user will probably be disadvantaged if they accurately judge their own photo compared to others who automatically give their own photo a 10. Now in the case of a fabulous and obvious winner I think the best photo will still get the best rank, but in close run contests voting for your self and strategically voting low for other close competitors could tip the balance.

How much this all matters, I don't know. I also know the site admins do a good job of moderating the competitions for abnormal practices. I still applaude the effort to increase the value of the competitions to all entrants.

Maezyra
29-09-2011, 7:59pm
As a person who usually gets no votes or very few votes, I think this is a GREAT thing!! It'll let me know if my photos are nowhere near standard or if it's just that other people's photos just happen to be better. I'm loving it, please consider keeping it!!

ricktas
29-09-2011, 8:09pm
As a person who usually gets no votes or very few votes, I think this is a GREAT thing!! It'll let me know if my photos are nowhere near standard or if it's just that other people's photos just happen to be better. I'm loving it, please consider keeping it!!

Once we have run a couple more trials to completion so members have a chance to use the system a few times, we will put the concept to a poll and see what members think, from that we will adopt/adapt the competition voting method(s) that we use in future. So if you want it to be permanent, look out for the poll in a few weeks, and have your say.

falke
29-09-2011, 8:49pm
I am ok with it. I liked how in the end after voting, at the confirm page the pictures are listed again in order of points I have given. That ranking is good and a nice double check in case I have made a mistake.

hawko02
01-10-2011, 7:25pm
Hello. I have tried twice to vote for the photo of the month "Sports Fever" tonight. No success as yet. My internet connection is often "sus". Bigpond prepaid wireless on the laptop. Tonight it says a good connection but voting times it out and ausphotography needs me to sign in again and all is lost. I tried just voting for the couple that i particularly liked so they at least would get a vote but a message came up saying I must vote for all 53. I will try again tomorrow. Voting for a lot of entries might be out for me.

Kym
02-10-2011, 8:19am
Hello. I have tried twice to vote for the photo of the month "Sports Fever" tonight. No success as yet. My internet connection is often "sus". Bigpond prepaid wireless on the laptop. Tonight it says a good connection but voting times it out and ausphotography needs me to sign in again and all is lost. I tried just voting for the couple that i particularly liked so they at least would get a vote but a message came up saying I must vote for all 53. I will try again tomorrow. Voting for a lot of entries might be out for me.

If you have time-outs I presume any vote would be a problem as it needs to display all the images.
The new voting is NOT a bigger web transaction than the old vote. Very little data is sent on a vote.

Your timeout should have nothing to do with AP as the AP vBulletin session timeout is 3 hours.

All I can suggest is get to a stronger signal area or use an alternate connection.

Kym
02-10-2011, 7:51pm
FYI Rick has set this weeks POTW as old style voting.

ricktas
02-10-2011, 7:56pm
FYI Rick has set this weeks POTW as old style voting.

Thanks Kym. We thought a variety of both voting systems over a few weeks, would let members more thoroughly gauge which they like better, and in a couple of week's time, we will run a poll asking members which one they prefer as the standard for future competitions.

Dylan & Marianne
02-10-2011, 8:09pm
it made for interesting viewing to see how I rated images when compared with the average - I'm not sure how others scores were with regard to the mean, but there were only a few cases where I judged an image too harshly or too generously when compared to the mean. It was also good then, to go back to that image and perhaps take a relook as to why I perhaps misjudged it. Also, I rated an image from one of the categories highest in the qualifying vote then changed my mind when it came to the final vote after some more consideration. I found it a great experience and thumbs up to the mods for making it happen at least as a trial.

Kym
02-10-2011, 8:18pm
it made for interesting viewing to see how I rated images when compared with the average - I'm not sure how others scores were with regard to the mean, but there were only a few cases where I judged an image too harshly or too generously when compared to the mean. It was also good then, to go back to that image and perhaps take a relook as to why I perhaps misjudged it. Also, I rated an image from one of the categories highest in the qualifying vote then changed my mind when it came to the final vote after some more consideration. I found it a great experience and thumbs up to the mods for making it happen at least as a trial.

I don't think anyone mis-judged. It is simply different view; tastes and so forth.
Eg. at a camera club I'll often (as do others) have a very different opinion from the judge.

ricktas
02-10-2011, 8:20pm
it made for interesting viewing to see how I rated images when compared with the average - I'm not sure how others scores were with regard to the mean, but there were only a few cases where I judged an image too harshly or too generously when compared to the mean. It was also good then, to go back to that image and perhaps take a relook as to why I perhaps misjudged it. Also, I rated an image from one of the categories highest in the qualifying vote then changed my mind when it came to the final vote after some more consideration. I found it a great experience and thumbs up to the mods for making it happen at least as a trial.

Thanks Dylan. And I think that is the crux of it. Members can use the score to evaluate and assess the outcome. It provides more information than simply seeing an entry get 10 votes, 22 votes, 1 vote etc.

stoogest
02-10-2011, 8:27pm
I've just finished analysing my vote compared to the average and it was really fascinating! In general my vote was about 1 higher than the average, but I suspect that is because I was making sure I gave the best image a 10, and scored everything else relative to that (ie. I suspect that the majority of voters were simply giving raw scores out of their own definition of a '10'). This was even more obvious when looking at my favourites, where my scores of 8/9/10 were normally 2-3 marks higher than the average.

Love the change!
Andrew.

ricstew
02-10-2011, 9:52pm
Could someone please point me in the direction for woking out the averaging or scores or the votes allocated......how ever it works lol ??
cheers
Jan

stoogest
02-10-2011, 10:29pm
These votes were only applicable to the Qualifying round. So all you need to do is click on 'Competitions' at the top of the page. The scroll down and click on the 'Qualifying Vote (Poll)' for the POTW that you are interested in. It will then list all the entries in descending order of average vote, with your vote next to it on the right hand side (eg. 6.241 : 8).

Here's one of them: http://www.ausphotography.net.au/forum/showthread.php?92285-Top-4-Poll-203-Photo-of-the-Week-274-Ships-or-Boats-Intermediate

Hope this helps!
Andrew.

ricktas
03-10-2011, 5:50am
Links to all of them are here: http://www.ausphotography.net.au/forum/showthread.php?92654-New-Voting-Rating-System-POTW-274-See-your-score-here

You can also get to the voting polls at any time, via the threads in the POTW forum : http://www.ausphotography.net.au/forum/forumdisplay.php?43-.-Photo-of-the-Week-.
Or for other comps, the relevant sub-forum in the Competitions Forum (http://www.ausphotography.net.au/forum/forumdisplay.php?41-Competitions)

Kym
03-10-2011, 9:18am
Re: the performance index report ...

The performance report is correct.
It divides the raw vote (for points polls) by 10 (i.e. a max of 10 points) so it sort of relates to the old system.
But with the new system you actually always get a vote from everyone not just the top 5.
The new system has more votes so your score will be higher.
If it divided by number of voters then it would be a lower vote.

We will have to live with it.

Kym
09-10-2011, 8:55pm
This weeks 59 entires... for me it was quite straight forward

rene52
10-10-2011, 2:37pm
I was a little dissappointed when I foumd out that the 276 was also being scored this new way - I wouldn't have entered if I had known as I took a lot more time than I wanted scoring each shot - I found by the end that I was just soring in reference to my first scored shot so as not to spend time working out if they met the cirteria. Beacuse I entered and there might be a small chance to actually win it I had to vote - I do like this site and I will continue to cc other users photos as I have the time but now will refrain from entering any comps unless I am sure that the scoring system is something easy as well as quick (this is only my opinion and shouldn't be viewed as a complaint just an observation).

beeps
15-10-2011, 5:35pm
I am a new member and only voted twice on the old system, but have to say that i liked the newer system better as it was a lot easier to rate each picture as I went passed it.

nouveau1
17-10-2011, 12:02pm
hi there all..

do like the new scoring system especially now that one can see one's selection ranked before confirming; sometimes felt some difficulty excluding good shots with the old system. so well done:th3:

just a thought - i find it extremely difficult to be objective about my own entries (not that i enter that frequently!) and have as a consequence never voted/selected my own shots; seems there are a few similar comments/references in the thread. as an idea, should one be prevented from voting for oneself? If it's possible/feasible/easy for the software gurus to apply an averaged (or median?) score of votes from other people to one's entry?

rick

Kym
17-10-2011, 12:30pm
hi there all..

do like the new scoring system especially now that one can see one's selection ranked before confirming; sometimes felt some difficulty excluding good shots with the old system. so well done:th3:

Thanks!


just a thought - i find it extremely difficult to be objective about my own entries (not that i enter that frequently!) and have as a consequence never voted/selected my own shots; seems there are a few similar comments/references in the thread. as an idea, should one be prevented from voting for oneself? If it's possible/feasible/easy for the software gurus to apply an averaged (or median?) score of votes from other people to one's entry?

rick

Yes we could exclude self voting, but this leads to a small statistical distortion if not every entrant votes; so we won't implement that.
Lets face it, I'm sure Julia G voted for herself at the last election :D

nouveau1
17-10-2011, 12:41pm
Thanks!
Lets face it, I'm sure Julia G voted for herself at the last election :D

and that's a good reference?.... bit like saying T Abbot is impartial....:D