The GMC Jam Suggestions Topic

R

Rusty

Guest
I just don't see the problem in explaining things if they need to be explained. It's pretty simple really:
As well as your votes you can also cast votes for the following award categories:
Best Concept - Most unique overall game ideas
Best Devlog - Most enjoyable developer's log to read
Best Graphics - Best use of or design of graphics
Best Sound - Best use of or design of sound effects and music
Best Story - Best theme setting or story included in a game
I am 100% behind being explicit but the only way we're going to be able to avoid confusion is to make sure we're all using the same definition of the same words. It doesn't matter what you call the award, if somebody doesn't understand it and it isn't properly explained to them there is going to be confusion.

It is better for the user, and more effective for everybody involved, to educate rather than to avoid the issue.
 

Alice

Darts addict
Forum Staff
Moderator
Well, then, I guess the voting format remains the same as it was before. I'll keep the "Best Concept" name, too; maybe as one of additions in the Jam player, I'll add brief descriptions for best-of awards, too.

With that said, the last matter to settle before the next Jam...
...the voting system!
And then all hell broke loose...

--------------------

The voting (tallying) system determines how the game rankings should be tallied to final results. Current system is the normalised 1/(rank+1). It means first 3 places take 1/2, 1/3 and 1/4 score respectively, and so on until the last ranked place 1/(n+1), and all remaining games get the score of 1/(n+2). The games are then sorted by the total score.

The advantage is that the system is relatively easy to understand and check; basically, one can make a spreadsheet with scores breakdown and if someone understands how the system works, they can easily verify the results on their own.
Also, it reduces the adverse effects of people completing and voting only on a few games, including some joek game; under non-normalised 1/rank (where unranked games get no score at all) it would give considerable boost to the joke game, likely contrary to the voter's intent who wants the joek game to score as low as possible.
The normalisation however has a drawback - it results in shorter rankings giving somewhat smaller "voting power". In particular, in rankings with top 3 entries only, the third entry gets measly 0.05 worth of score above unranked entries, in contrast to nearly 0.25 score it would get if the ranking was full. Also, the first place would get 0.3 points above unranked entries instead of nearly 0.5.

From what I recall, across the various Jams the following other systems were considered (and some actually used):
- plain "game with most 1st place votes wins"
- top-3 scores summed up (e.g. 1st place gets 7 score, 2nd gets 5 score, 3rd gets 3 score, the game with highest score wins)
- 1/rank (1st place gets 1/1 score, 2nd gets 1/2 score, 3rd gets 1/3 score and so on, unranked entries get 0)
- 1/(rank+1) (1st place gets 1/2 score, 2nd gets 1/3 score, 3rd gets 1/4 score and so on, unranked entries get 0), to soften up the effect of random rankings (e.g. giving 1st place to friend or something)
- repeated instant-runoff voting (i.e. pick 1st place entry using IRV, add it to the end of the ranking, remove the entry from rankings, rinse and repeat on altered rankings until there are no more entries)
- mIRV - a variation on repeated IRV, but I have no idea how it works exactly (apparently it's meant to allow games with lots of 2nd place ranks but no 1st place win against game that scored 1st place in few rankings)
- ranked pairs - it's a cool system, but tricky to understand and has fancy maths going on that goes over people's head; it promotes entries consistently scoring well rather than controversial ones

Another system I considered would be "repeated 1/rank" (or "repeated 1/(rank+1)"; heck, maybe "repeated 7/5/3" would work, too). Basically, pick 1st place using 1/rank, add it to the end of the ranking, remove the entry from rankings, rinse and repeat on altered rankings until there are no more entries.
It allows games with lots of 2nd place ranks to win against those with few 1st place ranks, and the repetition should eventually even out random votes (such as dubious 1st place vote or joek game 10th place vote, because the voter had time to play only 10 games). At the same time, it doesn't lessen the support for voter's picks if the voter ranks few entries (in contrast with the current system).
This system might be slightly damaging to controversial entries compared to single-step 1/rank variations, but not as much as ranked pairs.

The repeated IRV, mIRV, ranked pairs and repeated 1/rank systems cannot be presented in a fancy easy-to-verify spreadsheet, but it's not a problem for me personally. Also, I'm able to make and share a program automatically tallying the results using all the mentioned systems except for mIRV (because I have no idea how it works exactly).

--------------------

One more thing: Should voting be awarded by 1st place's worth of ranking? In other words, if at least one of authors of the given entry casts their vote, should the entry get a bonus 1st-place vote to its advantage?
The rationale behind it is that since you can't vote for your own entry, the very act of voting puts you at a disadvantage. 1st-place vote award makes the voting actually beneficial and might encourage more people to vote (who generally wouldn't mind voting, but care about their rank, too)?
 

dadio

Potato Overlord
GMC Elder
Lol, surely by now everyone is just tired of arguing about the voting system.
Personally, I think continue with what we have been using - it works, and there aren't many (are the any?) complaints.
I would highlight that fun fact you brought up tho:

The normalisation however has a drawback - it results in shorter rankings giving somewhat smaller "voting power". In particular, in rankings with top 3 entries only, the third entry gets measly 0.05 worth of score above unranked entries, in contrast to nearly 0.25 score it would get if the ranking was full. Also, the first place would get 0.3 points above unranked entries instead of nearly 0.5.

This is actually a good thing, it should encourage people to vote for more games, rather than just top 3.
I had missed that fact when scratching my head over a few results, but in short - people should (in the Voting Topic) know that...
The more entries you vote for, the more impact each of your votes has on the result.
 
R

Rusty

Guest
The more entries you vote for, the more impact each of your votes has on the result.
How is this right or fair?

When I vote in the Jam (as I have done in the past) I have made sure to play every entry that has been available to me, yet I am penalized for only voting for the first few and not wanting to do the extra legwork of comparing every single entry against each other. I mean it's just stupid to ask for the community to get involved out of the will of their heart and yet punish those who don't commit to as they would for a paid event.

It's been a while since I last voted in a Jam because I don't really see the point of it anymore. My vote has been made worth less than others.
third entry gets measly 0.05 worth of score above unranked entries, in contrast to nearly 0.25 score it would get if the ranking was full
It would take 6 of me and my opinion to outweight one person's opinion. That isn't even to say that the other person has put more thought or effort into their vote, just that they voted like everybody else does and then just randomly listed the other entries. Even the people who use scoring systems just list the additional entries in order of their overall score, regardless of if they believed a game actually deserved 17th place over places 18th and 19th because nobody cares enough to have a strong opinion that low down the list.

If my vote was worth more, I'd be more likely to take part in the reviewing and voting section of the Jam but as it stands, I don't want to list 15th place but putting my votes up against anybody elses' their 3rd place vote is almost worth as much as my 1st. Pointlessness. It's really a shame because reviewing and helping other developer's is something I really enjoy, just not at the Jam anymore.
 
Last edited:

sylvain_l

Member
One more thing: Should voting be awarded by 1st place's worth of ranking? In other words, if at least one of authors of the given entry casts their vote, should the entry get a bonus 1st-place vote to its advantage?
getting something yes, getting first rank no.

more something like a 3rd place (it's the best I can think of for now)

my idea, is that it's clearly unfair to give nothing for participants who vote as it give full advantage to the other participants, and help participants who don't vote!
But I won't give someone who make a joke vote on 3 entries a full 1st place reward for his particpation.
So something like a 3rd place sounds a good in between.
(it encourage voting. you vote for 3 games... well you gain nothing more than the 3rd and all the others; vote for 4 or more; you get more than the majority. and more you vote more you get compared to most of the other entries. except the 2 first.
It' doesn't prevent the joke vote of randomly ranking all the entries. but in that case you give still more score to 2 entries than yours. And if you vote fairly; most of the time shouldn't be hard to find 2 or 3 others entries as good or better than yours. And for the rest you get a score as good or even better than what you would get if you voted fairly for your own entry)
 

Alice

Darts addict
Forum Staff
Moderator
@Rusty: I understand your point, and I personally agree with the notion that people shouldn't be "penalised" for voting short. In fact, the current system wasn't aimed against the people who play all games but decide to rank top 3. Rather, it was to address the issue with people playing 10 games, ranking all 10 including a joek game they hated, and inadvertently giving significant boost to the 10th place over all the other entries (something like that has happened multiple time in the past). Giving top-3-only votes less weight was an unintended side effect. Back when we tried to think of a voting system that would address partial-play votes problems, we were mostly working with the 1/rank family (non-repeated).

Out of curiosity, does repeated IRV or repeated 1/rank sound better to you than the current system? Both are intended to make votes have the same impact no matter how long they are, with the latter being "fairer" towards consistently-well-scoring-but-rarely-first entries (I remember some people not liking overly favouring 1st places at the cost of 2nd or 3rd places). Also, I recall regular 1/rank giving pretty solid winners in general, so I guess if we first use it to pick a winner, then pick the winner of remaining entries and so on, we could get pretty good results, too...?

(from what you talked earlier, I suppose you would rather just have simple scoring system like 10/5/1 or 7/5/3 or something, but others still care about how the games in further places are doing, and I need to take that into account as well)

@sylvain_l Hmmm, a third place's worth for voters' entries? Doesn't sound half-bad, it does give some boost while not being overpowered, and it's still beneficial to most participants, maybe except for those with reasonable chances for top 3 overall position (not that many, really). I wouldn't mind going with that idea.
 
R

Rusty

Guest
(from what you talked earlier, I suppose you would rather just have simple scoring system like 10/5/1 or 7/5/3 or something, but others still care about how the games in further places are doing, and I need to take that into account as well)
This is more than it ever had to be. This is not a local election, there is nothing up for grabs, there is no reward other than recognition of the other entrants because frankly only the people entering the Jam give a damn about the Jam rankings. It doesn't even commercially matter if a game won the GMC Jam 1st place because by default it has to be free and linked through the forums, so the developer doesn't even earn from it. It never, ever needed to be this complicated. I don't even fully understand why it's being run by the GMC Mod team.

So what if a person plays 10 games and votes for 3? It's their right to vote however they want to as long as they are obeying the rules. You're telling me that new rules were put into place to try and dictate how people are allowed to think. If people want to understand where they rank, 0 votes is last place. It is only in the GMC Jam that I have seen people squabble over which of them is the first of last place. If I get rejected from a job, I don't argue with the other entrants over which of us didn't get the job the most, I take the feedback and try to do better next time.

Saying my game scored 17th isn't really a victory since I know that it's because the GMC Jam is using the dodgiest voting system I've seen in a long while. If you want to more inclusive, push the mandatory votes up to 5 or even 10. Anybody who doesn't get a vote, didn't get a vote. You can't include people who don't qualify for anything. I didn't come third in the "Best Reviewer" category of the last Jam because I did zero reviews. I didn't qualify for a listing.

You're basically ranking broken games, games that don't run, games that crash upon start, games that had no playable qualities and games that simply didn't make the cut for an actual listing and why exactly? What exactly could the developers of these games gain that they couldn't get through the reviews their games got? What exactly could "32nd" tell them that "your game needs graphics" couldn't?
 

dadio

Potato Overlord
GMC Elder
Rusty makes a fair few decent points.
I'd actually like to see a simpler system used, no benefit for voting, and a requirement for 5 positions of votes/reviews...
just to see how it pans out. Pretty much as Rusty is saying there. Could be best for all.

By the way Rusty, Mods run the Jams because you need Mod priveleges to sticky topics and clean/edit the various Jam topics.
 

Alice

Darts addict
Forum Staff
Moderator
Alright, um...

First of all, I might like some systems better than the others based on various advantages and disadvantages, and with that in mind I'll likely highlight the advantages of the systems I like and the dangers of the other systems. If it was up entirely up to me, I'd probably pick something like ranked pairs or repeated 1/rank, because I have a good understanding of these systems and I have general grasp of their properties, and I like these properties. Also, I'd be wary when it comes to using 3-ranks system, because it can come with various odd effects.

However, my opinion isn't any more important than that of the other users. In the end, it should be community that decides which properties are the most desirable and what the tallying system should be used, out of options presented; and I'll make sure to include limited-ranks system among these (like only 3 ranks count, or 5 ranks, or 10 ranks, or whatever). However, it goes the other way as well - no single user's opinion should outweigh the others. And, @Rusty, you wrote your long post in response to this quote of mine:
(from what you talked earlier, I suppose you would rather just have simple scoring system like 10/5/1 or 7/5/3 or something, but others still care about how the games in further places are doing, and I need to take that into account as well)
I assume you quoted that because you disagree with that part, and it's like you don't agree that "I need to take into account that others care about further places". Sorry, but I simply cannot accept that; I won't ignore people's opinion just because one of users is particularly vocal, just like I won't ignore your opinion just because I "know better" (which I don't).

In the end, after the tallying system ideas are gathered together, a poll will be setup. If the people in general don't care about far rankings and think that a simple system will work well enough, such a system will be chosen. If instead they want to "squabble over which of them is the first of last place" and have all games sorted, or just believe a more complex system will reflect the games' popularity more accurately, they will get the more complex system. Depriving them of their "squabbling" despite their will to do so would be like "dictating how people are allowed to think", wouldn't it?

My job here is to ensure that various viable tallying system options are included, and to allow people to make an informed choice on that matter, aware of various advantages and drawbacks of specific systems. It's possible that the tallying system will be chosen over multiple polls (unless one has overwhelming advantage over all of the others), because apparently some people care about the system quite a lot and slight margin of victory in multiple-choice poll wouldn't cut it. Whether I, or any other single user like the system ultimately chosen, it won't change the decision.

Also, just for the record, the system we used recently has been chosen by the community as well (after the Jam #20 came with some really goofy results). One of users (@ghandpivot, I believe) put forward the idea of modifying the 1/rank system by giving all unranked entries shared last+1st place, I suggested some other system, spreadsheets have been made comparing the Jam results for different systems and, eventually, people decided that out of systems presented normalised 1/(rank+1) worked best. So if anyone "put the new rules in place to try and dictate how people are allowed to think" (I still can't fathom where in the world you got that from! It's not like the tallying system changes one's preferences about the games...), they were people themselves. Granted, they could only choose from 1/rank family (dadio's favourite at the time), but the point still stands.

---------------------

With that out of the way, I actually wouldn't mind requiring more ranks for a valid vote, although I'm still not convinced about 3-ranked systems (or other limited-ranks systems) being superior e.g. variations of 1/rank. The thing is, such systems tend to be rather coarse-grained, and can produce some really random results, especially if some people won't be able to play all games. In particular, I did some really quick and crude checks on previous Jam results, and I noticed that, for example, in Jam 21 a perfectly fine and playable game such as "Chasing Night" would be doomed to the last place among other "non-qualified" games (despite getting some 6th or 5th place votes), behind a joke game "dadiomadememakethisgame" that got a single 3rd place score, with the next best ranks being 14th and 20th places (the only interaction in the game was advancing the story).

Sure, some games may have outstanding qualities that go over some voters' head (like specific kind of humour in "dadiomadememakethisgame"), but this basically gives a single voter the power to make some serious shifts in the final rankings. I... don't really like that, to say the least.

(of course, at the same time, I won't complain if people decide to choose a limited-ranks system for the next Jam; if they want it, they'll get it, not much to do about that)
 

GameDevDan

Former Jam Host
Moderator
GMC Elder
I like simple 1/rank.

I'm not a fan of the "punishment" element of 1/(rank+1), which I know many people don't see as a problem for various reasons, but in principle I don't like the idea that someone may choose to rank only the top 10 games because the rest are equally bad, but then have their voting power reduced because we assume they were too lazy to play every entry.

In terms of mitigating the people who vote for only a few games and they're all "joke" games, we could do a system of score += totalEntries - rank; then any entry not ranked by the user could receive a score of totalEntries - (numberRanked+1);

So if an entry is voted 10th out of 54 entries they would receive 44 points. If that person only ranked 20 entries, everyone they didn't rank would get points as if they had been ranked 21st (in this case 21st / 54 entries would give them 33 points each).

The above is trying to do a similar thing to the system we used for the last few jams (by giving a flat number of points to games a user didn't rank) but without punishing people who choose not to differentiate between low quality entries.

Edit: Although obviously the flat rate of totalEntries - rank; means that giving someone first place over second isn't as much of an advantage as a system like 1/vote where the score gets significantly smaller every time.

Edit2: I disagree with Rusty's assertion that ranking the lower games isn't helpful though. Yes beyond about 15th place your position is probably a bit of luck, a bit of randomness, but people like closure :p Just saying "you came joint last with 20 other people" isn't very fulfilling!
 
Last edited:
R

Rusty

Guest
However, my opinion isn't any more important than that of the other users. In the end, it should be community that decides which properties are the most desirable and what the tallying system should be used, out of options presented; and I'll make sure to include limited-ranks system among these (like only 3 ranks count, or 5 ranks, or 10 ranks, or whatever). However, it goes the other way as well - no single user's opinion should outweigh the others.
Well that's a contradictory statement, as my main issue with the voting system is that you can outweigh me, me and 4 other me's. The weight of our opinions are not equal, the influence they have over the final result are not equal and ultimately my opinion on the final ranking is made irrelevant as a one-to-one vote is no longer a tie, but an overwhelming majority in favour of the opposition.

Depriving them of their "squabbling" despite their will to do so would be like "dictating how people are allowed to think", wouldn't it?
Well, no, we are allowed to squabble. There have been arguments against various parts of the Jam (and everything) pretty much since it started (or people were born).

With that out of the way, I actually wouldn't mind requiring more ranks for a valid vote, although I'm still not convinced about 3-ranked systems (or other limited-ranks systems) being superior e.g. variations of 1/rank. The thing is, such systems tend to be rather coarse-grained, and can produce some really random results, especially if some people won't be able to play all games. In particular, I did some really quick and crude checks on previous Jam results, and I noticed that, for example, in Jam 21 a perfectly fine and playable game such as "Chasing Night" would be doomed to the last place among other "non-qualified" games (despite getting some 6th or 5th place votes), behind a joke game "dadiomadememakethisgame" that got a single 3rd place score, with the next best ranks being 14th and 20th places (the only interaction in the game was advancing the story).
Well if we boost the number of required voting placements to 10, so that people have to vote from 1 to 10 to have their votes counted, then all counted votes will be treated equally. But he's the thing with voting systems, the results are usually unpredictable. I mean, we all know of "Boaty McBoatface", right? We all realise that Donald Trump is the president of America? That Britain is leaving the EU? Shock victories happen all the time, it doesn't mean that the system is broken it just means people didn't vote the way you expected them too. If somebody liked "dadiomadememakethisgame" more than "Chasing Night" then that's up to them to decide, not you. If you liked Chasing Night more than dadiomadememakethisgame then that's your opinion, not one shared by every voter. Some people genuinely enjoy the humour of joke entries.

Sure, some games may have outstanding qualities that go over some voters' head (like specific kind of humour in "dadiomadememakethisgame"), but this basically gives a single voter the power to make some serious shifts in the final rankings. I... don't really like that, to say the least.
Your current system allows 1 person to outvote me 5 times over. I'd say that's some serious power to make a serious shift in the final rankings. The only way to prevent "serious shifts" is to give everybody a fair voice. So that 12 votes can whitewash 3 votes. Because under the current system, that just isn't happening. A very small minority of people can pull a serious power play and push an entry up to a ridiculous rank despite the fact that nobody else agrees with them. You're giving preferential voting to mainstay reviewers in a way that destroys any form of importance that a more casual voter might have on a free community event with no real reward or recognition. Something that needs the support of casual developers and voters to grow.

If "Chasing Night" received zero (legal under the voting system) support then it went nowhere. The same way that in the poll you are about to create, if an option gets no poll votes you don't just declare it as second place and dismiss an option you don't personally like.

@GameDevDan nobody finds fulfillment in placing last, but people do. If their entry had been more fulfilling for the voter to play, perhaps they may have found more fulfillment in their final ranking.
 
Last edited:

chance

predictably random
Forum Staff
Moderator
Well that's a contradictory statement, as my main issue with the voting system is that you can outweigh me, me and 4 other me's. The weight of our opinions are not equal, the influence they have over the final result are not equal and ultimately my opinion on the final ranking is made irrelevant as a one-to-one vote is no longer a tie, but an overwhelming majority in favour of the opposition.
Rusty, I think Alice's comment (that you quoted) was referring to the weight of member opinions about the voting system. Not the weight of their vote during the actual Jam itself.


@GameDevDan nobody finds fulfillment in placing last, but people do. If their entry had been more fulfilling for the voter to play, perhaps they may have found more fulfillment in their final ranking.
@Rusty: as often in these discussions, I have difficulty understanding what you're talking about. Are you agreeing with Dan? Disagreeing? Or discussing something else entirely?

Anyway... I don't care what voting system is used. Although I suppose the choice of voting systems does affect the outcome. But I suspect the 10+ top games stay on top, and lower 10+ games stay at the bottom, and the middle 20-30 games get jiggered around a bit. And this varies Jam to Jam, so I don't have strong feeling about the vote system either way.

Guess I'm saying here that the current system is fine with me.
 
R

Rusty

Guest
Rusty, I think Alice's comment (that you quoted) was referring to the weight of member opinions about the voting system. Not the weight of their vote during the actual Jam itself.
Oh I am aware, I just found comedy in her argument of "people's say must be equal" to defend the system of inequal value votes that is currently in place.

@Rusty: as often in these discussions, I have difficulty understanding what you're talking about. Are you agreeing with Dan? Disagreeing? Or discussing something else entirely?
I'm saying you get return on what you give. If a developer gives the voter fulfillment in playing their game, they might later find that the voter will return the favour by giving them an actual ranking position, rather than inventing them one because the last place the developer actually got isn't fluffy enough.
 

Alice

Darts addict
Forum Staff
Moderator
@GameDevDan: In other words, in "totalEntries - rank" when someone votes for top 3, the 1st place of 50 would get 49 points, the 2nd would get 48 points, the 3rd would get 47 points and all the others would get 46 points? Alternately, 1st place gets 3 points, 2nd place gets 2 points, 3rd place gets 1 point and all the other get 0?
That certainly *is* simple and yet sorts out the entries further down the road. At the same time, the effect lessening the power of short votes is still there, though not as severe (and the shorter votes don't mess with final rankings), and it would mostly matter against entries that get similar ranks. Hard to tell how it would work in action, though...
(in particular, whether it would have similar adverse effects as the current system or not)

@Rusty:
I just found comedy in her argument of "people's say must be equal" to defend the system of inequal value votes that is currently in place.
There are two misconceptions here.

First, in the current tallying system my vote can give stronger support for different entries only if you don't rank as many entries as I do. Every voter has an equal chance to make their impact, the tallying system is publicly known and everyone can learn about its implications. If at the time of voting you know that shorter rankings result in weaker support for top entries, and still, for whatever reason, decide not to rank further entries, then it's your responsibility that your votes are outweighed by those who did bother to make full rankings. If you really want your vote to be no less meaningful than any other vote, you put some extra effort into ranking the further entries. Nothing in the tallying system stops anyone from making the strong vote. With that said...

I'm far from defending the current system, and because of the very issue that's raised against it. If someone plays all the games and ranks only top 3, then I believe they shouldn't give significantly weaker support compared to those who made more complete rankings. It's just that the limited-rank system, better in that particular regard, fails in other aspects. If there were 50 entries, and one game was consistently getting 10-ish places from all 50 voters, while another game consistently gets places 40-50 except for a single 3rd-place vote; I don't see how it's fair to the game that's doing better, and clearly preferred by the vast majority of voters (49:1). I really think there are alternatives that don't have the big drawback of the current system, yet work better than limited-rank systems (Dan's system might be a step in a good direction?).

Basically, don't try to convince me that the current system is seriously flawed and should be changed, because I already agree with that. I just specifically find the limited-ranks system (especially 3-ranks) even worse.

(still, as flawed as I find the limited-ranks systems, if people decide to go with one, I will, obviously, carry out their wish)

If you liked Chasing Night more than dadiomadememakethisgame then that's your opinion, not one shared by every voter. Some people genuinely enjoy the joke entries.
Actually, 16 people ranked "Chasing Night" higher than "dadiomadememakethisgame" (leading to overall 18th vs 31st place of 41), many of them having ranked (and thus played) both. The one person who voted the latter over the former apparently didn't play "Chasing Night" in the first place, although there's a good chance they'd still pick "dadiomadememakethisgame", anyway. The point still stands, it's 1 person outweighing 16 and these 16 people have no way to counter that except for changing own preferences. It might even get as ridiculous as one random 3rd place vs 4th place vote overriding preferences of thousand (millions, billions...) 100th place vs 4th place votes; that gives single voters an overwhelming power to change the rankings, incomparable to the differences under the current system.

The current system *doesn't* allow full-ranking voter to outweigh 6 top-3 voters, no matter how you look at it. It takes 2 top-3 voters to outweigh full-ranking 1st place, 3 to outweigh the 2nd place and 5 to match the 3rd place. But even if it did, top-3 voters can add stronger support without changing own ranking; they just need to spend extra time making their ranking more complete. No tactical reordering is needed (like: "I like game A better, but it's doing well enough, so I'll place game B 3rd so that it won't get behind the joke game"), just a little bit more effort with voting.

Once again, I'm not a fan of the current system, let alone a zealous one. But even then, you are seriously downplaying the flaws of the limited-ranks system while grossly exaggerating the flaw of the current system (the "1 vs 6" is not only misleading, ignoring the fact that anyone has equal chance to make a meaningful vote; the 1/6 figure is demonstrably false).

The same way that in the poll you are about to create, if an option gets no poll votes you don't just declare it as second place and dismiss an option you don't personally like.
Do you think I'd stoop so low to even consider downright dismissing the results of the poll I created just because they don't match my hopes/expectations? That's just as rude and disrespectful as it's baseless. >.<

If that's not what you wanted to imply at all, you really should pay attention to what you write. Speaking of dismissing the poll results, especially in the context of the poll I'm about to create (instead of just some hypothetical poll), seriously makes it sound like you find the possibility at least somewhat realistic.


EDIT: Nevermind, that part has been cleared up.
 
Last edited:
  • Like
Reactions: var
R

Rusty

Guest
Do you think I'd stoop so low to even consider downright dismissing the results of the poll I created just because they don't match my hopes/expectations? That's just as rude and disrespectful as it's baseless. >.<

If that's not what you wanted to imply at all, you really should pay attention to what you write. Speaking of dismissing the poll results, especially in the context of the poll I'm about to create (instead of just some hypothetical poll), seriously makes it sound like you find the possibility at least somewhat realistic.
Don't have time to read this yet, just clearing this up. "you don't just declare it", meaning I was giving an example of something you don't do, because I know it's something you wouldn't do.
 

Alice

Darts addict
Forum Staff
Moderator
Aaah, alright then, I take that part back then.

I kinda understood the "you don't just declare" part as something like "you are not supposed to declare", like in "if you see someone has collapsed, you don't just carry on like nothing happened" (but rather call for ambulance and apply first aid, or something). Now that you explained it, I know it was meant more like "you wouldn't just declare" instead. ^^'

Either way, thanks for clarification, and sorry about the misunderstanding.
 
R

Rusty

Guest
Just real quick before I get into replies on that post. @Alice I actually have a lot of respect for you generally. Like I've stated before (in this thread I think), I've seen you around during some of the earlier pre-Jam discussion topics and although I've not always agreed with you, although often we have, I've always received the impression that you've wanted to do what is right and fair.

When I am replying to these posts, anything said is not an attack on you as a person, an opinion or your character, I am simply making an argument against your point. This is not under any assumption that convincing you will somehow effect the final outcome of the vote by defacto decision, but towards the goal of swaying the opinions of the lurkers reading this before casting their votes, which is quite usually my primary focus when debating anything with anybody.

First, in the current tallying system my vote can give stronger support for different entries only if you don't rank as many entries as I do. Every voter has an equal chance to make their impact, the tallying system is publicly known and everyone can learn about its implications. If at the time of voting you know that shorter rankings result in weaker support for top entries, and still, for whatever reason, decide not to rank further entries, then it's your responsibility that your votes are outweighed by those who did bother to make full rankings. If you really want your vote to be no less meaningful than any other vote, you put some extra effort into ranking the further entries. Nothing in the tallying system stops anyone from making the strong vote. With that said...
I have three jobs plus contracts. For the time I already invest in the Jam, it is often a bit of a burden already as it tends to back up those contracts. When I do review Jam games (like I used to), I tend to make a short list then select the most enjoyable and memorable from that list to fill my voting tiers. I most definitely do not have the time, nor any desire, to go through the entire review system again and reselect everything into a specific order. Nor do I wish to randomly list the additional entries. Diluting my voting capability because I don't have the time to go through the process of ranking broken, unplayable and equally mundane games puts me off the idea of spending time doing it at all. Waves against the castle wall and all that.

Basically, don't try to convince me that the current system is seriously flawed and should be changed, because I already agree with that. I just specifically find the limited-ranks system (especially 3-ranks) even worse.
Again, I'm making the argument for the sake of those reading, not for those writing. If you're writing in the argument it's usually safe to assume that your opinion is already set firmly enough to weather the storm of an internet forum debate. So while I will continue to debate this, know that this isn't aimed directly towards you or an attack on you in anyway. I'm simply refuting points so that the points you raise don't go unchallenged in the eyes of the lurker.

Actually, 16 people ranked "Chasing Night" higher than "dadiomadememakethisgame" (leading to overall 18th vs 31st place of 41), many of them having ranked (and thus played) both. The one person who voted the latter over the former apparently didn't play "Chasing Night" in the first place, although there's a good chance they'd still pick "dadiomadememakethisgame", anyway. The point still stands, it's 1 person outweighing 16 and these 16 people have no way to counter that except for changing own preferences. It might even get as ridiculous as one random 3rd place vs 4th place vote overriding preferences of thousand (millions, billions...) 100th place vs 4th place votes; that gives single voters an overwhelming power to change the rankings, incomparable to the differences under the current system.
If they didn't care about the entry strongly enough to give it (legal) support then I doubt they cared or noticed that it placed last. A good counter for this is again, to expand the number of mandatory votes to become more inclusive. If you haven't listed a game in your top 5 or 10 then you can't really complain when it doesn't make the top half of the listings.

they just need to spend extra time making their ranking more complete.
Which is exactly what I am being penalized for under the current system. I make time for the Jam, I (would) make time for the voting, but I have three jobs, plus contracts, plus (something of) a social life. If you're not going to pay me for my time, don't expect a commercial effort from me. I'll give one if I feel like it, but to punish me for not giving it is ridiculous.

the "1 vs 6" is not only misleading
5 to match the 3rd place
So it takes 6 to beat 1. It's not misleading at all. It's a valid example.
 

dadio

Potato Overlord
GMC Elder
No time to properly read all that, and take what I have to say no with a grain of salt cos I'm half asleep and don't have time to read back over the specifics of each system...
but I will just add that I think something that Alice is arguing against/pushing away from is possibly exactly the feature I liked about the system that I used (I think, alotta systems have been used thus far).
My biggest problem with flattening/offsetting/averaging systems is that they negate the impact of those few people who happen to give an otherwise low place game, a 1st 2nd or 3rd place vote. I have probably done exactly this myself in the past when casting votes - and the reason I would have done it would be to get that entry more noticed/ played after the fact. If my 1st, 2nd, or 3rding an "odd" game doesn't really help it's position and visibity... well, then I feel bad for that game that likely took some kind of risk/ pushed it's humour too far/ was more outside the box/ unusual in some way.
I guess what I'm trying to say is something I think I've said before when discussing voting systems...
personally, I'm not a fan of prioritising "average/ acceptable" games over "divisive/ unusual/ creative" games.
I would rather see unusual, "standout" games pushed, which is why (I think) I preferred whatever system I was using for the Jams I ran.

Also @ chance: I think in actuality, all the results jiggle around quite a lot depending on which system is used, and I seem to remember a few 1st, 2nsd and 3rd places shifting around when comparing output from different systems before.

Having said all that, if everyone else/ the majority are fine with the current system, then I see no reason to change it.
And I respect Alice a lot for giving the community a chance to choose the system rather than just dictating what it will be, but I have a feeling the results of polls/ opinions on that will be very mixed.
 

Cat

Member
I have a suggestion for a voting system. Full disclosure: I'm not exactly what you would call gifted with maths, so this suggestion is doubtless horribly flawed :p. I'm mostly just looking for feedback.

What if instead of ranking games comparatively (e.g. 1st, 2nd, 3rd, etc.), they were ranked based off of some rubric. For example:
Gameplay: x/40
Presentation: x/30
Concept: x/10
Audio: x/10
Theme: x/10
Total: x/100

Here's the catch: all unranked games would default to 50%. This way, no one will come in last place simply for not being visible (or everyone will :p). From here, it's simply a matter of averaging each total, then sorting the results. I can see numerous benefits to this method. As for drawbacks, here are a few off the top of my head:
-It's stricter. Not as much room to casually say "I liked A more than C, but B was right in between." and call it a vote.
-It's less dynamic. "I liked the gameplay." How much?
-Probably a lot more, please enlighten me :p

A solution to the latter problem could be to build an automated voter, with a dialogue such as "Hi there, how much did you like the gameplay? a. Loved it b. It was good c. Had some problems d. Hated it". It could then translate the results to "a. 40 b. 28 c. 16 d. 4".

One benefit of this is that there is no minimum voting requirement. The only reason for one to exist in this system would be to prevent fake votes, and this system was designed with the assumption that people wouldn't try to cheat (which is never a good assumption :p).

In case this needs clarification, I have nothing against any of the previously used / currently being discussed voting systems.
 
Game jams are more srs business than I thought.

I don't care how votes are counted, but I do wish monetary prizes were allowed, so I'd have more incentive to join these things. X'D
 

Alice

Darts addict
Forum Staff
Moderator
@Rusty: Whatever. I do think that the system shouldn't underpower people who can't be bothered, for some reason or another, to rank all entries.

Just a technical note about 1 vs 6 thing: you are mostly* correct that it takes 6 top-3 3rd place votes to outvote 1 full-ranking 3rd place vote. However, you specifically said: "you can outweigh me, me and 4 other me's" (i.e. total 6 "me's") and, before the edit, "Your current system allows 1 person to outvote me 6 times". The figures weren't out of nowhere, but you were using them incorrectly ("1 person > 6" instead of original "it takes 6 to be more than 1").

* Now that I think about it, technically it takes "only" 5 people to outvote 1 at 3rd place, because 1 person gives 0.25 minus a little bit, while each of 5 people gives exactly 0.05 to the score, but I didn't really pronounce the "minus a little bit" and myself said that "5 match 1".

Either way, I'll gladly replace the current system with an alternative that gives proper weight to top 3 rankings.

@dadio: I understand the need to promote outstanding entries and, in general, I don't mind that myself. The troubling thing is the trade-off: the more you promote the outstanding entries, the more the system is prone to random votes (single people making shifts against common opinion), and stabilising the system causes the outstands to fade. Basically, that's something that requires balancing. I'm open to the idea that, sometimes, 3 people loving one entry can outvote 10 people disliking the entry and preferring some other. However, if 1 person loves the entry and others generally find it to be bottom-half material, while there's another game that's consistently liked among sizeable chunk of voters (say, top 10-20%), I feel that other game should be preferred.

@Cat: So, a rating-based system instead of ranking-based system?

I'm open to the idea, but instead of having specific categories I'd rather have people rate games on scale from 0 to 1 (or 0 to 10, or 0 to 100; basically, a scale between min and max). To avoid situation with some voter being more critical than the others, and rating only a few games, it could be made so that lowest game always gets 0, highest game always gets 1, and all the others are spread proportionally between. Then, the game scores would be averaged and sorted, and the game with the highest average score wins.

I imagine this system might be easier to use, or at least deciding if the game should get 54/100 or 55/100 sounds easier than deciding if one 50-ish game is better than the other 50-ish game. At the same time, one should score all the games they're playing (instead of selected favourites), so that there's a comparison to other entries. The games that wouldn't be scored would simply have their score unaffected.

One possible drawback I see is that this system would heavily lean towards averages rather than stand-outs (as explained by dadio). Another, possibly more serious: if someone happens to play mostly good games or mostly bad games, they could lower the score of some decent entries or lift the score of good ones. Still, I think it's a system worth considering.

--------------------

In general, there's another thing I dislike the limited-rank system for: it doesn't fully take one's preferences into account. It means that if I have 15 games I particularly like and want to win over the other 35 games, but there are only relevant 10 spots, then I have no way to support the remaining 5 games.

Ideally, I'd like the tallying system to allow one to rank as many games as one cares about. If someone cares only about their top 3, they vote for the top 3 (and give about as strong support to the top 3 as others' support for their top 3). If someone cares about top 7, or top 12, or top 25, or top everything, I'd like the system to take into account all of these tops one cares about (the point about strength of the support remains). Arbitrarily deciding that one can have exactly 3, 5, or 10 favourites and doesn't care about the remaining entries will result in top 3 or top 7 voter arbitrarily picking additional "favourites" they don't care about, and top 12, top 25 or full-ranking voters to have their actual favourites completely ignored. At the same time, having top 3 voter pick entries they don't care about just so that they can give stronger voting support isn't favourable, either.

--------------------

I wonder if I should make some simulations to try out different tallying systems properties in practice. In particular:

- outstands: how much it promotes outstanding entries (entries that score very high a few times, but mostly somewhat low) compared to all-round entries (entries consistently scoring well, but without extra high positions)
- inertia: how much shift in rankings a single voter can cause against 20, 100, 500 voters (dunno how about the others, but I feel that, in general, single voter's power should diminish with more existing votes)
- preferences: how much the final results match pairwise preferences between entries; it doesn't necessarily need to be 100% match, but something like 1 preference overriding 10 should be probably avoided...
- clones: how many copies of voter A, B, C, D, etc. it takes to make the ranking match their top 3/full preferences (if the system is equal, it should take similar number of clones; I'd expect the current system would put top-3 clones at disadvantage)

This should give a good overview of how different tallying systems work in practice, I guess?
 
Last edited:

chance

predictably random
Forum Staff
Moderator
Alice said:
I wonder if I should make some simulations to try out different tallying systems properties in practice. In particular:
<snip)
This should give a good overview of how different tallying systems work in practice, I guess?
Those areas capture this discussion nicely, so your simulation results would be interesting. It wouldn't necessarily settle the debate, of course. After all, some members might actually favor systems where a few renegade voters could push an outlying game up the charts. And some members might like top-tier preferences remaining fluid, even with many similar votes.

In other words, what some voters view as flaws in a particular system, others might consider desirable.

That said, the simulation could make people better informed about how particular systems work. So that's good.
 
Last edited:
S

Storyteller

Guest
what about adding in an objective measure as well as subjective?
currently, systems mentioned are highly subjective and related almost entirely to whim and opinion.
Is it possible to develop a heuristic that can be applied objectively to games that can be reduced to an ordinal or linear scale?

Perhaps a series of categories (interface, graphics, music, lack of bugs, completeness etc) can be developed and then a likert type scale can be used.

Alternatively, and more objectively, a series of very specific criteria can be established, (frame rate at a given resolution, number of hazards, penalty for hazard severity, risk reward ration, time to beat, compliance with color theory, use of flow theory etc), at which point they could be scored. While some of those things may still be subjective, much is very objective and not open to interpretation. At the end of the day, some people like certain kinds of games more than others. Objectivity should be included, and quantitative measures should be employed where possible.

***

It is also possible to apply statistical analysis to voting systems to eliminate outliers and not get a simple average, such as X-squared (chi squared).
Perhaps also using the standard deviation could help illuminate some of the scores.

That is:

'game A scored 85 +/- 6, while game B scored 82 +/- 3, game C scored a 70, +/- 25'

These are much different answers than,

'game A scored 85, game B scored 82, game C scored 70.'

Thus we can see most people did not like game C, but some who did, really liked it. That indicates a certain subset of players will enjoy it more than others.
Game A is the obvious winner, but by a low margin. We can see that it got a higher score than B, but we can also see its range means some scored it lower than B. B however got lower scores over all, but were more consistent. There was more variance in A, some scored it much lower than B, while B did score higher than A some, its overall score was lower, but there was less fluctuation in that score. In short, A might have scored 91 vs B at 85 (or 79), but A could also have scored 79, vs B at 85 (or 79). So, including some standard deviation alone might be a worthy addition, not to mention some of the more robust statistical analysis.

either way, developing a heuristic that is based on objective criteria would be good. aesthetic heuristics should be developed for games, as they have been for art, music and theatre.
 

Alice

Darts addict
Forum Staff
Moderator
@Storyteller Nononononono, objective measures won't work. The voters are people, not auditors, and neither I nor anyone else can arbitrarily decide what makes a "good" or "bad" game and expect voters to follow. Neither I am competent to decide which parts of game design make a good or bad game; it strongly depends on both the context and the individual preferences.

Whether the game is generally liked or not might not be a perfect indication of its quality, but I feel it generally works better than measuring some arbitrarily chosen objective values. A system that promotes outstanding entries (loved by few) by commonly popular (liked by many) could also give some boost to the games that are particularly great in specific categories or belong to a niche genre (there's some results stability cost associated with that, though).

@sylvain_l I'm open to various ranking systems, but I'd rather keep them deterministic. With deterministic system, one can gather the votes, perform computations on their own side and get the same final ranking. With probabilistic system the results can be different every time, and so the results cannot be easily verified. ^^'

Probabilistic systems might be fun to consider academically, but I seriously doubt they'd work in Jam conditions...
 

chance

predictably random
Forum Staff
Moderator
Alternatively, and more objectively, a series of very specific criteria can be established, (frame rate at a given resolution, number of hazards, penalty for hazard severity, risk reward ration, time to beat, compliance with color theory, use of flow theory etc), at which point they could be scored.
One can only imagine the flood of interesting games such a scoring criteria would produce. And so original.
 
S

Storyteller

Guest
@Storyteller Nononononono, objective measures won't work. The voters are people, not auditors, and neither I nor anyone else can arbitrarily decide what makes a "good" or "bad" game and expect voters to follow. Neither I am competent to decide which parts of game design make a good or bad game; it strongly depends on both the context and the individual preferences.

Whether the game is generally liked or not might not be a perfect indication of its quality, but I feel it generally works better than measuring some arbitrarily chosen objective values. A system that promotes outstanding entries (loved by few) by commonly popular (liked by many) could also give some boost to the games that are particularly great in specific categories or belong to a niche genre (there's some results stability cost associated with that, though).

@sylvain_l I'm open to various ranking systems, but I'd rather keep them deterministic. With deterministic system, one can gather the votes, perform computations on their own side and get the same final ranking. With probabilistic system the results can be different every time, and so the results cannot be easily verified. ^^'

Probabilistic systems might be fun to consider academically, but I seriously doubt they'd work in Jam conditions...
Yesyesyesyes. Objective measures are generally better than subjective. Objective measures tell why so many people like game a vs game b. Subjective rating/ranking only tells you that some people claim to like a game.

the key is to not choose those objective values arbitrarily.

aesthetics is devoted to determining why a piece of art or music is pleasing to many. in art, paintings, you examine how the eye flows around the canvas, color contrast and compliment, form etc. This is how we got to 'color theory'. These things are the basis of 'Art Appreciation 101'.

what I am saying, is that games as an art form are primed to mature in this fashion. In some few years, we may see 'Game Appreciation' alongside Art, Music and Theater. Identifying those criteria will help shape said branch of aesthetics.

It may be beyond the scope of this forum or jam, but a system that can not be influenced by one or two people who use popularism to sway other votes, that is based on criteria vs opinion is inherently better. It can tell each entrant exactly what they did that worked and what did not. As said, it may be out of scope for a small community of artist-coders in this atelier, but then again, small groups and individuals have shaped art throughout history.

Imagine which small community establishes these guidelines, these heuristics, will potentially shape games as an artform for decades.

@chance limitations encourage creativity. Ive heard it said here a dozen times. consider a criteria 'functional' or 'complete' that establishes a minimum set of things each game must have, that it loads, runs and exits without crashing, that it is a game and not a non-interactive demo, that there is a goal or purpose, etc. I do not intend to establish those criteria at present, yet I contend some form of these criteria are beneficial. Having a 'functional' category would help ensure that games were finished and bug free, that developers would be encouraged to polish their game, even a short game, with some level of quality control.

edit: added 'not' to a sentence for clarity
 
Last edited by a moderator:

chance

predictably random
Forum Staff
Moderator
Objective measures are generally better than subjective. Objective measures tell why so many people like game a vs game b. Subjective rating/ranking only tells you that some people claim to like a game.
I don't think you've thought this through carefully. Or perhaps you aren't explaining yourself well.

I agree there are some widely accepted objective factors -- such as having functional controls, being finished, and bug free (as you said). So of course, voters take those into account. But beyond those obvious factors, we have subjective feelings about what's innovative, visually pleasing, and most of all fun. Those vary by person, and cannot be determined by objective formula.

Objective criteria can explain why some games make the top 10, and why some are near the bottom. But deciding among those top games to pick winners is a subjective exercise.
 
I have an idea for a system. This is certainly not perfect, but at the very least I hope it can provide a new idea or two. Personally, I don't care that much, I just want to make a game. So feel free to completely ignore this if you want too.

To start, I think that a review/rating system would be WAY easier to actually vote with. (Bear with me). It's extraordinarily difficult to decide where games should be ranked compared to each other. When there's multiple games that are almost equally cool, it's not really fair that there is no way to just give them the same amount of points. A review system between 0-100 (with separate categories) would not only allow more specific feedback and fair reviews, but it would allow multiple really good games to have the same amount of points. From a voting perspective, reviewing is just plain easier and more fun than ranking them, (at least personally). However, this doesn't really lead to fair rankings. After all, a person who want a certain game to win just gives that one 100/100, and the rest of them 0. Other problems have already been discussed above as well.

My suggestion involves two phases, a review phase and a voting phase.

Reviewing: Reviewers can review the games on a scale of 0-100, based on several categories that combine to make the final score. They can add notes about each section (using a newly modified voting tool, possibly) and post their reviews to the GMC like normal, or link to external files that have the reviews. The purpose of the reviews is only so that the people who made games can see some valuable feedback on their project, and learn from their fellow designers. I feel like the GMC Jam should focus on building a community of game designers, after all, there's no monetary prizes so it's not like anyone here is hoping to make big bucks on this.

During the reviewing phase, each reviewer may also nominate 1 game per "best of" category. Like best aesthetics, best sound design, best concept, best gameplay, etc.

Voting: After the reviews are in (two weeks or so), the voting phase happens. During this phase, everyone may vote for any of the nominated games for them to win each category. The game with the most votes for each category wins that category. After this, the top three are chosen based on how many votes they received during the voting phase, all the votes they got from every category they were nominated for combined). This means the top games are the best games overall, as they had the most votes for them in every category. (The best overall games might not necessarily be the best in any of their categories though).

Pros:
-Because the reviews don't truly effect the final victors, every single reviewer is free to review as many or as few games as they want, no pressure. The goal is once again, only to provide some community feedback for our GM designers and developers. (In order to nominate a game for a category, they do have to review the game though).
-The voting phase is easier as well, because we'd only be choosing the best of each category, and it comes down to a simple majority rules. Also, the voters can look at some of the reviews if they don't have time to play all of the games, AND the voting is narrowed down to the games that convinced the reviewers to nominate them.

Cons:
-This doesn't let everyone know how well they ranked. However, they can still see how their reviews compared to others, and if we really wanted to, we could rank all of the games based on how many nominations and votes they got. Personally, I don't think it should matter so much who wins or looses though. Once again, there's no money here, I think it'd be more fun if we were less stressed out about exactly where each game got ranked.
-Reviewing takes longer than voting. Obviously, writing out long reviews will take way longer than the shorter voting form we use now. However, because the reviews don't actually effect score, once again, people don't have to review every game. As long as every game got a few reviews, the voting phase is equal chances for everyone. Also, the voters can still play the games to decide who they vote for in each category, even if they don't write reviews. Finally, some people have been writing full reviews for everyone every Jam anyways. (Major props to those people!)
-It's new and unfamiliar.

That's all I've got for now. Obviously there are problems with this system, but I hope I at least gave everyone some new ideas. If there are glaring holes in this system I missed (other than you just don't like it or something) please comment and let's see what we can come up with.

In the long run, I don't care really. Like I said, I just want to make games.
 
Last edited:
In two months, we'll send out ballots to each district, and they will vote among a number of representatives who will convene together in two houses. These two houses will independantly come up with a master voting proposal and then combine the ideas together to make a final proposal everyone likes. The president (Alice) will then veto this and initiate an executive order stating that no voting proposals are needed and that the Jam theme, and the winners, will be decided by a board of penguins in an Australian zoo. Then, the theme will be released, which is "fish", and we will all work feverishly to make games for the Jam. When we send in the games, we will find out that the penguins have all starved to death, because they weren't saying the theme was "fish", they were saying they needed to eat.
 

Aviox

Member
I like the concept of two voting phases. For example:

Phase 1
  • All creators (and any additional voters) play the games and simply "+1" up to 15 entries. (ceiling could change)
    • creators can't +1 their own game (it would be pointless because it would just cancel out votes)
    • creators must +1 at least 10 games for their entry to be eligible to make it to phase 2.
  • All the votes are counted (but not revealed), and the top 10 games make it to phase 2. 10-15 make it to "honorable mentions" (but aren't disclosed until the end)
  • all special mention (e.g. best use of theme) nominations are made in this phase (they don't have to be from the voters allotted +1 votes, but can be). the topmost nominations also make it phase 2, even if they aren't in the top 10.
  • if people want to write reviews for games during this phase just for fun, that's fine.
Phase 2
  • All voters replay (probably with more depth) rank the top 10 (+special mention top nominees) games in their favorite order (same as current voting system)
  • revoting on special mentions seems redundant, but could be done?
  • Reviews of top 10 are encouraged from all voters
Pros
  • Voting is simplified. Voters don't have to play and review 80 games and mentally torture themselves putting a lot of review effort into the lower-tier entries.
  • de-incentivizes entries that are intentionally bad, because they won't get much attention.
  • removes the gut-punch for the lowest ranked games. Creators will only know if their games made it to the top 10 or not, rather than knowing "I ranked 56 out of 60." (or whatever)
Cons / other thoughts
  • May discourage voting in phase 2 for the creators that know their game didn't make it to top 10.
  • "top 10" is arbitrary. top 15, top 20, etc. could also work. Much more than that defeats the purpose.
  • A two-phase voting process doesn't have to look like this exactly. I am mainly looking to put less load on voters when jams have a lot of entries.
 
R

Rusty

Guest
Phase 2
All voters
replay (probably with more depth)
No.

I like the idea of two voting rounds, the first to determine a top 10 or etc and the second round to order those qualifying entries. It limits vote division significantly, so we end up with a result everybody is generally happy with.

Asking people to devote so much more time though makes the entire point against the current voting system redundant.
 

sylvain_l

Member
  • All voters replay (probably with more depth) rank the top 10
count on me to just repost those entries in the same order they where in round one.

the last jams I played&ranked all the entries, dedicating enough time to evaluate all on a list of personnal parameters and then compute a note based on.
(the result is still very subjective, as I can encounter or not a bug, other have/don't have; or I like/dislike the genra, I un/luckyly did/n't find a critical key to advance in the platformer, etc... but it's why we can all vote, to get the community evaluation, not just mine)
 

Alice

Darts addict
Forum Staff
Moderator
@Aviox: I feel like two voting phases may be unnecessarily complicated and bothersome for some voters...

...but how about having partial results posted after the first week or so, so that particularly busy voters can decide between the games that are close to each other in ranks? Those who want to rank all games still have a possibility to do so and can ignore the partial results altogether.

(also, I still haven't forgotten about the thread, and still consider making the tallying system simulation; I just have been busy recently and tallying system doesn't need to be decided as soon as, say, themes handling, anyway, so it takes slightly lower priority for now)
 

chance

predictably random
Forum Staff
Moderator
...but how about having partial results posted after the first week or so, so that particularly busy voters can decide between the games that are close to each other in ranks?
I'd recommend against this. Early poll results can cause a "bandwagon" effect on later votes. A similar bias can arise in political elections from exit polling and media forecasts. And in countries with many time zones (US and Canada), eastern polls close well before western polls -- which can lead to bias and reduced voter turn out in western districts.

If "busy voters" want to scan the voting topic themselves, and focus only on popular entries, that's their business. But I wouldn't want to encourage this by helping them.
 

Cat

Member
We could use a simple ternary system for voting. After playing a game, users can vote 1. I enjoyed my time with this entry, 0. I was neutral or ambivalent towards (or I did not play) this entry, or -1. I did not enjoy my time with this entry. I feel that the best way to get more people to vote would be to remove as many barriers as possible. While this does result in less precise voting, it has no effect on feedback quality. If anything, this method could facilitate feedback. If I enjoy both a joek game and a very high quality entry, I'll probably want to explain the difference.

One problem with this is that polarizing entries would be punished. Additionally, the final "score" for each game should be either hidden or adjusted. If I'm new to GMS and just barely get my game to a playable state for the Jam, I'll likely be rather proud. Nothing kills that quite as quickly as receiving a negative number for a score. :p
 
R

Rusty

Guest
Two voting phases sounds fine to me, it stops vote division. Tally up all the placing votes (1st-3rd place) and organize the top 10. Then put the top 10 to a single vote poll format to organize them into their final ranking order.

It's pretty much how the French election is currently being held. The top are selected and then people choose what they want most out of the top ranking options so that they went up with somebody who is (at least partially) representative of a true majority.
 

chance

predictably random
Forum Staff
Moderator
Two voting phases works fine if most voters participate in both voting rounds. But if significant numbers of voters only vote once, the outcome may be skewed by a minority. Either the 1st-round choices, or the 2nd-round winners could be determined by a minority of voters, if lots of voters are too bothered to vote in both rounds.

We already have enough trouble getting people to vote once. I don't think we're ready for a two-round vote.
 
W

Wraithious

Guest
Hmmm I think the creators should not vote for the games, instead, the rest of the comunity should vote for the best games with a rule that they must play all the games.
 

Cat

Member
Hmmm I think the creators should not vote for the games, instead, the rest of the comunity should vote for the best games with a rule that they must play all the games.
This... is not going to work, I'm afraid. Firstly, by eliminating the creators, arguably the most invested portion of the voters is eliminated. This essentially bars ~50 people from voting because... reasons? Secondly, suggesting that any given voter must play all 50+ entries winnows the pool of eligible voters down to around three people. Three. While this isn't necessarily a bad thing, it essentially reforms the voting process from a democracy into a council of people with more free time (and in some cases, willpower :p) than everyone else. While this might (might) result in higher quality voting, in effect, it just excludes some odd 50 people.

Then comes the issue of feedback. Is one allowed to leave feedback if one doesn't match the specified criteria? If you exclude Jam entrants, this potentially results in the feedback being given by "armchair" developers. If you exclude those who don't have time to play through every entry, you wind up with almost no feedback being given, especially to the lower ranking entries. The simple solution is to say "That's ludicrous, anyone should be allowed to leave feedback.". Well then, why is feedback valued, but ranking not? What really sets the two apart? To be honest, I'd be fine with a Jam in which there are no places, no voting, and only feedback being given. However, competitions are fun, because it feels good to win (and I don't begrudge anyone that), so I doubt we'll ever see something like this.

Clarification: I really don't have a preference as to whether or not the Jam is competitive. I'd just much rather receive actual thoughts from people who played my game than a number signifying that on average, people preferred my game to certain others.

It's pretty much how the French election is currently being held.
However, this isn't the French election, wherein the winner takes all. Here, being second is better than third, and so on. In an election, (to my understanding) second through last place get the exact same "prize": watching the winner accept the title.

representative of a true majority
Howso? It seems like it does quite the opposite: allowing people to shuffle their votes around when they see that their candidate has no chance. Imagine we have three candidates: Red, Blue, and Cyan. Red receives 45% of the vote; Blue, 35%; and Cyan, 20%. Red is the clear winner. However, when Round 2 commences and Cyan voters see that they have no chance, they immediately jump ship for Blue (as Cyan has more in common with Blue than Red :p), and push blue into the lead with 55%. It really seems like the second phase of voting is more about which entries people don't want to win.
 
W

Wraithious

Guest
This... is not going to work, I'm afraid. Firstly, by eliminating the creators, arguably the most invested portion of the voters is eliminated. This essentially bars ~50 people from voting because... reasons?
you're right on this, but my reasoning is no biased voting, the rest of the community would only care what game is the best, and I really think that if you don't play all the games it is extremely unfair
 

ghandpivot

Member
I feel like this is getting very complicated. I'd like to see small, impactful changes over time so we always can see what works and what doesn't.
For the next jam I'd love to see the voting system from the previous jam but with something to reduce the amount of damage you take from voting on other entries, e.g getting a certain amount of score for posting votes. I feel like that is enough, do people really want to revamp the entire thing? That's not the feeling I've been getting during the jams.
 

dadio

Potato Overlord
GMC Elder
I agree with ghandpivot. There weren't many (or any?) complaints about the voting system last time.
 

chance

predictably random
Forum Staff
Moderator
...do people really want to revamp the entire thing?
In this case, I don't think so. People wander into this topic, leave suggestions, then leave to wait for the next Jam. As @dadio said, I don't think this discussion reflects widespread disappointment with the voting system.

But it's useful to have these discussions periodically. If a majority supports a particular change, @Alice will consider it.
 
R

Rusty

Guest
I agree with ghandpivot. There weren't many (or any?) complaints about the voting system last time.
The last jam voting turnout was 17. 16 if you don't count the organizer (Alice, as she is the organizer and therefore does not count towards event turnout). So the previous voting system was used by 16 people (Since FunkyB posted but didn't vote).

I have been quiet with the Jam for some time, in the past when I tried to discuss such dissatisfactions I was told by that no such discussion would be held in regard by the organizer (then Nocturne). I have not continued to waste my time by complaining about it but please do not mistake my silence as my acceptance nor anyone else's. Instead I ask you to consider the 16 person turnout in the last Jam's voting phase. Less than half the number of entries, around 40% of the number of entrants and a fraction of the community now so small it's becoming less of a community event and more of an event for a niche group of Jam loyalists.

What Alice is doing here is trying to reach out beyond the select group of Jam participants to re-engage the wider community. Please don't insult her efforts here with such statements as "there is nothing wrong". If there is an argument to be made for the current voting system then discussing that may prove benefitial, but "there weren't many complaints about the last voting system" is not a real argument after most of the opposition to the introduction of new and complicated systems were ignored, disillusioned and eventually abandoned the event long before this new and ridiculous system was placed.

Howso? It seems like it does quite the opposite: allowing people to shuffle their votes around when they see that their candidate has no chance. Imagine we have three candidates: Red, Blue, and Cyan. Red receives 45% of the vote; Blue, 35%; and Cyan, 20%. Red is the clear winner. However, when Round 2 commences and Cyan voters see that they have no chance, they immediately jump ship for Blue (as Cyan has more in common with Blue than Red :p), and push blue into the lead with 55%. It really seems like the second phase of voting is more about which entries people don't want to win.
If 55% of the people decide that Blue is a better game than Red than a majority have spoken and a majority have decided that outcome. Besides, what you're forgetting is that the Jam is not a three horse race. Say that Red gets 30% of the votes, Blue gets 25%, Green gets 20% and Cyan gets 15% and Yellow gets 10%. Putting Red, Green and Blue through to a second voting round allows the Cyan and Yellow voters to re-vote for an entry that stands a better chance of winning. Cyan will be in 4th place, and Yellow is 5th, but Red, Green and Blue will be re-organized to standard, that by your own example, "55%" of the voters are now happy(ier) with. What exactly is 55% if not a majority? In this system, at the very least, 34% of the voters have selected the winning entry.
 
Last edited:

Alice

Darts addict
Forum Staff
Moderator
Hmmmm, to be fair, the voting system isn't exactly my *immediate* priority, either. I'd like to continue that discussion, but with the next Jam drawing near I might not have time to wrap it before then.

The easiest way for now would be just to keep things the way they are, or indeed, try out the suggestion with the bonus for voting participants (either 1st place's worth or 3rd place's worth of vote seem most intuitive, if we were to use it).

Another alternative I could consider for the next Jam would be using regular 1/(rank+1) instead of normalised 1/(rank+1) (current). Both give 1/2, 1/3, 1/4 etc. points to consecutive ranked entries and the regular system gives no points to unranked entries, while the normalised one gives last+1st place's worth of score instead.

Going by the recent votes, barely anyone plays only a fraction of games and ranks them all (they were mostly the reason why normalisation was introduced), and we might try to minimise that even further by informing people that the games ranked last still get support compared to games not ranked at all (e.g. by Jam Player showing a proper dialog if someone exports their rank despite not having played all games). On the other hand, while the normalised system weakens short votes, regular system in turn weakens complete votes (i.e. effective support for top 3 games is stronger if someone ranks only top 3, rather than top 10 or all games), and discouraging full ranking doesn't sound desirable, either.

Yet another option would be fixed-ranks system (i.e. a fixed number of votes count, the rest doesn't, like @Rusty proposes), as it removes the problem with different ranking lengths altogether (at the cost of ranking flexibility). If someone has other relatively simple that would make votes equal regardless of games ranked (while still keeping the ranking-based vote structure), I wouldn't mind.

--------------------

Note that these are temporary, relatively simple measures for the next Jam. It's just that with the next Jam around the corner I'd rather prioritize my efforts on Jam Player (there is some demand for additions there, and I plan making it open-sourced so that C#-literate Jammers could possibly help out, too) and other areas. To me, it feels like there are lots of options considered regarding voting system, with ranking-based systems (like currently) and score-based systems alike (like e.g. Ludum Dare; where voter assigns scores rather than orders the games), that deciding on the options available is going to take considerable effort that might be better spent on other improvements first.

Not saying I won't have time to resolve the voting system before the next Jam, but I might not. If I won't make it till then, we might bring back the voting system debate right after the Jam. In fact, if that discussion would be carried out around the Jam time, other people who don't visit the Jam forum so often might give their input, too (in particular, what they found difficult about voting and/or what eventually stopped them from making a vote).

I still can make some simple changes to try out for the upcoming Jam (in particular, rewarding voting participants and/or changing the tallying system a bit), but for the time being I'd rather wrap this matter up and leave for later, until other areas (simpler and more "rewarding") are addressed. I hope suspending the voting system debate for now isn't too much of a problem?
 

Aviox

Member
count on me to just repost those entries in the same order they where in round one.
I think that would be fine. Replaying might not be necessary for some voters. I just know that I would put extra effort into playing through the whole game for the top 10 games in a two-phase voting system, as on the first playthrough I'm just trying to "get the feel" for each entry. Remember, phase 1 wouldn't be about deciding ranking. It's just about "upvoting" 10 of your favorite entries.

I would enjoy giving my full attention to the top 10 games. I lose steam when trying to give full effort to every entry.

To make a 2-phase system even easier, round two could also just be about "upvoting" your top 3 favorite entries, rather than ranking them.

I guess I just don't like ranking 50+ games. all the middle ones don't get well placed.
 
Top