Defining controlled comparisons

Continuing the discussion from Patron based proposal for mechanism 1.1 (instead of $-based goals):

That’s not how to do study something, that’s exactly how to refuse to study something. There is no such definition about them staying the same. That’s circular.

If you want to see whether some effect exists, you must start with comparable situations. Then, you make something different, then you see if they end up differently with the same input. That is how you find evidence for the effect.

Common example: we study if a medication has an effect by having two groups of people, choose the one difference to be whether they take the medication or a placebo pill, observe if they come out differently.

We don’t decide in advance that if the people have different reactions that the experiment was wrong. The whole point is to see the difference. If they stay the same, it means the medication has no effect.

In our case, if the two situations stay the same, then it means there’s no difference between the two mechanisms. Because they change when given the same input, it means they behave differently.

In my example, you take as a given the starting point of the same crowd with the same budgets giving the same money, and you add to it the same added pledges. The one variable we are testing is whether we define the goal as crowd-size or dollar-amount. We observe that changing that one variable results in different outputs. Therefore, the two mechanisms do not behave the same way to these inputs.

1 Appreciation

tl;dr: all that’s needed to observe objective mathematical differences between crowd-goal and patron-goal is (A) pledges are not all the same amount and (B) we have payouts before reaching 100% of the goals.

The reason the games behave differently is: In crowd-based mechanism, players can have only binary 0 or 1 direct impact toward reaching the goal. In dollar-based mechanism, players can have a variable impact toward reaching the goal.

For the two games to be the same, we would have to provide both variable dollar pledge-size option and let each pledge choose whether to count as a partial patron or multiple patrons etc.[1]


After our phone chat this evening, we worked to find some falsifiable situation that would show how the two mechanism variations are different games, not just different framings/labels that would affect real people with emotions and imagination etc.

Well, @mray here’s my example to falsify your assertions of utter sameness in the core math:

  1. I actually got out 3 dice and rolled them 24 times to get a pretty random pool of 24 patrons who have different size pledges. It came out to 241 total, so I reduced a random pledge by 1 to get an easier round 240 total, average of 10.

    • We can consider for this game that this is all the money that exists (but patrons get a fresh allotment of it each month). The patrons have no choices. They just do pledge all their money to the project.

    • Here’s the 24 pledge sizes: 13, 12, 10, 10, 10, 8, 7, 11, 7, 12, 10, 6, 13, 10, 9, 9, 8, 12, 9, 9, 14, 16, 9, 8.

  2. In order to compare at all, we set the project goals to 100% of the available crowd and money from this limited universe.

    • Crowd goal is 24 patrons
    • Dollar goal is $240
  3. I rolled a single die to decide how many patrons would join before each of a series of payout points.[2]

The following charts do drop some partial cents, but the rounding issues don’t matter

1st payout: initial 2 patrons

higher than $10 average, so dollar goal gives higher output

Pledges crowd-goal % crowd-goal donations dollar-goal % dollar-goal donations
$13 1/12 $1.08 25/240 $1.35
$12 1/12 $1 25/240 $1.25
crowd-goal total: $2.08 dollar-goal total: $2.60

2nd payout: up to 7 patrons

new pledges bring average back to $10, so outputs are the same

Pledges crowd-goal progress crowd-goal donations dollar-goal progress dollar-goal donations
$13 7/24 $3.79 70/240 = 7/24 $3.79
$12 7/24 $3.50 70/240 = 7/24 $3.50
$10 7/24 $2.92 70/240 = 7/24 $2.92
$10 7/24 $2.92 70/240 = 7/24 $2.92
$10 7/24 $2.92 70/240 = 7/24 $2.92
$8 7/24 $2.33 70/240 = 7/24 $2.33
$7 7/24 $2.04 70/240 = 7/24 $2.04
crowd-goal total: $20.42 dollar-goal total: $20.42

3rd payout: up to 11 patrons

continues the $10 average, so still equal

Pledges crowd-goal progress crowd-goal donations dollar-goal progress dollar-goal donations
$13 11/24 $5.96 110/240 = 11/24 $5.96
$12 11/24 $5.50 110/240 = 11/24 $5.50
$10 11/24 $4.58 110/240 = 11/24 $4.58
$10 11/24 $4.58 110/240 = 11/24 $4.58
$10 11/24 $4.58 110/240 = 11/24 $4.58
$8 11/24 $3.67 110/240 = 11/24 $3.67
$7 11/24 $3.21 110/240 = 11/24 $3.21
$11 11/24 $5.04 110/240 = 11/24 $5.04
$7 11/24 $3.21 110/240 = 11/24 $3.21
$12 11/24 $5.50 110/240 = 11/24 $5.50
$10 11/24 $4.58 110/240 = 11/24 $4.58
crowd-goal total: $50.42 dollar-goal total: $50.42

4th payout: up to 14 patrons

average drops below $10, so the crowd goal gets slightly more money

Pledges crowd-goal progress crowd-goal donations dollar-goal progress dollar-goal donations
$13 14/24 $7.58 139/240 $7.53
$12 14/24 $7.00 139/240 $6.95
$10 14/24 $5.83 139/240 $5.79
$10 14/24 $5.83 139/240 $5.79
$10 14/24 $5.83 139/240 $5.79
$8 14/24 $4.67 139/240 $4.63
$7 14/24 $4.08 139/240 $4.05
$11 14/24 $6.42 139/240 $6.37
$7 14/24 $4.08 139/240 $4.05
$12 14/24 $7.00 139/240 $6.95
$10 14/24 $5.83 139/240 $5.79
$6 14/24 $3.50 139/240 $3.48
$13 14/24 $7.58 139/240 $7.53
$10 14/24 $5.83 139/240 $5.79
crowd-goal total: $81.08 dollar-goal total: $80.50

5th payout: up to 20 patrons

The average goes down further below $10, resulting in a greater reduction to the dollar-based total compared to the crowd-based total.

Pledges crowd-goal progress crowd-goal donations dollar-goal progress dollar-goal donations
$13 20/24 $10.83 195/240 $10.56
$12 20/24 $10.00 195/240 $9.75
$10 20/24 $8.33 195/240 $8.13
$10 20/24 $8.33 195/240 $8.13
$10 20/24 $8.33 195/240 $8.13
$8 20/24 $6.67 195/240 $6.50
$7 20/24 $5.83 195/240 $5.69
$11 20/24 $9.17 195/240 $8.94
$7 20/24 $5.83 195/240 $5.69
$12 20/24 $10.00 195/240 $9.75
$10 20/24 $8.33 195/240 $8.13
$6 20/24 $5.00 195/240 $4.88
$13 20/24 $10.83 195/240 $10.56
$10 20/24 $8.33 195/240 $8.13
$9 20/24 $7.50 195/240 $7.31
$9 20/24 $7.50 195/240 $7.31
$8 20/24 $6.67 195/240 $6.50
$12 20/24 $10.00 195/240 $9.75
$9 20/24 $7.50 195/240 $7.31
$9 20/24 $7.50 195/240 $7.31
crowd-goal total: $162.50 dollar-goal total: $158.44

6th payout: the whole crowd has now joined

I don’t need a chart here, both goals are reached, all patrons give their full pledges for $240 total.

The two highest pledgers in the whole list at $14 and $16 come in here to raise the average back to $10 after a couple rounds of being below $10.

Totals over the whole time

  • Crowd-based goal: $556.50
  • Dollar-based goal: $552.38

Because it randomly happened that more of the payouts had crowds below the total crowd average, the dollar-based mechanism produced slightly reduced total income.

If it worked out the other way, that more of the payout periods were higher than average (i.e. if the most above-average patrons happened to pledge early instead of late), then the difference would be reversed.

Conclusions

These two mechanisms behave differently given identical setup and identical inputs and identical crowds. The only change is which mechanism we use. The only thing necessary to observe a mathematical difference is to have payouts happen before reaching the goal point.

The subtle difference could have been more significant if the variations and clustering of the pledges happened to be more extreme. I used this random dice-roll approach to help illustrate that I was not choosing a particular contrived example just to illustrate some contrived point.

We cannot conclude that one mechanism or the other will increase the total funding. That depends on the particular pattern of pledges over time. If I ran this random-dice experiment a thousand times, the difference would average out so that both mechanisms would get the same totals over all thousand trials combined. But the fact that the mechanisms behave differently is enough to recognize that the behavior can influence patron motivations toward higher or lower pledges without any appeal to framing effects.

How to study the difference via game theory modeling

Once we recognize that the two mechanisms behave differently, we can ask whether this motivates different patron behavior if we allow it. In other words, if we give the patrons an option: “you may pledge all your money, but you don’t have to”, then if as assign them certain motivations, they can make their choices to optimize their goals.

Here is a game-theory instance (far from the only instance possible) which would be analogous to the sorts of game-theory setups used to study prisoner’s dilemma and snowdrift dilemma games:

  • Players are specified as having a goal of maximizing funds for the project at the lowest cost to themselves
    • We could specify how many points they get for money to the project and how many points they lose for money they spend

Given such defined players, they do indeed have a mathematically factual (not emotional, this could be computer programs, and this is completely unrelated to the meaning of any labels) strategy of minimizing their pledge in crowd-based goals and maximizing their pledge in dollar-based goals.

The strategies for other players would of course be different. Some other games we could set up:

  • The player’s goal is to maximize the crowd size
  • The player’s goal is to freeride (such players will never pledge ever)
  • The player’s goal is to get as much money as possible to the project and has no concern for personal cost (such players will always pledge all their possible money)

The key distinction I made before still holds. If the players know that the crowd will hit the goal no matter what, then their behavior will be the same for either crowd-goal or dollar-goal games. If they do not know when or if the goal will ever be hit and can assume payouts will happen along the way, then their calculations will be different for crowd-goal and dollar-goal games because the games are not the same mathematically.

None of the differences are conclusive in terms of which model is better, primarily because that is a judgment about our mission (do we want larger crowds or more dollars?) and because the behavior of real people and real projects in how they set goals and make pledges will probably be the far more decisive difference in practice.

But I hope we can finally drop the argument about whether the models are exactly the same and it’s only framing. They are not the same. We can accept that and still recognize that the way forward is based on the practical matter of real projects and people and indeed about how we frame things. The mathematical differences aren’t enough to use directly to favor one or the other mechanism.


  1. To keep the two games distinct but show that it's not about the labels, we could swap the labels. We'd limit patrons to a fixed dollar amount but let them choose whether to be counted as donating on behalf of one project user or call themselves a partial user or say they are donating on behalf of multiple users or any portion of users. ↩︎

  2. Obviously, there's no distinction between the systems if they both just jump to 100%, my whole point is that they behave differently on the way to 100% ↩︎

1 Appreciation

Oh and:

Well, the money that all the patrons have in this case is $240 every single payout period (so that’s $1,440 the project could have gotten if everyone pledged immediately and all 6 payouts were at 100%).

The money that the project doesn’t get just disappears from the game. It goes unused. All the patrons who have not pledged yet at a payout time, their money isn’t spent that period. And all the pledged patrons spend only a portion of their money (that’s the whole definition of the mechanism).

The only situation where all the money is spent is when you get to the goal point.

1 Appreciation

While starting to read up on the post I stumbled upon relevant questions I’d rather ask in person.

E.g.

  • what about that “partial patron” concept? I have no idea where that comes from or why.
  • you seem to set a crowd goal at 24 equal to a dollar goal of $240 – why?

I think it makes little sense to process all this information in bulk when calarification is needed, maybe we can get in touch an you fill in the details?

I was referring to money that gets spent in one scenario – but not the other.

oh and that was just the start… there is more that I would need clarification on.

If patrons pledge all their money, should this include the money they received in previous months but didn’t pay? E.g. the $13 patron pays

  • $1.08 in 1st payout, i.e. $11.92 is left, i.e. should plege $24.92 in 2nd payout (crowd-goal)
  • $1.35 in 1st payout, i.e. $11.65 is left, i.e. should plege $24.65 in 2nd payout (dollar-goal)

Is that making sense?

1 Appreciation

Because I picked a random crowd size that was round-enough number, rolled dice to pick how much money each person in the crowd would have each month, and it randomly came to $241 total, so I took $1 away from one patron just to keep the math easy.

I didn’t otherwise choose anything, it’s just a random number input. Turns out that 3 dice tend to average the number 10, though there’s some spread around something like a bell curve.

1 Appreciation

Well, this would be a different game in a sense. A different set of assumptions. The version you describe involves patrons changing their pledges every payout period, fussing with their pledges to adjust based on what happened last payment period. That’s not what we are proposing for crowdmatching or what would be realistic.

In the real world, patrons who do not pay all their pledge to a project during a month will just keep that money for other things in their life. They will not grow and grow and grow their pledges month by month just to counteract the crowdmatching mechanism.

For reductio ad absurdem: If I pledge $1,000/month for a goal of a million patrons but right now the crowd is under 1,000 patrons, I wouldn’t just up my pledge by nearly $1,000 every month that goes by until I’m at $40,000 pledge or something over the years. Anyone doing this completely misses the concept of the pledge in the first place and just is wanting to donate unilaterally without doing crowdmatching.

2 Appreciations

whoever reading this:

@mray and I had a good voice call a couple days ago. We got far enough to conclude that this is not a case of believing fundamentally incompatible ideas. Instead, all or nearly-all of the confusion is around communication challenges. I take my 100% responsibility for the situation by recognizing that I could ask more questions and especially keeping in mind the big picture and avoiding getting stuck on specific sentences or words.

As an immediate next-step, I plan to work out some updated ways to get unstuck and to build better communication patterns. I’m otherwise happy to feel reassured that we are mostly all seeing the same things and have aligned views overall.

Maybe this extra meta framing was unnecessary, but it helped really identify some of the communication issues, so we just need to learn the lessons this situation is here to teach us.

4 Appreciations