What is objective or measurable?

Feedback & Support
,
1

Continuing the discussion from Patron based proposal for mechanism 1.1 (instead of $-based goals):

We should really clean up this language that’s leading to misunderstanding.

The mechanism math cannot determine good or bad

Purely within the math of two mechanisms, we can acknowledge differences such as:

  • In dollar-goal: increasing the average pledge size results in a smaller crowd reaching the goal.
  • In crowd-goal: increasing the average pledge size results in the goal point having larger funding from the same-size crowd

Or this distinction:[1]

  • in dollar-goal, increasing one patron’s pledge increases both the mean and the median donation
  • in crowd-goal, increasing one patron’s pledge increases only the mean and not the median donation

(The latter is another way of pointing out the many-times-emphasized idea about “extra matching”)

But we cannot say objectively within the pure math that this difference is good or bad or that it will or won’t result in larger or smaller crowds or more or less funding.

But we could do empirical research

However, we can in principle make further objective measurements of the differences between crowdmatching mechanisms. But that would require empirical research.

It’s definitely possible to run controlled experiments with real people and even real projects and observe objectively that one mechanism or another has a statistically significant difference in either or both crowd size and funding totals.

Put another way: it’s likely true that one mechanism would in fact result in more project funding than the other. But measuring to discover which one is far from trivial.

If we ran experiments, how would we know if the results are representative for all sorts of projects and crowds? Maybe GNU-type free software projects will behave one way while journalism projects will behave a different way. There’s so many variables. And even if there is a real difference, we might fail to capture it in our measurements.

I don’t think we should discard the idea of such research. Maybe it will be extremely stark. One mechanism could utterly fail compared to the other. We should make plans to at least try to get low-cost feedback and simple research to see how people respond. Maybe projects will join us with dollar-goal and refuse to participate with crowd-goal. Maybe one or the other will be 100 times more successful at patrons just “getting it” after the elevator pitch. To find out these potential differences, we have to do the empirical observations.

Such studies can be objective and measurable. So, we shouldn’t say that neither mechanism has an edge in measurable objective benefits. We just don’t yet have any evidence to make that conclusion.

  1. this ignores the tiny edge case of the single pledge switching which side of the median it is on ↩︎

3 Appreciations
2

One question is: “can we afford to let this hold us back?”

I want to see this project having a positive impact soon. I’m waiting for years.

We discuss this topic now for 3 weeks (?) and i haven’t seen much new information. It’s more like trying to explain the same thing over and over again. I still have the same opinion as before. It’s even stronger since i can argue better for it. For me, it’s 100% clear what the right decision is. I’m tired of repeating myself.

To finish this topic, we could just do a democratic vote and let the majority decide. But i would be very unhappy if my opinion don’t win, because i fear that means less success for the project. Maybe the other side thinks the same, so a consensus would be better. But i don’t see how we can come to a point where we agree on how to proceed.

With more effort, we can actually do a (small) study. We can ask scientists for help. Maybe the people involved in Talk: "Analyzing Tens of Terabytes of Public Trace Data & Open Source Sustainabil" would like to work with us on it.

We should decide how much time we want to invest here.

1 Appreciation
3

This topic is significant enough that we want to try for alignment or at least consent. Anyone feeling just overruled will be an unfortunate outcome.

Relevant:

We need to define the process and have a fallback option (we can’t be absolutely stuck and unable to ever make progress if alignment is impossible). We could define an arbitrary timeline perhaps.

We should not go for the fallback without the attempt at stronger alignment first (though we’ve certainly already put a lot into this).

1 Appreciation
4

One new thing that could happen is that I could catch up on the conversation and try to mediate. If the discussion dies out, then there is an additional cost to starting up again. So, it might be better for me not to wait much longer, even though I still think that work on the bylaws & website is a higher priority.

1 Appreciation
5

For me this (unfortunately very long and exhausting) discussion isn’t just about progress in decision-making anymore, it started to become a question of how well we understand the mechanism and how far we can go when framing it, too.

I think it is dangerous to make a decision on how we frame our mechanism in our communication while having a less then optimal understanding of its very nature. In my eyes that’s extremely important as skeptical minds will sooner or later check to see if what we say is waterproof (rightly so!).

We have to avoid to applying framing that might work well, – or even “better” than another framing – but that is based on a train of though that is not really tied to the mechanism as we assume it is. When we frame and “sell” our mechanism, we need to not make wrong assumptions and sell them as a great effect. Neither to ourselves now, in a debate, nor to our community later.

As of the actual debate, I completely agree with wolftune that at this point we just don’t have the data to come to an “objective” conclusion of which approach is better. All we have got right now is our gut feeling and a number of associated thoughts that make more or less sense. That’s not the best of starting points for a productive discussions, and I don’t expect it to be easy, but assumptions based on math that favor one mechanism over the other should remain a no-go.

1 Appreciation
6

There are some pretty pure-math ways to study this in terms of game theory.

We understand the rules each system defines. We understand that we are talking about iterative games with no end (monthly forever) — that detail creates fundamentally different game play as well-studied in prisoner’s-dilemma and other game-theory.

I think there’s room to study the games further separately from the empirical research.

We should lay it out clearly with all the parameters. We should recognize the players are different patrons and the projects. We can start with one-off (non-iterative) 2-player games (say one patron and one project, or with fixed project and 2 patrons). What are the goals of each player? We can all put aside the knowledge that the real thing will be iterative and have huge crowds and competition for patrons among projects. If we can all understand the simplest version of the games, we can work toward the more complex versions.

With understanding of the game theory, we could settle on a list of hypotheses that we want to test by presenting the games to real projects and patrons (it could start with just explaining it and getting impressions, but we could run fake-money tests too, and we could even potentially plan real A/B tests of actual designs if we can get the resources to do that). I’d like to convince some behavioral economists to work with us and study these questions if possible (we had one game-theory focused professor on our early steering committee incidentally). But we need clear game definitions and clear hypotheses to test in order to know how to go forward on empirical observations.

2 Appreciations
7

Getting insight from people that know the math and the models behind this would be awesome, indeed! It would not surprise me, though, if the model would consist of a third player being the crowd. At some point it probably stops making sense to model after each unique patron…

1 Appreciation
8

Yes, of course. I was suggesting 2-player as just the starting point. It’s as invalid for our actual operations as the non-iterative scenario. But we should know “this is the effect of adding a 3rd player” just as “this is the effect of making it iterative”.

While getting experienced help would be ideal, it should be doable ourselves. We just need to specify each player’s goal(s) and assign some points and then play out some actual scenarios, model it etc.

1 Appreciation