A "valid comparison"

Feedback & Support Forum meta etc
#1

Continuing the discussion from Patron based proposal for mechanism 1.1 (instead of $-based goals):

I think @mray’s use of the word “valid” may be core to some of the disagreement here. Here are some related questions:

  • Is the comparison fair?
  • Should we consider the comparison (at all)?
  • How much weight should we give to the comparison?

I think we may not agree on what “the comparison is (not) valid” means with respect to those questions (and probably a few others I missed).

1 Appreciation
#2

I’m just saying a “valid” comparison won’t assume different ideas of what counts as a “successful goal” – or what is “100%”.
I believe there is an agreement that comparing a mechanism that sets lower goals can’t be compared against a mechanism that sets higher goals.

  1. It’s never 100% fair as there is no 100% objective 1:1 relation between the ways of expressing success. With sensible assumptions I think it can be fair to compare both, though.
  2. my whole point is we should not consider it, as to the degree that it matters – there is nothing to compare about, both mechanisms mechanically create the same (or almost identical) results.
  3. None, as said before we are talking about mechanisms that (to the degree that it matters) produce identical results. We should put weight on the framing that comes with either version, though.
#3

I am not so sure. My answers in this scenario assuming that we don’t make any other changes in addition to raising the goal.

Is the comparison fair?

No. Because we would probably make other changes in addition to raising the goal. So, whether the comparison looks favorable depends on where we freeze the other variables.

Should we consider the comparison (at all)?

Yes! It is useful to notice, “When we pick these choices for the other variables, it makes the comparison favorable to lower goals; when we pick these other choices, it makes the comparison favorable to higher goals.”

How much weight should we give to the comparison?

It depends on what we’ve already decided.

Made-up scenario to make this more concrete

Imagine/pretend that lower goals make more sense with patron-goals, and higher makes more sense with dollar-goals. Additionally, imagine that we reach alignment that, without considering other factors, we prefer patron-goals and separately prefer high-goals.

  • We could pick patron/dollar goals first, without thinking about high/low, which we would choose after.
  • We could pick high/low goals first, without considering patron/dollar, which we would choose after.

I think either of those is a mistake. In order to make progress, we probably need to decide on something first, but we should make the first decision while considering which of the second options works better with it, based on which of the second options we like more.

2 Appreciations
#4

This hits at the heart of the problem: changing the dollar goal may seem as one single change. But given that you change a dollar goal begs the question: what patron goal change relates to that? The dilemma is that there is no objective way to find the “right” amount of patrons to any dollar goal-change. We just have to assume it would be one that does not have an impact on the final result.

1 Appreciation
#5

I can’t tell if we agree or disagree. I will try restating my opinion in different words. I think:

  • We should decide whether we prefer crowd vs dollar goals using only fair comparisons.

  • We should choose whether to use crowd or dollar goals based on all of our preferences (crowd vs dollar goals, high vs low goals, etc), including which combinations of choices work best together.

    • We might decide that we prefer crowd goals in isolation, but that we will use dollar goals, because they work better with other options that we prefer (high vs low goals).
2 Appreciations
#6

For example, say I am living in Hamburg, and I want to come visit you in Freiburg. I need to decide: should I drive by car or take the train?

The fairest comparison would be: Imagine there is a direct road / train tracks side by side, directly between our houses. A taxi and a train are leaving at the exact same time, moving the same speed, and cost the same. I am the only passenger. Which would I prefer? The train, no question. It has more leg room, maybe a table, and I can stand up to stretch.

Of course, this is not what I’m actually comparing. Neither takes a direct straight line; one route may be more scenic. The train leaves from the station while the car leaves from my house. The train runs on a schedule and makes stops, while I can take the car any time and drive directly — but the car may get stuck in traffic and requires me to drive (I cannot sleep on the way). The train is more environmentally friendly… as long as enough people are riding it. I can go on, but you get the point.

It is fine to reject all those criteria when comparing the framing of the vehicle I’m traveling in. Yes, the train has a nicer frame :stuck_out_tongue:. But when I actually decide which one I will take, I should consider the other differences, even though they are not a fair comparison.

2 Appreciations
#7

I guess we agree.

  • One reservation is that I don’t expect that we will find either one “working” better (as of their similarity) but that they might just be a better fit for framing reasons. Other than that cautious interpretation of “working” I think we agree.

  • In regards to “fair” my other reservation would be to stress: crowdmatching scenarios that yield significantly different outcomes despite the only change being “patron count” -or- “donations total” can’t be regarded fair.

Given those two caveats I fully support your opinion.

2 Appreciations
#8

I guess in your analogy the question would be in which train waggon you decide to ride. You may have different classes and different qualities in different cabins, but in terms of which arrives faster – there is no question. You ride the same train with all benefits and downsides that a train may have :stuck_out_tongue_winking_eye:

1 Appreciation