You have a room of N would-be decision makers.  Each has a history of making these types of decisions, having recommended a decision M times in the past.  All people in the room have previously opined on the same M decisions and these can be compared to historical results to determine correctness.  Thus each person’s history equates to a number of successes S and a number of failures (M-S).

Each person in the room is secretly either a sage or a fool.  Sages have some accuracy A != 0.5 of making a correct decision.  Accuracies < 0.5 are considered sages despite their apparent inaccuracy because the negation of their decisions are wise.  Fools have exactly P=0.5 stemming from a randomized decision making process.

No information other than what can be deduced from the decision histories is known about the correlation or other relationships between the decision maker’s decisions.   Similarly, there may be a hidden internal structure to the decisions which can only be deduced by observing which sages get which ones correct.  Nothing is known about the respective numbers of sages & fools in the room – only that they sum to N.

Problem:

Describe a method which uses the histories to select a committee – a subset of the N decision makers – and a voting method of combining their decisions to reach a group decision which is optimal based on the available knowledge.  That is, maximize P(correct) for the committee’s vote.  An empty committee represents the assertion that  no committee and voting system is better than random.

Observations:

• Fools in large groups can be surprisingly lucky – even overwhelmingly so.  For example if N is in the millions (say a nation of people) and M is in the <20 range and nearly all decision makers are fools, you would expect numerous lucky fools with perfect track records by pure chance.  In that case it is clearly impossible to separate the fools from even perfect sages with total accuracy, and thus as a practical matter impossible to form a sages-only committee.  Generalized, this means the optimal committee sometimes must contain fools.
• Sages can have interesting relationships.  Consider one “master” sage with  A = 0.9 and another “student” sage who randomly chooses between two behaviors: the student makes the same decision as the master 0.5 of the time, a a random decision the other 0.5.  The student sage would thus have A = 0.7 – much better than a fool.  But any committee which contained the student could be improved by removing the student and increasing the master’s voting weight.  Thus any committee which contains a student is sub-optimal.  Generalized, this means the problem is not merely separating the sages from the fools even when that is possible – the optimal committee may contain only some of the available sages.
• The committee can be wiser than any individual sage.  Consider two classes of sages – ones that always get even numbered decisions correct and guess randomly on odd numbered decisions, and their opposite who get odds correct and guess on evens.  If such sages were added to the committee in pairs with all committee members having equal voting weight, accuracy for the committee would quickly approach 1.0 as pairs were added while A = 0.75 for any individual sage.
• Your method should work such that including duplicates of the same sage (ie. two or more sages that always vote the same on every decision) in the room should not change the composition of the committee other than possibly substituting (partially or wholly) duplicates for the original – nothing that would change the resulting committee decision.  This appears obvious since adding duplicates offers no new capabilities that couldn’t be achieved by adjusting the voting weight of the original.  Thus the new optimal committee can’t be any better or  worse as a result of adding a duplicate sage.