nips nips2011 nips2011-3 nips2011-3-reference knowledge-graph by maker-knowledge-mining
Source: pdf
Author: Jacob D. Abernethy, Rafael M. Frongillo
Abstract: Machine Learning competitions such as the Netflix Prize have proven reasonably successful as a method of “crowdsourcing” prediction tasks. But these competitions have a number of weaknesses, particularly in the incentive structure they create for the participants. We propose a new approach, called a Crowdsourced Learning Mechanism, in which participants collaboratively “learn” a hypothesis for a given prediction task. The approach draws heavily from the concept of a prediction market, where traders bet on the likelihood of a future event. In our framework, the mechanism continues to publish the current hypothesis, and participants can modify this hypothesis by wagering on an update. The critical incentive property is that a participant will profit an amount that scales according to how much her update improves performance on a released test set. 1
[1] J. Abernethy, Y. Chen, and J. Wortman Vaughan. An optimization-based framework for automated market-making. In Proceedings of the 12th ACM Conference on Electronic Commerce, 2011.
[2] J. E. Berg, R. Forsythe, F. D. Nelson, and T. A. Rietz. Results from a dozen years of election futures markets research. In C. A. Plott and V. Smith, editors, Handbook of Experimental Economic Results. 2001.
[3] Y. Chen and D. M. Pennock. A utility framework for bounded-loss market makers. In Proceedings of the 23rd Conference on Uncertainty in Artificial Intelligence, 2007.
[4] Y. Chen and J. Wortman Vaughan. A new understanding of prediction markets via no-regret learning. In Proceedings of the 11th ACM Conference on Electronic Commerce, 2010.
[5] T.M. Cover, J.A. Thomas, J. Wiley, et al. Elements of information theory, volume 6. Wiley Online Library, 1991.
[6] T. Gneiting and A.E. Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477):359–378, 2007.
[7] R. Hanson. Combinatorial information market design. 5(1):105–119, 2003. Information Systems Frontiers,
[8] R. Hanson. Logarithmic market scoring rules for modular combinatorial information aggregation. Journal of Prediction Markets, 1(1):3–15, 2007.
[9] R. Hanson, R. Oprea, and D. Porter. Information aggregation and manipulation in an experimental market. Journal of Economic Behavior & Organization, 60(4):449–459, 2006.
[10] Nathan Lay and Adrian Barbu. Supervised aggregation of classifiers using artificial prediction markets. In ICML, pages 591–598, 2010.
[11] J. Ledyard, R. Hanson, and T. Ishikida. An experimental test of combinatorial information markets. Journal of Economic Behavior and Organization, 69:182–189, 2009.
[12] J. Wolfers and E. Zitzewitz. Prediction markets. Journal of Economic Perspective, 18(2):107– 126, 2004. 9