Faculty have high stated expectations for synthesis, but reviews of dissertation committee comments on doctoral dissertation literature reviews demonstrate that the bar for synthesis is substantially lower in practice than what faculty's stated expectations.
Learning about a new area of knowledge is challenging for novices partly because they are not yet aware of which topics are most important. The Internet contains a wealth of information for learning the underlying structure of a domain, but relevant sources often have diverse structures and emphases, making it hard to discern what is widely considered essential knowledge vs. what is idiosyncratic. Crowdsourcing offers a potential solution because humans are skilled at evaluating high-level structure, but most crowd micro-tasks provide limited context and time. To address these challenges, we present sys/Crowdlines, a system that uses crowdsourcing to help people synthesize diverse online information. Crowdworkers make connections across sources to produce a rich outline that surfaces diverse perspectives within important topics. We evaluate Crowdlines with two experiments. The first experiment shows that a high context, low structure interface helps crowdworkers perform faster, higher quality synthesis, while the second experiment shows that a tournament-style (parallelized) crowd workflow produces faster, higher quality, more diverse outlines than a linear (serial/iterative) workflow.
a bit of a chicken-and-egg here maybe.. if Z: Effective synthesis is hard for everyone, then reviewers may not have adequate expertise to judge whether a synthesis is adequate. Probably doesn't change the numbers here that much though. and also remember that these aren't necessarily interdisciplinarity papers.
Crowdsourcing offers a powerful new paradigm for online work. However, real world tasks are often interdependent, requiring a big picture view of the difference pieces involved. Existing crowdsourcing approaches that support such tasks -- ranging from Wikipedia to flash teams -- are bottlenecked by relying on a small number of individuals to maintain the big picture. In this paper, we explore the idea that a computational system can scaffold an emerging interdependent, big picture view entirely through the small contributions of individuals, each of whom sees only a part of the whole. To investigate the viability, strengths, and weaknesses of this approach we instantiate the idea in a prototype system for accomplishing distributed information synthesis and evaluate its output across a variety of topics. We also contribute a set of design patterns that may be informative for other systems aimed at supporting big picture thinking in small pieces.
Our synthesis apparatus was probably ok for a world in which there was less to synthesize, and possibly less Scatter. But the world has changed (e.g., interdisciplinarity is way more of a thing, there is way more to synthesize now, way more Scatter), and our synthesis apparatus has hardly changed at all. So it's quite plausible that our effective synthesis rate would fall quite a bit behind where we need it to be.
ON the flip side, a more qualitative approach (e.g., evidence lines), as exemplified by @clarkMicropublicationsSemanticModel2014 and @brushSEPIOSemanticModel2016, is less readily quantifiable in a sophisticated sense, but possibly more useful for the early stages of synthesis, particularly where there is interdisciplinarity
Conversely, we see the infrastructure that people do rely on (e.g., Google Scholar, Web of Science, and so on) consistently breaking down and thereby becoming visible when people try to use it for difficult synthesis tasks, especially across disciplines. They also often cannot really transfer their bricolage solutions from previous tasks or projects to these new domains they have to navigate.
There is also the related but distinct sense of the level (or lack thereof) of synthesis in the ordinary course of research papers being written. This is a bit closer to what is analyzed with the @holbrookLevelsSuccessUse2008 set of studies. There is still that question of whether the lack of visible synthesis belies a lack of "real" synthesis under the hood. Even if there is a difference, the effect on the community might be the same, especially if there is a dearth of effective review papers.
Several pieces and talks by Berna Devezer about the insufficiency of reproducibility for advancing discovery, also making the case for formal methods (possibly synthesis) - reminded me of the argument that Z: Scientific fields stall without adequate theoretical synthesis, even if reproducibility is maintained. Possible connections to P/Synthesis Infrastructure?
In @clarkMicropublicationsSemanticModel2014, there is a general sense of support, with the lowest being some kind of authorship attribution. But I'm pretty sure the support/challenge relationship is binary, in the formal sense. I'm not quite sure then how this is translated into computational reasoning. Maybe it's implicit here, so we explicitly have to reason about the support *types* (has attribution only, has data and methods). THis might lead to quite different dynamics, although maybe slower to start with, might lead to deeper synthesis than including a belief number??
Intuitively, having building blocks that are too "large" or complex would make reassembly into a new whole impractical or impossible. Thus, the most basic requirement for synthesis is having appropriate(ly sized) building blocks to start with.
The dual goals of efﬁciency and effectiveness when writing a literature review are considered. Effectiveness is concerned with producing a synthesis of the published knowledge. A systematic approach to reviewing is at best a partial approach to efﬁciency because the foundations of the academic publishing system are rigidly locked into old technology. An outline for redesigning academic publishing to jump literature reviewing efﬁciency to a new level and enhance the productivity of many other aspects of scholarship is proposed.
New stuff following up on R: zhangpengyiComprehensiveModelCognitive2014 thread with a really nice focus on cognitive mechanisms, probably SOTA rn on models of sensemaking - still at different level of analysis (much more granular and much less object-focused, and for quite different tasks than scholarly synthesis)
Compressing these debates into a single number is technically and mechanically valuable, needed even, for synthesis at realistic scale. But you run a high risk of short-circuiting the needed reasoning to make it happen in a trustworthy manner.
We need to consider first principles — transmission properties of the disease, controlled biophysical characterizations alongside observational data, partially informative RCTs (primarily with respect to PPE), natural experiments (28), and policy implementation considerations — a discursive synthesis of interdisciplinary lines of evidence which are disparate by necessity (9, 29). (p. 3)
It's hard to study the level of synthesis directly (and in particular draw a causal link between that and "progress", which is in itself really tricky to measure, since we rely a lot on scientometrics and bibliometrics, and Z: Citation practices in science are far from optimal), but we have lots of anecdotal evidence at least.