Open and Sustainable Innovation Systems (OASIS) Lab working notes

Powered by 🌱Roam Garden


This is an "Orphan" page. Its core content has not been shared: what you see below is a loose collection of pages and page snippets that mention this page, as well as snippets of this page that were quoted elsewhere.

Referenced in


> "...if one is to stand on the shoulders of giants, one must first climb up their backs, and the greater the body of knowledge, the harder this climb becomes." (p.284) synthesis

June 29th, 2020

As a byproduct, we'll get to understand via infrastructural inversion the key elements of a synthesis infrastructure (at least at the individual and collaborative level)

Z: Doctoral students struggle to effectively synthesize literature

Faculty have high stated expectations for synthesis, but reviews of dissertation committee comments on doctoral dissertation literature reviews demonstrate that the bar for synthesis is substantially lower in practice than what faculty's stated expectations.


Claim "A quality synthesis will result in a progressive problem shift" (p357)


Learning about a new area of knowledge is challenging for novices partly because they are not yet aware of which topics are most important. The Internet contains a wealth of information for learning the underlying structure of a domain, but relevant sources often have diverse structures and emphases, making it hard to discern what is widely considered essential knowledge vs. what is idiosyncratic. Crowdsourcing offers a potential solution because humans are skilled at evaluating high-level structure, but most crowd micro-tasks provide limited context and time. To address these challenges, we present sys/Crowdlines, a system that uses crowdsourcing to help people synthesize diverse online information. Crowdworkers make connections across sources to produce a rich outline that surfaces diverse perspectives within important topics. We evaluate Crowdlines with two experiments. The first experiment shows that a high context, low structure interface helps crowdworkers perform faster, higher quality synthesis, while the second experiment shows that a tournament-style (parallelized) crowd workflow produces faster, higher quality, more diverse outlines than a linear (serial/iterative) workflow.

Z: Synthesis is creating a new whole out of component parts

synthesis is fundamentally about creating a new whole out components (strike1983types). This means that synthesizers need to be able to compose individual ideas (evidence, hypotheses, concepts, claims) into a larger conceptually integrated understanding, such as a theory or argument.

C: Compression

Atomicity is a key precondition for synthesis.


a bit of a chicken-and-egg here maybe.. if Z: Effective synthesis is hard for everyone, then reviewers may not have adequate expertise to judge whether a synthesis is adequate. Probably doesn't change the numbers here that much though. and also remember that these aren't necessarily interdisciplinarity papers.

October 26th, 2020

Crowdsourcing offers a powerful new paradigm for online work. However, real world tasks are often interdependent, requiring a big picture view of the difference pieces involved. Existing crowdsourcing approaches that support such tasks -- ranging from Wikipedia to flash teams -- are bottlenecked by relying on a small number of individuals to maintain the big picture. In this paper, we explore the idea that a computational system can scaffold an emerging interdependent, big picture view entirely through the small contributions of individuals, each of whom sees only a part of the whole. To investigate the viability, strengths, and weaknesses of this approach we instantiate the idea in a prototype system for accomplishing distributed information synthesis and evaluate its output across a variety of topics. We also contribute a set of design patterns that may be informative for other systems aimed at supporting big picture thinking in small pieces.

Z: Most scholarly communication infrastructure operates on the document as the base unit

@harsDesigningScientificKnowledge2001 articulates some good {{alias: example-of examples of}} the kinds of queries that are both crucial for synthesis and functionally impossible to answer in our current scholarly communication infrastructure, such as V7DA5A8_-

Z: Ideas develop simultaneously at multiple timescales, levels of granularity, and completeness

This is another hidden reason why incremental formalization is powerful for creative synthesis in particular, and may not become as important for, say, a systematic review or meta-analysis: need to bridge and connect the multiple overlapping layers of ideas.


Claim Many dissertations are deemed acceptable by examiners despite comments that clearly indicate an unsatisfactory level of synthesis, by scholarly standards (p. 1028-1032)


Claim "A successful synthesis will satisfy the formal criteria for good theories" (p357)


Claim "A quality synthesis will clarify and resolve, rather than obscure inconsistencies or tensions between material synthesized" (p357)


Scholars varied substantially in the ecology of tools used for synthesis: no single tool was used to encompass every (sub)task, and few tools were common across all scholars.


Claim Only about 20% of published systematic reviews in orthondotics are "good" by AMSTAR standards of review quality; 20% are considered "poor"! (Table 2, p.246) synthesis


systematic review teams very frequently (on the order of 70% of the time) need to contact authors for additional details (context) for reported findings synthesis

January 17th, 2021

good article on uncertainty, and how sometimes we can't run RCTs, so need to rely on synthesis and nuanced undersatnding of uncertainty


Claim Compiling a concept matrix while reading can help promote synthesis instead of mere summarization of the literature (p. xvii)

November 22nd, 2020

Scott Kahan connected with me on Twitter via shared interest in sys/RoamResearch, directs National Center for Weight and Wellness, possible connection to Chinarut's interests in synthesis for wellbeing?

Z: Science is getting less bang for its buck

Our synthesis apparatus was probably ok for a world in which there was less to synthesize, and possibly less Scatter. But the world has changed (e.g., interdisciplinarity is way more of a thing, there is way more to synthesize now, way more Scatter), and our synthesis apparatus has hardly changed at all. So it's quite plausible that our effective synthesis rate would fall quite a bit behind where we need it to be.

July 28th, 2020

Conversely, we see the infrastructure that people do rely on (e.g., Google Scholar, Web of Science, and so on) consistently breaking down and thereby becoming visible when people try to use it for difficult synthesis tasks, especially across disciplines. They also often cannot really transfer their bricolage solutions from previous tasks or projects to these new domains they have to navigate.

Z: How can we support explicit contention with evidence when synthesizing knowledge claims?

Reasoning and synthesis at any reasonable scale benefits from some kind of computational and/or quantitative aggregation.


Claim Some types of synthesis are simpler to produce than others (p. 355-356)


Cited by @watsonBeingSystematicLiterature2015 to suggest a concept-centric reading/organization strategy to promote synthesis over mere summarization


strong example-of synthesis - proposes a new "agenda seeding" model to account for conflicting theoretical accounts (elite vs. pluralistic models) of the efficacy of different modes of protest for subordinate minorities

Z: Compression is necessary for synthesis

We might wonder: if we break complex documents down in a synthesis infrastructure, what should the component parts look like? What defines an "idea" level, or an appropriately "small" building block for scholarly synthesis?

Z: Compression is necessary for synthesis

Z: Synthesis is creating a new whole out of component parts. But what should the "parts" look like? What kinds of building blocks would facilitate synthesis?

April 30th, 2020

There is also the related but distinct sense of the level (or lack thereof) of synthesis in the ordinary course of research papers being written. This is a bit closer to what is analyzed with the @holbrookLevelsSuccessUse2008 set of studies. There is still that question of whether the lack of visible synthesis belies a lack of "real" synthesis under the hood. Even if there is a difference, the effect on the community might be the same, especially if there is a dearth of effective review papers.

November 22nd, 2020

Several pieces and talks by Berna Devezer about the insufficiency of reproducibility for advancing discovery, also making the case for formal methods (possibly synthesis) - reminded me of the argument that Z: Scientific fields stall without adequate theoretical synthesis, even if reproducibility is maintained. Possible connections to P/Synthesis Infrastructure?

Z: Synthesis is-necessary-for effective problem formulation

In the majority of cases, an effective synthesis of the problem domain is what enables a good problem formulation. Without such a synthesis, creators cannot adequately transform their Ill-structured problem into a tractable one they can make progress on.


Not quite synthesis, but seems like a place where synthesis shoud be happening and helping (p. 30-31)

Z: How can we support explicit contention with evidence when synthesizing knowledge claims?

In @clarkMicropublicationsSemanticModel2014, there is a general sense of support, with the lowest being some kind of authorship attribution. But I'm pretty sure the support/challenge relationship is binary, in the formal sense. I'm not quite sure then how this is translated into computational reasoning. Maybe it's implicit here, so we explicitly have to reason about the support *types* (has attribution only, has data and methods). THis might lead to quite different dynamics, although maybe slower to start with, might lead to deeper synthesis than including a belief number??

April 21st, 2020

Related: a lot of problem formulation (and synthesis) relies on appraisal ; often this isn't really a Claim that authors themselves are making. But it still needs to be grounded in some way. This is critical grist for

Z: Synthesis is a creative act

The core idea of synthesis is that it creates a new whole out of parts, such that the whole has novel properties that are distinct from the sum of its parts.


I wonder how easy it is to transition from the relatively playful association-style structure to more powerful structured representations for synthesis like causal models (cf. Judea Pearl and Bayesian Networks)


systematic reviews can take 5-6 people more than 1000 hours without special tech (cited in @ervinMotivatingAuthorsUpdate2008) synthesis


is an example-of how context is critical for synthesis (in this case, showing the kinds of context queries scientists look for to try to do synthesis over contradictory findings in a systematic review


The dual goals of efficiency and effectiveness when writing a literature review are considered. Effectiveness is concerned with producing a synthesis of the published knowledge. A systematic approach to reviewing is at best a partial approach to efficiency because the foundations of the academic publishing system are rigidly locked into old technology. An outline for redesigning academic publishing to jump literature reviewing efficiency to a new level and enhance the productivity of many other aspects of scholarship is proposed.

April 20th, 2020

New stuff following up on R: zhangpengyiComprehensiveModelCognitive2014 thread with a really nice focus on cognitive mechanisms, probably SOTA rn on models of sensemaking - still at different level of analysis (much more granular and much less object-focused, and for quite different tasks than scholarly synthesis)


> The Data Extraction Process seemed particularly demanding to those involved in the corresponding tasks. The Coordinating Editor (P1) described systematic reviewers as being "enslaved to the trapped data", with reviewers "chiseling the mine" to get at the data they needed. (p.208) synthesis

WP: JCDL Where the semantic publishing rubber meets the scholarly practice road

sys/GlamorousToolkit - not purpose-built for synthesis, but very easy to do it in here, with huge advantages for Multiplicity due to its superior programmability


synthesis is notably lacking in the model, but can be sort of mapped to the Assembling sub-primitive of Writing

January 24th, 2021

Prices in markets as boundary objects and synthesis, aggregating only the necessary information that is distributed across actors, and made available to individual actors to make (approximately) rational economic choices


appraisal Has some really nice example-of the outputs of synthesis, but I wish there was more about the process! HOW do you get to those outputs? Think harder? Work longer?

Z: Synthesis is creating a new whole out of component parts

In this vein, systematic reviews can be construed as a narrow subset of synthesis (and literature reviewing)

Z: Science is getting less bang for its buck

It's hard to study the level of synthesis directly (and in particular draw a causal link between that and "progress", which is in itself really tricky to measure, since we rely a lot on scientometrics and bibliometrics, and Z: Citation practices in science are far from optimal), but we have lots of anecdotal evidence at least.


Ineffective synthesis is not a reason to fail a dissertation! Qualities associated with literature reviews in "acceptable" (or even "very good") dissertations fall well short of implicit and explicit criteria for effective synthesis (e.g., @strike1983types, @booteScholarsResearchersCentrality2005


Claim A useful synthesis helps to judge how "healthy" a research program is, and suggests future directions for it, often by interrogating/assessing crucial assumptions (p. 358)