An illustration of alchemist characters and other ancient symbolsContrary to popular understandings, alchemy was not just wacky medieval wizards trying to turn lead into gold or to create a panacea that would allow them to live forever. Instead, alchemy was a widespread and well-respected practice of learning and development, often with very utilitarian goals like better glassmaking, medicinal distillation or metalwork. Centuries of alchemy eventually became the precursor for the science that we know and love today!
Both alchemy and science actively encouraged systematic experimentation, documentation, learning and improvement, but a key difference between the two can be understood as reproducibility. Alchemists usually worked alone for their own benefit or profit. They did document their work, often at great length, but they did so primarily as an aide memoire. Their notes were written in code, with abstract language, extensive metaphors, and frequent reference to subjective features from their own life experiences. This complexity and secrecy ensured that others could not use the information and that they alone (or possible a small number of apprentices) would possess the valuable knowledge and its associated prestige. In effect, alchemy operated under the same principles as ‘secret family recipes’, shared only on the death bed to ensure respect and power within social circles. Alchemy was thus a process of private and personal improvement.
In contrast, scientists liked to discuss the things they had observed or learned and valued the insights gained when others examine their work. Like alchemists, they valued the respect and prestige of having learned things, but they gained status from being known as the first to contribute something new to a shared body of knowledge. As a result, scientists strove to carefully document their observations, ideas and studies in relatively clear language and with detailed measurements so that when they shared their work others would be able to see the value and importance. Scientific discoveries and advances were shared through laboriously produced notes, letters and books, and through invitations to present their results publicly. Thus, science operates under different principles, with respect and power going to those that share the most valuable knowledge as widely as possible. Science therefore became a collective process of development through personal effort.
A detailed methods section or careful observations printed in journals found at any public library are more reproducible than alchemical manuscripts written in code, but researchers today work in very challenging contexts. They cannot expect to work individually or to only spend time on tasks that suit their natural talents, but they do have many more tools to hand. As a result, reproducibility now involves open access to datasets and original code as you might expect, but increasingly also means access to draft papers, meeting notes, decision making processes, and more. Better still, storing all the materials in an online repository that includes detailed history of who made what change and when allows for even more reproducibility through transparency, documented collaboration and clear records.
The future of reproducibility
The future of reproducibility means opening the entire research process to scrutiny, warts and all. Essentially, this is ‘showing your work’ instead of just showing the results, including the ‘whys’ and ‘hows’ of the research process as well as the ‘whats’ and ‘whens’. This transparency means that the research process becomes accessible to others who might be collaborative, critical, indifferent or a combination of all of these. All of this can feel overwhelming, especially when researchers are not given examples to follow or instruction to apply. In practice, researchers may feel exposed to criticism, isolated in a competitive and fast-moving research context, and pressured to document without training on how to do that effectively. Understandably, reproducibility is not always valued in this context, with researchers instead working to perfect their work before sharing it with others or putting their effort into progressing as far as possible rather than into documenting their progress so far.
For that reason, researchers are invited to join a five-week bootcamp on reproducibility. The bootcamp explores the important concepts, including what the benefits are for individual researchers, collaborative groups of researchers and wider research communities. The bootcamp also introduces various tools that researchers can use to improve their documentation, collaboration and transparency, often in ways that are automatic or easy, rather than the manual or painstaking methods that researchers might assume are needed.
Bootcamp participants will also discuss examples of best practice in a supportive environment, so allows them to ask questions, learn methods and habits to build reproducibility into their work, and become more comfortable with feeling personally vulnerable within a research community. This bootcamp invites researchers to let go of the ‘secret recipe’ personal status that typified alchemy, to not be consumed by competitive status of early scientific work in which researchers rushed to be the ‘first to publish’, and instead to embrace the transparency and vulnerability of truly reproducible collaborative work that is so necessary in research today.