Electronic Lab Notebooks#

An Electronic Laboratory Notebook (commonly known as an ELN or a digital lab notebook) is a software system designed to help you document and maintain reproducibility of your research and share information more easily. ELNs enable researchers to organize and store experimental procedures, protocols, plans, notes, data, and even unfiltered interpretations using their computer or mobile device. They can be a digital analogue to the paper notebook most researchers keep. ELNs can offer several advantages over the traditional paper notebook in documenting research during the active phase of a project, including; searchability within and across notebooks, secure storage with multiple redundancies, remote access to notebooks, and the ability to easily share notebooks among team members and collaborators. ELNs make the practice of open notebooks [def] practical in a way that it is not for paper lab books. However, it is important when choosing an ELN solution to avoid giving up the advantages offered by a paper lab notebook.

Quick Pros & Cons of ELNs#

Electronic Lab Notebooks may provide, among other things, the following functionalities:

  • A text editor with similar functions as a paper notebook

  • A search function

  • Secure storage and back up (with distinct failure modes from paper notebooks)

  • Tools for working with tabular data (calculations and formatting of tables and graphs)

  • Templates for documenting standard procedures

  • Built-in Laboratory Inventory Management Systems (LIMS) functionality for managing and documenting samples, reagents, and apparatus, or integration with separate dedicated LIMS.

  • Collaboration tools for sharing experimental information

  • Some ELNs will allow you to comply with standards and regulations because of their certification processes

ELNs also have their limitations:

  • Costs: Most ELN solutions can only be used through paid plans, or the free plans have reduced functionalities in the number of users, storage space or file sizes. Your lab may also not have access to tablets or pens that would make the use of ELNs easier. Check if your institution is offering a solution that you can use without additional costs.

  • File format: Always check if files can be exported in your preferred file formats to avoid format lock-in. Some ELNs also have API’s that allow integration with other software and workflows.

  • Vendor-lock-in: Once you’re using a certain solution you may become dependent on it. Always have an exit strategy in case the solution is no longer offered or if you’re moving institutions and you no longer have access.

  • Sustainability: Choose a solution that has a larger chance of being around for a long time.

  • Drawing: Not all solutions have the tools or capabilities to include drawings or integration with drawing software.

  • Security: Before using an ELN solution you should check any backup plans and data security measures, especially if you’re working with sensitive data.

  • Laboratory situation: Your lab may not have an internet connection or electronic tools could affect samples, reagents or magnets in the lab.

  • Learning curve: ELNs have a longer learning curve than paper notebooks and it takes time to learn how to use them. User-friendliness and flexibility are important to accommodate the widely varying workflows of each lab member.

Choosing an ELN solution#

If you are in a position to choose an ELN to use, or are making a choice on behalf of an organisation there are some factors to consider:

Importance of getting it right#

Many organisations offer software that purports to solve the problem of ELNs. So choosing a suitable solution can be a major headache and responsibility. The choice of ELN is an incredibly important decision, and one that your organisation will likely have to live with for years, or even decades. You are putting the record of your research into the hands of the tool that you choose, and entering into a long-term relationship with the provider of your ELN solution. You are also making a choice about a tool that many of the people in your organisation will interface with closely every day. Having to use a tool that is not well fit to its function on a daily basis can be a major problem for productivity and morale. Conversely, a tool that is a good fit gets out of people’s way and makes their jobs easier can be a major boost to productivity and morale.

Failure modes in user adoption of ELNs#

Incomplete or inflexible ELN solutions which increase friction in workflows can cause issues. People will avoid using ill-fitting tools that they perceive to be getting in their way, especially scientists. They are creative, stubborn, resourceful, and impatiently focused on answering their research questions with whatever tools are available to them. Finding software with the flexibility, capability, and scope to keep up is not easy and if it doesn’t researchers won’t use it - at least not as you intended.

  • Shadow IT/notebooks

    • People using their own unsanctioned solutions which may be a compliance issue and introduce risks that you have not accounted for

  • Infrequent updates

    • People dump a copy of their work into the ELN only as often as necessary for compliance

  • Consistency issues

    • If there are paper and potentially multiple electronic copies of some information - what happens when it does not agree? What is the source of truth?

  • Sprawl

    • Partial adoptions can lead to different pieces of information being stored in different systems which might not be connected making it harder to record and find information which was previously in one place. This can make information recording and retrieval processes more complex, expensive and brittle

  • Practical use and interface issues

    • Laptops or tablets can introduce interface barriers

      • Lab gloves may hamper the use of touchscreens and track pads

      • Free-hand drawing is not always well supported which can be necessary when using notation that is not easily represented in simple text such as mathematical and chemical notation

      • Client issues, such as native mobile applications with incomplete featuresets, slow web-apps that may require an always-on connection and not be able to cache local changes

      • No spatial memory, whereas in a physical book people may remember where they wrote things and can open it roughly on the right page

      • Extra expense, maintenance and procurement issues with hardware suitable for use in your research environment

  • Lack of sufficiently available training on more complex aspects of the ELN solution leading to poor, mis-use, or under-use of ELN features

Always compare to paper notebooks#

When picking an ELN remember to always compare the ELN to a paper notebook and not just to other ELNs. It is likely paper that you are replacing so this is the benchmark against which to compare. Consider the differences between a paper copy of a lab notebook and an electronic copy. Consider also which properties of a paper copy is it important to you that you are able to retain when adopting an electronic alternative.

  • A paper lab notebook is physically under your control.

  • You (or more likely your organisation) own it.

  • You can control access to it physically.

  • Your physical possession of this resource means that it would be difficult for anyone to prevent you from accessing it.

  • Conversely it must be physically stolen to be accessed by a malicious third party.

  • On the other hand, ELNs - like all software - have security vulnerabilities.

  • You are not dependent on the functioning of any complex systems like computer networks in order to be able to use your paper lab notebook.

  • You do not need any specialist tools in order to access its contents.

  • There is no reason why you might have to pay a third party a fee in order to continue using it.

  • You do not have to agree to a ‘terms of service’ or ‘end-user license agreement’ with a third party to use and retain access to your lab book. These terms may be subject to unilateral alteration by that third party.

If your provider of paper lab books goes out of business it has almost no bearing on your ability to continue doing your work. One paper notebook is much like another, finding a new provider is pretty easy. Changing providers does not impact on your ability to access your past notebooks or to continue operating with the same workflow in future ones. This is not necessarily true of ELNs.

Archival function of notebooks#

Lab notebooks perform an archival function. Few active measures are needed to maintain the data in your paper notebooks. The primary vulnerability of paper lab books is that there is only one copy. Nevertheless, as long as they are kept in a cool, dry and dark spot (for example a fireproof safe) they will likely last decades.

Electronic data requires much more active upkeep. Electronic data can, if managed properly, be more resilient to physical threats such as fire and flooding as it can frequently be backed up in multiple locations. A hard drive, or even a solid state drive, however, cannot be left in a draw for a decade or more and have a high likelihood of working without some amount of bit-rot or compatibility issues when plugged back in.

For digital data it is important for archiving and maintaining the data to use formats that maximize the likelihood of recovering relevant information. Proprietary formats are antithetical to this. Proprietary solutions generally obfuscate the format, even if just through absense of generally available specifications and documentation. Proprietary formats generally have a single implementation from their creator, this creates a single point of failure. Open formats can have multiple different implementations of tools which read them. If there is a good enough specification then new tools can even be written for the format if none of the originals are available anymore. Proprietary tools may require some kind of license activation process to use this may not be possible if their vendor no longer exists. Using a proprietary solution for archival purposes is therefore taking a needless risk with the future of your data. Your data’s fate is tied to the format in which it is stored so choose it wisely.

Lock-in#

Most proprietary or open tools will permit you to export your data in one format or another. The quality of this export is critical to scrutinise. The ability to save all your lab books as PDFs is fine but if it is the only option you may lose a lot of metadata, and the ability to import your data into a new solution. The ability to export your data and retain the original structure is very important to evaluate.

Proprietary solutions are incentivised to attempt to lock-in their customers so that the cost of switching to a competitor is high - so be especially wary of poor export options. Note that this tends to change over time. New providers are focused on user acquisition so data portability is often good at first but tends to decline after a certain point in time. Firms tends to follow a pattern of shifting from a user acquisition phase with favourable terms, to a user retention phase with less favourable ones. Mendeley for example began encrypting their local database. This made it impossible to migrate your local library to different reference managers without going through the Mendeley online library feature. This occurred after they had reached a high degree of market penetration and had been acquired by Elsevier.

When looking for any critical piece of software for long term use the first questions to ask are:

  • Is there an Libre / open-source solution to this problem?

  • If it is a web app and/or has a server: Could I host my own fully featured instance, should I need to?

  • Is there a large/active community using the project, does it have institutional backing of some kind? This might take the form of a company which sells service contracts, or offers paid hosting ideally with feature parity with a self-hosted option. Or perhaps a foundation or other non-profit/academic organisation with robust funding.

  • What are the data export options and formats?

Open solutions provide you with the assurance that if you do the appropriate preparatory work you should be able to access all of your data in its native form. For example keeping copies of the ELN software that can be run offline in a virtual machine (VM) or similar computational environment in the future. This is typically much easier for open applications and can be impossible for proprietary ones. Even if the tools are no longer maintained they can still be used to read the data and interact with it in (mostly) the same way. The data will also likely be stored in an open format from which it can relatively easily be extracted and ported to a new format. This may still be a lot of work but it is much more attainable than reverse engineering a proprietary format for which there is no documentation, or worse - dealing with encrypted files.

You can pay for third party hosting and administration of open source applications. This is a solution when you don’t have the expertise or internal resources to administer a self-hosted instance of an open-source ELN solution. In this case you get the best of both worlds: the benefits of professional support and the reassurance of an open solution. You should still take regular local backups of exports from your hosting provider from which you could restore your ELN system with different hosting. This means that you retain the option to change providers. Conventional Software as a Service (SaaS) platforms are highly vertically integrated: the software development, hosting, administration, and support are package deal that you must take or leave as is. When using open tools each of these layers can separated, though they do not need to be. This gives you the flexibility to change your provider of any of them independently. This improves your bargaining position, but may increase procurement complexity.

Automation and integration#

ELNs with APIs have the potential for integration with instruments that would permit direct deposition of data from connected instruments into ELNs or other connected research data management infrastructure. These sorts of integrations are especially useful in workflows at core facilities. For example, bioimaging facilities can automate a substantial part of the image metadata deposition into image data management tools such as OMERO.

ELN standards and project governance#

ELNs offer many advantages over paper notebooks. Nevertheless, ELNs come with a considerable amount of additional complexity and additional risks - so it is important to have a mitigation plan. The ELN space is still quite young and rapidly changing. As such it is important to think about how the academic community wants the ELN infrastructure to be owned and governed in the long run. An interesting model to consider emulating here is moodle, an open source electronic learning environment popular with universities. It offers commercial support and hosting and has a network of developers across the education sector who contribute to the core project, as well as creating custom integrations and extensions to meet their particular needs. This structure allows institutions to pool resources to fund the development of features that they all need and keeps this infrastructural code in common ownership, avoiding the worst lock-in issues arising from the use of proprietary solutions. This more open distributed model also allows individuals to work on their own custom extensions and integrations instead of being reliant on the ability and willingness of a company to implement a feature that you need.

Without an open standard even if you are using an open source ELN and/or open file format this does not mean that you will necessarily be able to easily use your data with other ELN tools. Such a standard would, if properly adhered too, reduce the lock-in risk associated with using both open and proprietary tools. An initiative to devise ELN format standards would be most welcome.

Resources for choosing ELNs#

  • ELN finder a tool to select an ELN suitable for your purposes

Select Open Infrastructure ELN Options#

The following ELN options are each quite different but have many of the same core features.

  • eLabFTW

    • Has features such as: Laboratory resource scheduling feature for booking things like hoods and microscopes, automatic mol file previews for molecules and proteins, and support for free-hand drawing.

    • eLabFTW has a website, documentation and there is a demo deployment that you can try out.

    • Self-hosting is relatively simple according to the documentation. There is also a paid support tier which would be recommend for any larger deployment to support the ongoing development of the project.

    • Paid cloud hosting is available from the developer in a geographical region suited to your needs. A more expensive tier with hosting in France compliant with additional security and privacy certifications is available.

  • openBIS

    • Robust features for integrated metadata management, for example, linking to ontologies / controlled vocabularies.

    • openBIS has an API and can integrate with Jupyterhub for Electronic Lab Notebooks.

    • Complete LIMS where storage is integrated with protocols and experiment records, including keeping track of bar-coded stocks.

    • You can get a feel for it in the demo deployment.

    • openBIS is a bit more complex to administer based on its documentation.

    • As openBIS is developed at ETH Zurich it can be hosted for you under the openRDM service operated by ETH Zurich scientific IT services. No fixed pricing is available - cost would be dependent on your specific needs.

  • OSF

    • OSF is oriented towards sharing and collaborating on your work, including the ability to generate DOIs and host pre-prints directly on the main instance. See OSF for more information.

    • It is free to use OSF at the main instance at osf.io. For larger data you must provide your own additional storage add-ons, available from a number of cloud storage providers.

    • Whilst you could deploy your own self-hosted OSF this is not how it is intended to be used (except for development).

    • The OSF has strong sharing features, making it easy to share parts of your ELN publicly.

When using an open solution it is always important to consider how you can contribute to its ongoing development and maintenance, for example by donating to the project. If it makes sense given the makeup of your organisation and that of the project you may also be able to contribute developer time to the project.

Using non-ELNs for ELN things#

Note taking and personal knowledge management (PKM) applications of various kinds have been pushed into service as ELNs by many individuals and institutions, despite not being designed specifically for this task. Some tools are better suited to be used ‘off-label’ as ELNs than others. Some may have significant drawbacks compared to purpose built solutions, despite being better at some things than currently available dedicated ELN options.

Microsoft OneNote is quite popular as an ELN despite a proprietary notebook format with limited export options and that Microsoft has entertained getting rid of the good (desktop) version this of application in favour of its much less featureful ‘web’ version in the past. Though they have apparently relented, for now. OneNote is a good experience as a free-form note taking tool especially for hand-written or drawn notes and automatic linking of annotations to audio timestamps. The lack of a good version history is a distinct downside. Its popularity likely stems, in part, from its availability in the Microsoft Office suite (software already provided by institutions which do not incur additional costs).

Other generic note taking applications have featuresets which are not dissimilar to ELN solutions. For example, open source applications like logseq or zettlr, and proprietary ones like: obsidian or notion.

Additional ELN resources#