Machine Learning helps reveal cells' inner structures in new detail

Janelia’s COSEM Project Team has created a set of tools to make annotated 3D images of cells, showing the relationships between different organelles.

Open any introductory biology textbook, and you’ll see a familiar diagram: A blobby-looking cell filled with brightly colored structures – the inner machinery that makes the cell tick. 

Cell biologists have known the basic functions of most of these structures, called organelles, for decades. The bean-shaped mitochondria make energy, for example, and lanky microtubules help cargo zip around the cell. But for all that scientists have learned about these miniature ecosystems, much remains unknown about how their parts all work together. The COSEM Project Team has developed a set of algorithms that can map the organelles in microscope images of cells, creating detailed 3D representations of cells’ inner workings. Credit: COSEM Project Team

Now, high-powered microscopy – plus a heavy dose of machine learning – is helping to change that. New computer algorithms can automatically identify some 30 different kinds of organelles and other structures in super-high-resolution images of entire cells, a team of scientists at the Howard Hughes Medical Institute’s Janelia Research Campus reports.

The detail in these images would be nearly impossible to parse by hand throughout the entire cell, says Aubrey Weigel, who led the Janelia Project Team, called COSEM (for Cell Organelle Segmentation in Electron Microscopy). The data for just one cell is made up of tens of thousands of images; tracing all a cell’s organelles through that collection of pictures would take one person more than 60 years. But the new algorithms make it possible to map an entire cell in hours, rather than years. 

{media load=media,id=258,width=350,align=left,display=inline}

“By using machine learning to process the data, we felt we could revisit the canonical view of a cell,” Weigel says.

Janelia scientists also released a data portal, OpenOrganelle, where anyone can access the datasets and tools they’ve created.

These resources are invaluable for scientists studying how organelles keep cells running, says Jennifer Lippincott-Schwartz, a senior group leader and interim head of Janelia’s new 4D Cellular Physiology research area who is already using the data in her research. “What we haven’t really known is how different organelles and structures are arranged relative to each other – how they’re touching and contacting each other, how much space they occupy,” she says.

For the first time, those hidden relationships are visible. 

 
Using specialized electron microscopes, scientists capture thousands of images of cells, layer by layer. Within parts of those images, scientists painstakingly trace individual organelles by hand.
Stacking up the traced layers creates a 3D dataset. Then, scientists train machine learning algorithms on the data, teaching the computer to recognize different organelles. With enough example data, the computer can efficiently pick organelles out of images it’s never seen before, mapping organelle boundaries as shown here.
Those images are compiled into a 3D dataset with each organelle identified, revealing cells’ interiors in full detail. Additional tools can home in on specific organelles and pinpoint the places they interact. This image shows the interaction between mitochondria (orange) and the endoplasmic reticulum (green.)

 

Detailed data

The COSEM team’s journey started with data collected by high-powered electron microscopes housed in a special vibration-proof room at Janelia. 

For the past ten years, these microscopes have been churning out high-resolution snapshots of the fly brain. Janelia Senior scientist Shan Xu and senior group leader Harald Hess have engineered these scopes to mill off super-thin slivers of the fly brain using a focused beam of ions – an approach called FIB-SEM imaging. The scopes capture images layer by layer, and then computer programs stitch those images together into a detailed 3D representation of the brain. Based on these data, Janelia researchers released the most detailed neural map of the fly brain yet. 

While imaging the fly brain, Hess and Xu’s team also looked at other samples. Over time, they amassed a collection of data from many kinds of cells, including mammalian cells. “We thought that this detailed imaging of whole cells might be of larger interest to cell biologists,” Hess says. 

{media load=media,id=259,width=350,align=left,display=inline}

Weigel, then a postdoc in Lippincott-Schwartz’s lab, began mining those data for her research. “The resolving power of the FIB-SEM imaging was amazing, and we were able to see things at a level we couldn’t have imagined before,” says Weigel, “but there was more information in one sample than I could analyze in several lifetimes.” Realizing that others at Janelia were working on computational projects that might speed things up, she began organizing a collaboration. 

“All of the pieces were here at Janelia,” she says, and forming the COSEM project team aligned them towards a common goal.

Setting boundaries

Larissa Heinrich, a graduate student in group leader Stephan Saalfeld’s lab, had previously developed machine learning tools that could pinpoint synapses, the connections between neurons, in electron microscope data. For COSEM, she adapted those algorithms to instead map out, or segment, organelles in cells.

Saalfeld and Heinrich’s segmentation algorithms worked by assigning each pixel in an image a number. The number reflected how far the pixel was from the nearest synapse. An algorithm then used those numbers to ID and label all the synapses in an image. The COSEM algorithms work similarly, but with more dimensions, Saalfeld says. They classify every pixel by its distance to each of 30 different kinds of organelles and structures. Then, the algorithms integrate all of those numbers to predict where organelles are positioned.

Using data from scientists who have manually traced organelle boundaries and assigned numbers to pixels, the algorithm can learn that particular combinations of numbers are unreasonable, Saalfeld says. “So, for example, a pixel can’t be inside a mitochondrion at the same time it’s inside the endoplasmic reticulum.”

To answer questions like how many mitochondria are in a cell, or what their surface area is, the algorithms need to go even further, says group leader Jan Funke. His team built algorithms that incorporate prior knowledge about organelles’ characteristics. For example, scientists know that microtubules are long and thin. Based on that information, the computer can make judgments about where a microtubule begins and ends. The team can observe how such prior knowledge affects the computer program’s results – whether it makes the algorithm more or less accurate – and then make adjustments where necessary. 

After two years of work, the COSEM team has landed on a set of algorithms that generate good results for the data that have been collected so far. Those results are the important groundwork for future research at Janelia, says Weigel.  A new effort headed by Xu is taking FIB-SEM imaging to even greater levels of detail. And another soon-to-launch project team named CellMap will further refine COSEM’s tools and resources to create a more expansive cell annotation database, with detailed images of many more types of cells and tissue.

Together, those advances will support Janelia’s next 15-year research area, 4D Cellular Physiology – an effort led on an interim basis by Lippincott-Schwartz to understand how cells interact with each other within each of the many different kinds of tissue that make up an organism, says Wyatt Korff, Director of Project Teams at Janelia.

With new resources like those created by the COSEM team and the Enhanced FIB-SEM Technology group, Korff says, “we can actually begin to answer those questions, in a way that we haven’t had access to in the past.”

Swiss researchers develop new high-resolution PV production forecasts

In May of 2021, the International Energy Agency (IEA) has announced in its flagship report the measures it believes governments and policymakers should be taking to limit global warming to 1.5˚C. The report highlighted that to guarantee stable and affordable energy supplies now, and in the future, there needs to be a massive expansion of clean energy resources.

PVLIVE: high-resolution, large-scale data-driven photovoltaic (PV) system production forecasting

At CSEM – the prestigious Swiss Research and Development Organization – limiting global warming to 1.5˚C is not simply talk. Under the leadership of Pierre-Jean Alet, an experienced Sector Head, CSEM’s researchers are working on software that will drastically improve the management of PV-produced energy, from local energy communities to national grids. At CSEM, we listen and respond to the very real need of grid operators, who can’t always accurately predict PV energy generation. Currently, the operators must intervene to cover any energy shortfalls caused by inaccurate predictions and re-balance energy across their systems using alternative resources. This can be both costly and lead to additional carbon emissions,” he notes. “Accurate and robust PV production forecasts are thus an important and necessary tool to increase the amount of clean energy in the power system and ensure stable energy distribution across the power grid. They also help minimize any maintenance and balancing costs – this is why we developed PVLIVE,” enthuses Alet.  Graphic

Turning PV systems into ground-level weather stations

PVLIVE doesn’t just offer forecasting over a single or a few localized sites, it provides optimized PV production forecasting across entire countries, and its algorithms were tested for over a year on exceptionally large datasets. Uniquely compared to other forecasting systems, “CSEM’s solution is based on graph machine learning, the kind of AI algorithms that power social networks and recommendation systems in e-commerce,” says Rafael Carrillo, a Senior Researcher at CSEM. “We used this technique creatively to exploit the Spatio-temporal relationships between different PV systems: intuitively, if you are under the dominant wind direction, what you see on one system will eventually show up at another system further down the line. Effectively with PVLIVE, we turn PV systems into ground-level weather stations to forecast the production of their neighbors,” concludes Carrillo.

Using this technique, CSEM’s data scientists can provide forecasts across hundreds of PV systems that show a daytime normalized root-mean-square error of 13.8% with a forecasting horizon of 6 h in 15 min steps. This outperforms other platforms based on numerical weather forecasts and machine learning. Through a follow-up project, PVLIVE is now available as an interactive platform. You can view the platform in operation across the Netherlands and the project’s paper HERE.  

UMD's quantum supercomputer experiments show that combining its pieces does not have to mean combining their error rates

Pobody’s nerfect—not even the indifferent, calculating bits that are the foundation of computers. But JQI Fellow Christopher Monroe’s group, together with colleagues from Duke University, have made progress toward ensuring we can trust the results of quantum supercomputers even when they are built from pieces that sometimes fail. They have shown in an experiment, for the first time, that an assembly of quantum computing pieces can be better than the worst parts used to make it. The team has shown how they took this landmark step toward reliable, practical quantum supercomputers. A chip containing an ion trap that researchers use to capture and control atomic ion qubits (quantum bits).  CREDIT Kai Hudek/JQI)

In their experiment, the researchers combined several qubits—the quantum version of bits—so that they functioned together as a single unit called a logical qubit. They created the logical qubit based on a quantum error correction code so that, unlike for the individual physical qubits, errors can be easily detected and corrected, and they made it to be fault-tolerant—capable of containing errors to minimize their negative effects.

“Qubits composed of identical atomic ions are natively very clean by themselves,” says Monroe, who is also a Fellow of the Joint Center for Quantum Information and Computer Science and a College Park Professor in the Department of Physics at the University of Maryland. “However, at some point, when many qubits and operations are required, errors must be reduced further, and it is simpler to add more qubits and encode information differently. The beauty of error correction codes for atomic ions is they can be very efficient and can be flexibly switched on through software controls.”

This is the first time that a logical qubit is more reliable than the most error-prone step required to make it. The team was able to successfully put the logical qubit into its starting state and measure it 99.4% of the time, despite relying on six quantum operations that are individually expected to work only about 98.9% of the time.

That might not sound like a big difference, but it’s a crucial step in the quest to build much larger quantum computers. If the six quantum operations were assembly line workers, each focused on one task, the assembly line would only produce the correct initial state 93.6% of the time (98.9% multiplied by itself six times)—roughly ten times worse than the error measured in the experiment. That improvement is because in the experiment the imperfect pieces work together to minimize the chance of quantum errors compounding and ruining the result, similar to watchful workers catching each other's mistakes.

The results were achieved using Monroe’s ion-trap system at UMD, which uses up to 32 individual charged atoms—ions—that are cooled with lasers and suspended over electrodes on a chip. They then use each ion as a qubit by manipulating it with lasers.

“We have 32 laser beams,” says Monroe. “And the atoms are like ducks in a row; each with its own fully controllable laser beam. I think of it like the atoms form a linear string and we're plucking it like a guitar string. We're plucking it with lasers that we turn on and off in a programmable way. And that's the computer; that's our central processing unit.”

By successfully creating a fault-tolerant logical qubit with this system, the researchers have shown that careful, creative designs have the potential to unshackle quantum computing from the constraint of the inevitable errors of the current state of the art. Fault-tolerant logical qubits are a way to circumvent the errors in modern qubits and could be the foundation of quantum computers that are both reliable and large enough for practical uses.

Correcting Errors and Tolerating Faults

Developing fault-tolerant qubits capable of error correction is important because Murphy’s law is relentless: No matter how well you build a machine, something eventually goes wrong. In a computer, any bit or qubit has some chance of occasionally failing at its job. And the many qubits involved in a practical quantum computer mean there are many opportunities for errors to creep in.

Fortunately, engineers can design a computer so that its pieces work together to catch errors—like keeping important information backed up to an extra hard drive or having a second person read your important email to catch typos before you send it. Both the people or the drives have to mess up for a mistake to survive. While it takes more work to finish the task, the redundancy helps ensure the final quality.

Some prevalent technologies, like cell phones and high-speed modems, currently use error correction to help ensure the quality of transmissions and avoid other inconveniences. Error correction using simple redundancy can decrease the chance of an uncaught error as long as your procedure isn’t wrong more often than it’s right—for example, sending or storing data in triplicate and trusting the majority vote can drop the chance of an error from one in a hundred to less than one in a thousand.

So while perfection may never be in reach, error correction can make a computer’s performance as good as required, as long as you can afford the price of using extra resources. Researchers plan to use quantum error correction to similarly complement their efforts to make better qubits and allow them to build quantum computers without having to conquer all the errors that quantum devices suffer from.

“What's amazing about fault tolerance, is it's a recipe for how to take small unreliable parts and turn them into a very reliable device,” says Kenneth Brown, a professor of electrical and computer engineering at Duke and a co-author. “And fault-tolerant quantum error correction will enable us to make very reliable quantum computers from faulty quantum parts.”

But quantum error correction has unique challenges—qubits are more complex than traditional bits and can go wrong in more ways. You can’t just copy a qubit, or even simply check its value in the middle of a calculation. The whole reason qubits are advantageous is that they can exist in a quantum superposition of multiple states and can become quantum-mechanically entangled with each other. To copy a qubit you have to know exactly what information it’s currently storing—in physical terms you have to measure it. And a measurement puts it into a single well-defined quantum state, destroying any superposition or entanglement that the quantum calculation is built on. 

So for quantum error correction, you must correct mistakes in bits that you aren’t allowed to copy or even look at too closely. It’s like proofreading while blindfolded. In the mid-1990s, researchers started proposing ways to do this using the subtleties of quantum mechanics, but quantum computers are just reaching the point where they can put the theories to the test.

The key idea is to make a logical qubit out of redundant physical qubits in a way that can check if the qubits agree on certain quantum mechanical facts without ever knowing the state of any of them individually.

Can’t Improve on the Atom

There are many proposed quantum error correction codes to choose from, and some are more natural fits for a particular approach to creating a quantum supercomputer. Each way of making a quantum supercomputer has its types of errors as well as unique strengths. So building a practical quantum supercomputer requires understanding and working with the particular errors and advantages that your approach brings to the table.

The ion trap-based quantum supercomputer that Monroe and colleagues work with has the advantage that their qubits are identical and very stable. Since the qubits are electrically charged ions, each qubit can communicate with all the others in the line through electrical nudges, giving freedom compared to systems that need a solid connection to immediate neighbors.

“They’re atoms of a particular element and isotope so they're perfectly replicable,” says Monroe. “And when you store coherence in the qubits and you leave them alone, it exists essentially forever. So the qubit when left alone is perfect. To make use of that qubit, we have to poke it with lasers, we have to do things to it, we have to hold on to the atom with electrodes in a vacuum chamber, all of those technical things have noise on them, and they can affect the qubit.”

For Monroe’s system, the biggest source of errors is entangling operations—the creation of quantum links between two qubits with laser pulses. Entangling operations are necessary parts of operating a quantum computer and of combining qubits into logical qubits. So while the team can’t hope to make their logical qubits store information more stably than the individual ion qubits, correcting the errors that occur when entangling qubits is a vital improvement.

The researchers selected the Bacon-Shor code as a good match for the advantages and weaknesses of their system. For this project, they only needed 15 of the 32 ions that their system can support, and two of the ions were not used as qubits but were only needed to get an even spacing between the other ions. For the code, they used nine qubits to redundantly encode a single logical qubit and four additional qubits to pick out locations where potential errors occurred. With that information, the detected faulty qubits can, in theory, be corrected without the “quantum-ness” of the qubits being compromised by measuring the state of any individual qubit.

“The key part of quantum error correction is redundancy, which is why we needed nine qubits in order to get one logical qubit,” says JQI graduate student Laird Egan, who is the first author of the paper. “But that redundancy helps us look for errors and correct them because an error on a single qubit can be protected by the other eight.”

The team successfully used the Bacon-Shor code with the ion-trap system. The resulting logical qubit required six entangling operations—each with an expected error rate between 0.7% and 1.5%. But thanks to the careful design of the code, these errors don't combine into an even higher error rate when the entanglement operations were used to prepare the logical qubit in its initial state.

The team only observed an error in the qubit's preparation and measurement 0.6% of the time, less than the lowest error expected for any of the individual entangling operations. The team was then able to move the logical qubit to a second state with an error of just 0.3%. The team also intentionally introduced errors and demonstrated that they could detect them.

“This is really a demonstration of quantum error correction improving performance of the underlying components for the first time,” says Egan. “And there's no reason that other platforms can't do the same thing as they scale up. It's really a proof of concept that quantum error correction works.”

As the team continues this line of work, they say they hope to achieve similar success in building even more challenging quantum logical gates out of their qubits, performing complete cycles of error correction where the detected errors are actively corrected, and entangling multiple logical qubits together.

“Up until this paper, everyone's been focused on making one logical qubit,” says Egan. “And now that we’ve made one, we're like, ‘Single logical qubits work, so what can you do with two?’”