Bringing digital biology to life
Our mission is to empower our users by providing a state-of-the-art, affordable and easy-to-use IT environment.
The EMBL 3D Cloud is a flexible, scalable and virtualised GPU cloud infrastructure to support large-scale analysis and visualisation of “big” imaging data. It provides EMBL users with shared access to high-end graphics workstation performance on their local computer or mobile device from anywhere in the lab or remotely, for example, when presenting on a conference. The tight integration with the high-performance network, data storage and compute resources of the EMBL data centre provides additional benefits in terms of overall performance.
The project stemmed from a simple question: ‘Is there a better way?’. Scientists at EMBL produce vast amounts of increasingly complex image data, for example, from light and electron microscopy which efficiently need to be analysed and stored.
“On our most advanced microscope systems such as light-sheet or high-throughput microscopes we are dealing with hundreds of gigabytes up to several terabytes of data from one microscope session”, says Rainer Pepperkok, Head of EMBL Advanced Light Microscopy Core Facility (ALMF) and Head of EMBL Core Facilities Unit. “And our users run dozens of sessions every week.”
At the time, image data analysis in the ALMF was facilitated by multiple standalone high-performance graphics workstations. These machines were nearing the end of their lifecycle, and the scientists were keen to replace them with a solution that could tackle the challenges around increasing data load. Comparable local workstation-based analysis was found in most other groups working with imaging data involving similar challenges in their workflows.
Developed by IT Services, the EMBL 3D Cloud is based on input from key users across EMBL’s Core Facilities and scientific groups. It aims to help scientists tackle the afore-mentioned challenges and to take image data analysis at EMBL to the next level. Conceptually it virtualises graphical power in a central GPU-cloud and takes advantage of the overall performance and capacity of the EMBL data centre networking, compute and storage resources and greatly reduces the need to shift large amounts of data around.
From an IT perspective, responding to our scientists’ needs meant addressing several challenges:
| IT challenges | IT solutions | Technical details |
|---|---|---|
| Improve processing power Provide access to large amounts of CPU, RAM and fast disk resources in a cost-efficient manner. | Replace local desktop workstations with virtual machines (VMs). Integrate SSD disk technology to power the VMs. | VMware Horizon 3D software platform: partner to our existing VMware vSphere infrastructure. |
| Improve graphics power | Share the power of an individual powerful GPU graphics card across multiple virtual machines. | Nvidia GRID technology twinned with our selected VDI platform, to deliver the required 3D graphics and CUDA capabilities. |
| Make it scalable and flexible Ensure the system can keep up as user needs expand; also adapt to the specific needs of individual imaging software. | Provide multiple servers with an abundance of resources which can be allocated based on user demand. Add server hardware when needed. | 4 virtualisation servers hosting the VMs from a dedicated cluster. VDI platform provides shared GPU access for intensively graphics-based and CUDA-aware scientific software. |
| Increase efficiency Cut down on time and resources spent moving data from acquisition to storage and analysis platforms. | Process data at point of acquisition instead of moving to different location for analysis. | 10GbE network connectivity attaches the 3D Cloud directly to the EMBL data centre network backbone, ensuring the most efficient access to high-performance network file storage on campus. |
The EMBL IT Services team has been investing in virtualisation and cloud technology for many years. EMBL data centres are largely virtualised and the provision of scalable and robust IT services is standardised on established technology from market leading technology providers in this field. Based on this technology it was possible to seamlessly integrate specialised cloud GPU hardware from another market leader to form the basis for the EMBL 3D Cloud. “The 3D Cloud builds on proven performance, reliability and high standards in advanced virtualisation and GPU technology. As an added advantage the solution allows us to keep our IT environment lean and standardised while ensuring new services can be integrated seamlessly,” says Rupert Lueck, Head of IT Services at EMBL.
The EMBL IT team first tested the new architecture based on a pilot system using a first iteration of a so-called GRID GPU that allowed for shared GPU use. “The idea of the pilot was to really test what this solution could do in terms of delivering 3D graphics to multiple virtual desktops using a single GPU,” says Daniel Anderson, Manager Desktop Support & Windows Admin. The pilot also provided a baseline measure to compare the virtual machine performance to the traditional standalone workstations.
Following initial success, the pilot system was further extended in terms of hardware and developed to a standardised building block for the future hypervisor server. Additional networking, newer generation graphics cards and more RAM allowed for greater capacity and better resilience to support production level operation.
The system was then released for testing to key users from the ALMF and the Electron Microscopy Core Facility. “We were really impressed by the reliability, flexibility and performance of the 3D Cloud,” says Christian Tischer, a staff scientist at the ALMF. After the 3D Cloud was put into production, it was further expanded by subsequently adding hypervisors based on the previously defined standard to deal with the increasing demand from different scientific groups and core facilities at EMBL. “At the same time, we made major improvements in network connectivity, adding multiple fibre-optic connections to each hypervisor for improved networking performance and reliability,” says systems engineer Carlos Fernandez San Millan. In terms of hardware capacity the 3D Cloud presently offers access to 128 GB GPU memory, 33,000 CUDA cores and 2TB RAM.
Replacing local workstations with cloud-based graphics virtual machines includes additional advantages. The cloud-based system on the one hand saves on footprint in the lab and on the other now allows scientists to access and process their data remotely. “Scientists don’t need to be in the room, they can analyse their data basically from anywhere on the campus or even remotely, at any time,” says Anderson. “This isn’t something we set out to address, but it’s turned out to be very welcomed. Scientists are happy that they can analyse their data without having to worry if the room for data analysis is already booked,” San Millan adds.
EMBL 3D Cloud: The key benefits for EMBL scientists
| Faster access to data | Scientists can process their data on-demand in a centralised location without the need to move large datasets multiple times, saving time and resources. |
| Seamless integration | The virtualised graphics workstations support standard image analysis software packages and enable users to perform typical workflows and 3D operations with their image data just like sitting in front of a very powerful workstation. |
| Flexibility | Resources are easily allocated on-demand for short-term projects or for longer-term usage. |
| Remote access | Scientists can connect to their 3D Cloud graphics workstations while being away from the lab. Even with limited bandwidth available the underlying image streaming and compression technology provides highly efficient access to 3D visualisation and operations. |
| Scalability | Need more resources, add more servers. The use of EMBL 3D Cloud resources is automatically distributed across the available hardware resources. If exhausted, additional capacity can be added by adding hypervisors to the existing cluster. |
Others can now reap these benefits, too. “The EMBL 3D Cloud has become quite successful and is already used across a number of EMBL groups and core facilities involved in large-scale image analysis. We are committed to broadening the use of this platform to other projects and areas requiring access to GPU power,” Lueck concludes.