GraspJ: an open source, real-time analysis package for super-resolution imaging
© Brede and Lakadamyali; licensee Springer. 2012
Received: 5 June 2012
Accepted: 24 November 2012
Published: 5 December 2012
We present an open source, real-time data analysis and rendering tool for super-resolution imaging techniques that are based on single molecule detection and localization (e.g. stochastic optical reconstruction microscopy - STORM and photoactivation localization microscopy – PALM). The recent availability of commercial STORM and PALM microscopes has made these techniques accessible to a wide range of researchers. However, the availability of high speed data analysis and rendering software lags behind, requiring researchers to develop their own analysis platforms or rely on commercial ones. We implemented GraspJ (GPU-Run Analysis for STORM and PALM), an ImageJ plug-in with a convenient user interface, that allows high accuracy localization of single molecules as well as processing and rendering of high resolution images in real-time. GraspJ includes several features such as drift correction, multi-color, 3D analysis/rendering, and is compatible with a large range of data acquisition software. In addition, it allows easy interfacing with other image processing tools available with ImageJ. Overall we believe that GraspJ will be a valuable tool for the super-resolution imaging field.
KeywordsSTORM PALM Maximum likelihood GPU
In the past decade, a number of techniques have been developed that overcome the diffraction limit in fluorescence microscopy (Hell, , Bates et al., ). Among such techniques, those that are based on single molecule photoactivation and localization (e.g. STORM, PALM, fPALM) depend heavily on data processing and analysis to produce high resolution images. These techniques break the diffraction limit by taking advantage of photoswitchable fluorophores (Fernandez-Suarez and Ting, , Patterson et al., , Dempsey et al., ) that can be “photo-activated” in sparse numbers. By only activating a small fraction of fluorophores that are spatially well separated, one can determine their mean positions precisely, even under dense labeling conditions. Repeated cycles of “photo-activation” followed by “imaging” and “deactivation” allows mapping of a large number of fluorophore-positions which can then be rendered into a high resolution image of a macromolecular structure, such as microtubules inside the cell (Bates et al., ).
A software package for the analysis of single molecule localization microscopy data needs to incorporate multiple features: (i) High accuracy localization of single molecule mean positions, (ii) sufficient data processing options (drift correction, high resolution rendering, multi-color and 3D analysis and display) and (iii) high speed for real-time analysis and rendering. Initial analysis programs for single molecule localization based super-resolution microscopy used a Gaussian model of the point spread function, which was fit to images of single molecules using a nonlinear least-squares method such as the Levenberg–Marquardt algorithm (Rust et al., , Betzig et al., ). The computational demands of fitting and processing millions of molecules per image made these programs slow and several minutes to hours were needed for analyzing a single image. In the recent past, a number of algorithms have been developed to address the problem of speed in super-resolution data analysis (Smith et al., , Wolter et al., ). Some of these algorithms have been further developed into open-source programs with multiple features for fast analysis and rendering of super-resolution images (Wolter et al., , Henriques et al., , York et al., ). For example, QuickPALM is an ImageJ based plug-in that can process on the order of 2000–3000 molecules/second using an 8 CPU computer (Henriques et al., ). QuickPALM achieves this high speed by determining molecule positions via calculating the center-of-mass of single-molecule images instead of fitting (Gaussian) PSFs to them. However, the center-of-mass is an inaccurate method for determining molecule positions (Smith et al., , Cheezum et al., , Wolter et al., ) therefore is not the preferred algorithm for super-resolution imaging. In addition, the speed achieved by QuickPALM falls short of real-time analysis when high fluorophore densities and fast acquisition speeds are needed (e.g. 60 Hz or higher speeds used for live cell imaging (Jones et al., )). Finally, when used for real-time analysis, QuickPALM is only compatible with one data acquisition software (μManager), which is a drawback for incorporating the data analysis with pre-existing acquisition software. rapidSTORM is another open source program that uses a more accurate algorithm (nonlinear least squares fitting of Gaussian PSFs) for determining molecule positions and that can achieve high speed analysis of super-resolution data (Wolter et al., ). However, in the real-time analysis mode, rapidSTORM only works with Andor cameras and is not compatible with other acquisition software, as it directly takes control over the camera. In addition, since rapidSTORM is a stand-alone program it cannot be easily modified or interfaced with powerful image processing tools offered by ImageJ. Therefore, there is still need for open source, easy to use, high speed and high accuracy software packages that include essential data processing features and are compatible with a wide range of data acquisition software. Such analysis tools will make super-resolution microscopy more accessible to researchers.
Here, we present a software package that can be used as an ImageJ plug-in or as a stand-alone program and that incorporates multiple features for analysis and rendering of super-resolution data. GraspJ includes peak finding, 2D and 3D maximum likelihood estimation (with Gaussian PSFs) of peak positions, drift correction and multi-color and 3D rendering of high resolution images. Most importantly, GraspJ runs at very high speeds both on the graphics card (>1*106 molecules/second processed on the GPU) and the central processing unit (~6*104 molecules/second processed on a 6-core CPU) and can be used to render 3D, drift corrected, high resolution images acquired at maximum frame rates attainable by modern CCD cameras in real-time. Moreover, due to the way in which real-time analysis is implemented, GraspJ can be compatible with almost any data acquisition software.
Materials and methods
Design and implementation of GraspJ
GraspJ is written in a highly abstracted, object oriented manner, making it easily extensible. Most parts are written in Java; while computationally expensive parts, like peak fitting, are implemented in OpenCL. It requires Java 7 or higher and OpenCL 1.1 or higher in order to run. All features of GraspJ are accessible and configurable from the graphical user interface. More information on how to run GraspJ, analyze the sample files and supported hardware can be found in the user manual provided here as Additional file 1. Fiji with GraspJ plugin, sample data for analysis and sample workflow can be downloaded from the GraspJ website ( http://www.graspj.com).
Cell culture and immunostaining
BSC-1 cells were cultured inside chambered 8-well coverglasses (Labtek II, Nunc) in a 5% CO2 incubator using Dulbecco’s Modified Eagle Medium (DMEM) supplemented with 10% Fetal Bovine Serum (FBS). Cells were fixed prior to immunostaining with 3% paraformaldehyde (PFA) and 0.1% glutaraldehyde (GA) diluted in phosphate-buffered silane (PBS). Following fixation, cells were rinsed with PBS, and reduced with a fresh solution of 0.1% Sodium Borohydride in PBS. Cells were then rinsed once more with PBS and immunostained with primary and secondary antibodies as described previously (Bates et al., ). For tubulin and mitochondria labeling a rat-anti-tubulin primary antibody (Abcam, Cambridge, UK) and a rabbit-anti-Tom20 primary antibody (Santa Cruz Biotechnologies, Santa Cruz, CA) were used respectively. Secondary antibodies (Jackson Immuno Research, West Grove, PA) were custom labeled with activator and reporter dyes as described previously (Bates et al., ). For tubulin, a secondary antibody against rat labeled with Cy3 as an activator and A647 as a reporter dye was used. Similarly, for mitochondria a secondary antibody against rabbit labeled with A405 as an activator dye and Alexa 647 as a reporter dye was used.
For STORM imaging, cells were placed in a PBS imaging buffer containing 100mM cysteamine (Sigma Aldrich, St. Louis, MO) pH 8.5, 5% glucose (wt/vol), oxygen scavenging enzymes (0.5 mg ml-1 glucose oxidase) (Sigma Aldrich, St. Louis, MO), and 40 μg ml-1 catalase (Sigma-Aldrich, St. Louis, MO).
Images were acquired either on a “custom-built” microscope fitted with a 100 × 1.4 NA oil immersion objective (Olympus, Center Valley, PA) as described previously ( [Bates 2010]) or on a commercial Nikon STORM microscope (Nikon Instruments Europe B.V., Amstelveen, Netherlands) at ICFO’s Super-resolution Light Nanoscopy (SLN) facility. Multi-color images were acquired using a repetitive sequence of activation-imaging light as described previously (Bates et al., ). 532 nm light was used to activate the Cy3-A647 pair and 405 nm light was used to activate the A405-A647 pair. 647 nm light was used following each activation pulse in order to excite A647 and the fluorescence emission from A647 was recorded by an EMCCD camera (Andor Technology, Belfast, Northern Ireland) at a frame rate of 60 Hz. The localizations were later color coded based on whether the A647 emission was detected after the 405 nm, or 532 nm activation light. For 3D imaging, a cylindrical lens was used in the image path in order to introduce astigmatism to the single molecule point spread function as described previously (Huang et al., [2008a]).
Peak finding and fitting
The peak finding and fitting give robust results when the emitter densities do not exceed the range appropriate for single molecule fitting (i.e. as long as the emitters are not substantially overlapping in each frame, typically ~0.5-2 fluorophores per μm2) (Huang et al., , Zhu et al., ). Imaging at higher emitter densities requires the use of multi-peak fitting algorithms (Huang et al., , Zhu et al., , Babcock et al., ), which is currently not implemented in GraspJ but would be a useful future extension of its capabilities. Positions are finally filtered based on typical parameters that can be set by the user (e.g. uncertainty of the positions, minimum and maximum photon counts etc.). GraspJ also reports the average localization precision for a given image, which is one of the determining factors in the final image resolution.
Data processing and rendering
Data acquisition for super-resolution imaging typically lasts several minutes. During this time, there can be substantial stage drift that degrades resolution. Stage drift can be corrected in multiple ways. For example, fiduciary markers such as fluorescent beads can be tracked and their average motion subtracted from the mean positions (Rust et al., , Betzig et al., ). Alternatively, the dataset can be divided into a number of equal length time segments for which drift is assumed to be small. Each time segment is then rendered as a high resolution image and the shift between these partial images is determined using an image registration method (intrinsic drift correction) (Bates et al., , Huang et al., [2008a]). As part of GraspJ, we implemented the latter approach in which we use a phase correlation method to compare rendered images from subsequent time segments. This “intrinsic” drift correction performs well if the image contains a well-defined structure, which can be correlated between high resolution images constructed from subsets of frames. However, in the case of images that lack a structure (for example small clusters of localizations) this approach can still be applied by adding fiduciary markers to the sample. If the fiduciary markers appear in a frame that does not contain fluorescence from the single molecules (i.e. an activation frame), then the positions of these markers alone can be used for “intrinsic” drift correction (see User Manual for more details).
High resolution rendering of final images
Mean-position determination and drift-correction produces a list of molecular positions, typically in the range of 105-107 localizations depending on the dataset. In order to convert this list to a meaningful high resolution image, the fluorophore positions must be rendered. GraspJ uses a common rendering approach in which localizations are represented as Gaussian peaks of unit volume and width corresponding to the theoretical localization precision (Bates et al., , Huang et al., [2008b]). For single color data, all localizations are coded with the same color. Multi-color coding based on STORM imaging with multiple activator dyes is also possible (Bates et al., ). In this approach, localizations are color coded based on when they appear in a cycle consisting of “activation” and “imaging” frames (Bates et al., ). For example, for a sequence of one activation frame (405 nm laser), three imaging frames (647 nm laser) followed by one activation frame (560 nm laser) and three imaging frames (647 nm laser), the localizations appearing in the frame immediately after the 405 nm activation laser are coded in one color and those appearing in the frame immediately after the 560 nm laser are coded in a different color. All other localizations are coded as “non-specific”, since they cannot be directly linked to the activation laser. Each color is displayed in a separate window, which can later be overlaid using ImageJ. The use of different activators linked to the same reporter dye is prone to color cross talk due to the non-specific or false activations as previously described (Bates et al., , Dani et al., , Bates et al., , Lakadamyali et al., ). Statistical methods have also been developed to correct for color cross-talk in these multi-color images (Dani et al., , Bates et al., ). Future implementation of these cross-talk correction algorithms would further enhance GraspJ’s feature-set. For 3D data, localizations are color-coded based on their z-positions using a continuous color scale (Huang et al., [2008a], Huang et al., [2008b], Shtengel et al., ). The rendered 2D and 3D images can be visualized within GraspJ at multiple zoom levels and the image is re-rendered at high resolution for each zoom level. Overall GraspJ allows easy visualization of multi-color, 2D and 3D super-resolution images.
Data analysis is often a major bottleneck in super-resolution imaging techniques based on single molecule localization. Existing algorithms mostly rely on post-acquisition analysis, which dramatically slows down the progress of research. Decisions regarding the quality and information content of the images cannot be made until several hours after the image acquisition is completed. Therefore, typically several images are recorded only to be discarded later due to low localization density, high drift or other factors that lead to low image quality.
Additional file 2: Real-time analysis demonstration: The movie shows a high resolution, multi-color and 3D image of microtubules and mitochondria building up in real-time during a simulated STORM image acquisition in which a previously acquired 256 X 256 pixel dataset was re-written onto the hard disk at a speed of 60 Hz. The analysis was performed using a 256-frame analysis package. The first two windows show the separate channels (mitochondria and microtubules respectively) and the final window shows the combined channels with z-color-coding. Initially, the movie starts at a high zoom level in which the image of an individual mitochondrion can be seen building up in the first window. The zoom level is dynamically and synchronously changed during analysis in order to inspect different regions of the image. (WMV 4 MB)
Real-time analysis implementation
In order to make GraspJ compatible with real-time analysis of data acquired with a wide range of acquisition software, the data is read-out and analyzed in portions as it is being written to the hard disk. GraspJ loads a certain number of frames (referred to as one package) as they are recorded, performs the peak finding, mean position determination, intrinsic drift correction and renders the high resolution image in real-time. As more frames accumulate, the data is loaded and the rendered image is updated accordingly. This approach works well for images recorded in the form of raw data but is also compatible with most other data formats that can be loaded using the LOCI Bio-Formats library.
The provided additional movie file (Additional file 2) demonstrates real-time analysis and rendering of 2-color and 3D image of microtubules and mitochondria acquired at 60Hz. The first two windows display the individual channels (mitochondria and microtubules, respectively) rendered in 2D and the final window displays the merged image color-coded in 3D. A drift corrected, high resolution image is displayed and updated after each package, which in this case was chosen to be 256 frames. The user can examine the image at multiple dynamic zoom levels and pan to different positions within the image as the image is building up. This ability to inspect images at multiple zoom levels in high resolution makes GraspJ a very powerful tool for super resolution imaging as important decisions regarding the sample and data can be made instantaneously.
A number of open source programs are available for super-resolution imaging (Wolter et al., , Henriques et al., , York et al., ); however, the combination of high speed, high accuracy, compatibility with a range of acquisition software and interfacing ability with ImageJ makes GraspJ unique. The maximum likelihood estimator used for determining molecule positions achieves theoretically maximum accuracy (Abraham et al., ). The speed realized on the GPU surpasses the maximum required speed for real-time analysis with currently used frame rates and molecule densities. Recently, multiple-peak-fitting algorithms have been developed to increase the tolerable density of molecules in each frame (Huang et al., , Zhu et al., , Babcock et al., ). Future implementation of these algorithms would dramatically decrease the speed. However, since GraspJ is implemented on the GPU, the speeds achieved would still be compatible with real-time analysis while other open source programs that only run on the CPU may not achieve real-time analysis speeds with these algorithms (Huang et al., , Zhu et al., , Babcock et al., ). Overall, GraspJ should prove useful for researchers interested in super-resolution imaging.
In summary we developed a fully functional software analysis package that not only provides the list of molecular positions obtained from single molecule photoactivation and localization data but also renders these positions as a high resolution, drift-corrected image in real-time. The easy to use interface provides visualization options of the rendered high resolution images in multi-color and 3D. Since GraspJ can be used as an ImageJ plug-in, it can be easily interfaced with all other image processing tools available through ImageJ. In addition, users can easily extend the capabilities of GraspJ by creating and incorporating additional modules (such as different fitting algorithms, rendering types or additional analysis methods) in Java, without having to modify the remaining source code.
This work was supported in part by Fundació Cellex Barcelona. The peak finding, trail generation and drift correction algorithms are largely based on previous software that was generously provided by Dr. Mark Bates (Max Planck Institute for Biophysical Chemistry, Gottingen). We thank Angel Sandoval Alvarez (Institut de Ciències Fotòniques, ICFO, Barcelona) for sample preparation and Stefan Balint (Institut de Ciències Fotòniques, ICFO, Barcelona) for providing super-resolution data for analysis. We acknowledge the ICFO Super-resolution Light Nanoscopy(SLN) facility for providing support with STORM imaging using the commercial Nikon STORM microscope.
- Hell SW: Far-field optical nanoscopy. Science 2007, 316: 1153. 10.1126/science.1137395View Article
- Bates M, Huang B, Zhuang X: Super-resolution microscopy by nanoscale localization of photo-switchable fluorescent probes. Curr Opin Chem Biol 2008, 12: 505. 10.1016/j.cbpa.2008.08.008PubMed CentralView Article
- Fernandez-Suarez M, Ting AY: Fluorescent probes for super-resolution imaging in living cells. Nat Rev Mol Cell Biol 2008, 9: 929. 10.1038/nrm2531View Article
- Patterson G, Davidson M, Manley S, Lippincott-Schwartz J: Superresolution imaging using single-molecule localization. Annu Rev Phys Chem 2010, 61: 345. 10.1146/annurev.physchem.012809.103444PubMed CentralView Article
- Dempsey GT, Vaughan JC, Chen KH, Bates M, Zhuang X: Evaluation of fluorophores for optimal performance in localization-based super-resolution imaging. Nat Methods 2011, 8: 1027. 10.1038/nmeth.1768PubMed CentralView Article
- Bates M, Huang B, Dempsey GT, Zhuang X: Multicolor super-resolution imaging with photo-switchable fluorescent probes. Science 2007, 317: 1749. 10.1126/science.1146598PubMed CentralView Article
- Rust MJ, Bates M, Zhuang X: Sub-diffraction-limit imaging by stochastic optical reconstruction microscopy (STORM). Nat Methods 2006, 3: 793. 10.1038/nmeth929PubMed CentralView Article
- Betzig E, Patterson GH, Sougrat R, Lindwasser OW, Olenych S, Bonifacino JS, Davidson MW, Lippincott-Schwartz J, Hess HF: Imaging intracellular fluorescent proteins at nanometer resolution. Science 2006, 313: 1642. 10.1126/science.1127344View Article
- Smith CS, Joseph N, Rieger B, Lidke KA: Fast, single-molecule localization that achieves theoretically minimum uncertainty. Nat Methods 2010, 7: 373. 10.1038/nmeth.1449PubMed CentralView Article
- Wolter S, Schuttpelz M, Tscherepanow M, Van de Linde S, Heilemann M, Sauer M: Real-time computation of subdiffraction-resolution fluorescence images. J Microsc 2010, 237: 12. 10.1111/j.1365-2818.2009.03287.xView Article
- Henriques R, Lelek M, Fornasiero EF, Valtorta F, Zimmer C, Mhlanga MM: QuickPALM: 3D real-time photoactivation nanoscopy image processing in ImageJ. Nat Methods 2010, 7: 339. 10.1038/nmeth0510-339View Article
- York AG, Ghitani A, Vaziri A, Davidson MW, Shroff H: Confined activation and subdiffractive localization enables whole-cell PALM with genetically expressed probes. Nat Methods 2011, 8: 327. 10.1038/nmeth.1571PubMed CentralView Article
- Cheezum MK, Walker WF, Guilford WH: Quantitative comparison of algorithms for tracking single fluorescent particles. Biophys J 2001, 81: 2378. 10.1016/S0006-3495(01)75884-5PubMed CentralView Article
- Wolter S, Endesfelder U, van de Linde S, Heilemann M, Sauer M: Measuring localization performance of super-resolution algorithms on very active samples. Opt Express 2011, 19: 7020. 10.1364/OE.19.007020View Article
- Jones SA, Shim SH, He J, Zhuang X: Fast, three-dimensional super-resolution imaging of live cells. Nat Methods 2011, 8: 499. 10.1038/nmeth.1605PubMed CentralView Article
- Bates M: Jones S A, Zhuang X. In Imaging: A Laboratory Manual. Edited by: Yuste R. New York, Cold Spring Harbor Laboratory Press; 2010.
- Huang B, Wang W, Bates M, Zhuang X: Three-dimensional super-resolution imaging by stochastic optical reconstruction microscopy. Science 2008, 319: 810. 10.1126/science.1153529PubMed CentralView Article
- Cella Zanacchi F, Lavagnino Z, Perrone Donnorso M, Del Bue A, Furia L, Faretta M, Diaspro A: Live-cell 3D super-resolution imaging in thick biological samples. Nat Methods 2011, 8: 1047. 10.1038/nmeth.1744View Article
- Abraham AV, Ram S, Chao J, Ward ES, Ober RJ: Quantitative study of single molecule location estimation techniques. Opt Express 2009, 17: 23352. 10.1364/OE.17.023352PubMed CentralView Article
- Huang F, Schwartz SL, Byars JM, Lidke KA: Simultaneous multiple-emitter fitting for single molecule super-resolution imaging. Biomed Opt Express 2011, 2: 1377. 10.1364/BOE.2.001377PubMed CentralView Article
- Zhu L, Zhang W, Elnatan D, Huang B: Faster STORM using compressed sensing. Nat Methods 2012, 9: 721. 10.1038/nmeth.1978PubMed CentralView Article
- Babcock H, Sigal YM, Zhuang X: A high-density 3D localization algorithm for stochastic optical reconstruction microscopy. Optical Nanoscopy 2012, 1: 6. 10.1186/2192-2853-1-6View Article
- Huang B, Jones SA, Brandenburg B, Zhuang X: Whole-cell 3D STORM reveals interactions between cellular structures with nanometer-scale resolution. Nat Methods 2008, 5: 1047. 10.1038/nmeth.1274PubMed CentralView Article
- Dani A, Huang B, Bergan J, Dulac C, Zhuang X: Superresolution imaging of chemical synapses in the brain. Neuron 2010, 68: 843. 10.1016/j.neuron.2010.11.021PubMed CentralView Article
- Bates M, Dempsey GT, Chen KH, Zhuang X: Multicolor super-resolution fluorescence imaging via multi-parameter fluorophore detection. Chemphyschem 2011, 13: 99.PubMed CentralView Article
- Lakadamyali M, Babcock H, Bates M, Zhuang X, Lichtman J: 3D Multicolor Super-Resolution Imaging Offers Improved Accuracy in Neuron Tracing. PLoS One 2012, 7: e30826. 10.1371/journal.pone.0030826PubMed CentralView Article
- Shtengel G, Galbraith JA, Galbraith CG, Lippincott-Schwartz J, Gillette JM, Manley S, Sougrat R, Waterman CM, Kanchanawong P, Davidson MW, Fetter RD, Hess HF: Interferometric fluorescent super-resolution microscopy resolves 3D cellular ultrastructure. Proc Natl Acad Sci U S A 2009, 106: 3125. 10.1073/pnas.0813131106PubMed CentralView Article
- Shim SH, Xia C, Zhong G, Babcock HP, Vaughan JC, Huang B, Wang X, Xu C, Bi GQ, Zhuang X: Super-resolution fluorescence imaging of organelles in live cells with photoswitchable membrane probes. Proc Natl Acad Sci U S A 2012, 109: 13978. 10.1073/pnas.1201882109PubMed CentralView Article
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License ( http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.