Mesoscope features: real-time panoramas, ROI selection, zoom/pan, and dynamic ROIs

Here are some of the features we have been developing to make it easy to build panoramas, pick regions-of-interest, and to visualize ROI location and data during acquisition using the NLW Mesoscope (aka the Kraken) on the latest version of Scanbox.

The video below shows a brief demo of the following:

  1. The automatic creation of panoramas on-the-fly as data are collected.
  2. Selection of region-of-interests (ROIs) directly on the panorama view.
  3. Visualizing ROI data embedded in the panorama.
  4. Dynamically zooming and panning during data collection.
  5. Dynamically changing the location of ROIs by dragging them on the panorama during the collection of data.

Here are some raw data from two separate ROIs, one on each hemisphere.

The ROIs (full panorama FOV in this example is 4.0 x 3.5 mm)


And a short movie with raw data:

Thinking about getting a mesoscope?  Add your wish list of features in the comments below.  Better now than later, as some features can dictate design considerations.



Finding the optimal sample clock phase

Some time ago we talked about the advantages of synchronizing the digitizer sample clock to the laser.  Our new system incorporates an on-board PLL multi-phase clock that allows Scanbox to automatically measure the contrast in a test image as a function of sample phase delay and select the optimal value for your setup.

A typical measurement showing the change in normalized contrast of a target image as a function of phase delay (here the range from -8 to 8 covers the entire 12.5 ns period of the laser).


Scanbox automatically shows the resulting images, on the same scale, for each phase delay value as well:


Scanbox will also plot the normalized images obtained for the settings that yield the lowest and and highest contrasts:


One can clearly see from the images that phase does not simply scale the contrasts of the images, but has an obvious effect on their SNR.  (Why this occurs exactly is still a matter of debate here, but data rule and the results are clear.)

This option will be automatically available to those who adopt the new tower system.  What can you do if you have one of the older systems?  You can simply change the sample clock phase by extending the length of the cable running from the laser SYNC OUT to the external clock of the digitizer.  Extensions of 50cm in length can be connected together to yield phase steps of 1/8th of the laser period.  So if you want to optimize the contrast and SNR of your images take a day off to find your optimal delay and improve the quality of your data.

Intrinsic & epi-fluorescence imaging using the port camera

Some colleagues have asked about the possibility of integrating imaging acquisition from the port camera to do intrinsic and/or epi-fluorescence imaging ahead of targeting a region of interest with two-photon experiments in Scanbox.

We have now added this option to Scanbox.

To use this  option you will need to wire the I2C port on the faceplate of Scanbox.  As nobody seems to be making use of I2c sensors we changed the functionality of these pins.

You will first need a camera with an output TTL signal that provides a rising edge on each frame. Most advanced cameras provide such an option.  That signal needs to be connected to pin #2 (second from the left).  We also need a synchronization signal from our visual stimulus, so we know on which frames it is presented.  A rising edge on pin #3 will be timestamped by the system with the frame number when it occurred.

That’s all you need in terms of hardware connections.

Now, you will find an extra line in the scanbox_config.m file that allows you to select the format you want to use for the port camera.  Some cameras have an 8-bit depth format that is convenient and sufficient if all you are using the camera for is to navigate around the sample. For imaging, you will probably want to use formats with pixel depths ranging from 12 to 16.  You can use Matlab’s imaqtool to see what formats are available for your camera.  Set the pathcamera_format variable to a string reflecting your selection.  For example:

sbconfig.pathcamera_format = 'Mono14';

In Scanbox, you will see a new panel which, at the moment, is sparsely populated with only 3 buttons, but should be enough to get things going.  Two buttons behave in the same way as for the eye and ball cameras: they allow you to define a ROI that will be saved.  The third button “Grab” allows you to manually start and stop the acquisition.  You can only use this button when the port camera is active. Otherwise, you will get a message complaining about it. When the port camera is active, sending a command to start and stop sampling will engage this button as well.  So the same experimental scripts you are now using for two-photon imaging can be used for imaging with the camera port without any change at all.

The data are stored in two separate files, one that contains the images themselves and the other containing the TTL data from the visual stimulation.  To read them, there are two separate functions: sbx_get_ttlevents() and sbx_read_camport_frame().

The first function takes the file name as input and simply returns a list of frame numbers during which a rising edge was present in the “stimulation” input on the I2C connector above.

So, for example:

>> ttl = sbx_get_ttlevents('xx0_111_222')'

ttl =

    200  240  280  320  360  400

This means the onset of six stimuli occurred during those frames.

The second function takes the file name and a vector of frame numbers.  It returns a volume of data where the third dimension corresponds to the selected frames.

So, for example:

>> data = sbx_read_camport_frame('xx0_111_222',ttl(1)-10:ttl(1)+10);

Will read the volume from ten frames before the onset of stimulus #1 to 10 frames after.

As things are now you have to control illumination externally.  We will work to integrate illumination and histograms of ROI values soon.

New alignment and segmentation tools

Improved alignment and segmentation tools have now been released in the latest version of Scanbox, while retaining much of the functionality of the last version.

sbxaligntool. The new alignment tool, shown below, adds batch processing of files, including the processing of eye and ball motion if those data are present.  A region-of-interest (ROI) can optionally be selected manually or automatically.  For file entries where manual selection was specified, the program will stop and present a rectangle on the screen for the user to specify the ROI.  Typically, automatic ROI works fine, and it does not require the user to stand by the computer to specify the ROI each time a new file starts to process.


As the files are aligned, the Status column and Status message will display the progress. The alignment procedure can also be visualized by clicking the Live update checkbox, which will display the mean of the entire image stack as the process moves along.  Pan and Zoom buttons allow the user to inspect details in the live image, such as fine branches, as the system is carrying out the alignment. This tool performs rigid alignment and the result is stored in a *_rigid.sbx file.  The original data is left untouched. The tool can align images relatively fast (about 65 frames/sec in my computer), but it will take a few minutes to compute the reference image if the sequence is 15 min or more (please be patient). Alignment improves with the number of passes requested.  Usually one pass is very good, but you can try two or more passes by changing the appropriate entry in the column. The alignment algorithm has been improved.

sbxsegmenttool. The segmentation tool works in a similar way as before. After loading the aligned *_rigid.sbx file, it will display the correlation map.  Segmentation then proceeds as in the previous version.


Once a number of cells are selected, you must save the segmentation and then extract the signals by pressing the corresponding buttons. After the signals are extracted you can select a cell with the pull down menu on the bottom left and the traces corresponding to that cell (now highlighted in green) will be displayed.  The blue trace represents the average signal within the cell, the gray trace is the neuropil, and the trace is the estimated spike rate using the Vanilla algorithm with parameters optimized for GCaMP6f.

Improvements include an Undo button, which will remove the last cell segmented. The ability to load a previous segmentation (it will load automatically after you select the *_rigid.sbx file), to continue adding cells to it.  The ability to define a ROI in the correlation map to automatically increase the contrast of the correlation map as the most salient cells are selected. A zoomed version of the mean image on the right to go along with the correlation map.  And the tool now saves the neuropil and deconvolved signal as well.

Give these tools a try. Report back any suggestions for improvements or problems you encounter.

Latest Scanbox Release brings cool performance improvements

The latest Scanbox release brings a major performance improvement.  The main bottleneck in the acquisition loop consists of a function that re-arranges the incoming pixel stream into the images that are displayed and stored in real-time. This function now makes use of OpenMP to distribute the work among different physical cores available in your machine and thereby improving the performance in your machine.

Performance improvements should be reflected in your ability to scan at higher rates in both unidirectional and bidirectional modes without running into a buffer overflow fatal message.

The number of cores used in unidirectional and bidirectional modes are variables in the scanbox_config file.  By default they are both set to the actual number of physical cores in your machine:

sbconfig.phys_cores = uint16(feature('numCores')); % total number of physical cores
sbconfig.cores_uni = sbconfig.phys_cores; % number of cores in unidirectional scanning
sbconfig.cores_bi = sbconfig.phys_cores; % number of cores in bidirectional scanning

A second major change is the addition of more options for magnification, which now default to factors ranging from 1 to 16 in logarithmic steps, 4 steps per octave. Needless to say, be sure to reduce your laser power setting when using magnifications larger than 4.


These settings can be changed in the configuration file by means of two variables:

sbconfig.gain_galvo = logspace(log10(1),log10(16),13); % more options now!
sbconfig.gain_resonant = 1.4286 * sbconfig.gain_galvo;

If you want to customize your settings, please follow the instructions here.

The latest release requires the update of the firmware, which you can do by following the second part of the post here.

There are two associated variables, nroi_auto_size and ncolbi that need to be changed so they include entries for all of the magnification settings.

Finally, the PMT gain control panel has been simplified by removing the enable checkbox and zero gain button, as their behavior was causing some confusion. The behavior now is as follows.  If the microscope is not scanning the PMT gains are always zero.  Only when the microscope is scanning the PMT sliders are enabled and the PMT gains are dictated by their values.  Moreover, upon initialization and shutdown Scanbox enforces gains to be zero — as a defensive mechanism.

Note: Please save the latest working version of your Scanbox software so you can go back to it in case you run into any issues with this new distribution.  Also, please note that Scanbox no longer supports the 3D mouse. You have to use Knobby from now on.  Finally, make sure you install Visual C++ Redistributable 2015 which is required for this update as well.

Improvements in alignment and segmentation tools

Two GUIs to align (sbxaligntool) and segment (sbxsegmenttool) two-photon imaging data from Scanbox were recently updated.

Sbxaligntool allows users to perform alignment using rigid translation or a non-rigid deformation of the image stack. The tool now uses all the cores available in your computer along with the GPU.  Rigid alignment can take place at near 50 fps for 512 x 796 pixel images collected at 15.5 fps.  In other words, rigid alignment runs at more that 3 times the speed of data collection.


Non-rigid alignment is a bit slower. Still, running at 22 fps the system aligns images 40% faster than during acquisition.  In my system configuration the bottleneck of rigid alignment is in the disk system.   The bottleneck of nonrigid alignment is on the computation side, which could therefore be sped up by additional cores.


The segmentation tool also got a few improvements.  A pull-down menu provides the ability to zoom/pan within the image during segmentation.  The GUI now includes two different methods for segmentation–  region growing as described earlier, and a new SVD+kmeans automatic segmentation method.  You can switch between the two in a single segmentation session.  The Nhood variable determines the size of the local neighborhood used to perform the SVD+kmeans clustering.  At the end of the segmentation, after clicking Save and Extract, the extracted signals will be shown on the bottom of the window providing some qualitative assessment of the quality of the recording.


We will continue to improve these tools as we get user feedback.  So please, send bugs/suggestions our way. We are already working on extracting neuropil data and some basic post-processing (de-trending and deconvolution).


Non-rigid image alignment in twenty lines of Matlab

We have previously discussed rigid and non-rigid image alignment algorithms for calcium imaging data. If you have the image processing toolbox, a particularly compact solution for non-rigid image registration can be written in ~20 lines of code or so.  The code below returns a displacement field for each frame in the image sequence (output variable ‘disp’) that needs top be used to align the image.  The mean image after alignment is returned as well (output variable ‘m’).  The strategy follows the same recursive approach we used previously for rigid alignment.  The main Matlab function doing the calculation is imregdemons.

function [m,disp] = sbxalign_nonrigid(fname,idx)

   A = squeeze(sbxread(fname,idx(1),1)); % just one frame... easy!
   m = A;
   disp = {zeros([size(A) 2])};
elseif (length(idx)==2) % align two frames
   A = squeeze(sbxread(fname,idx(1),1)); % read the frames
   B = squeeze(sbxread(fname,idx(2),1));
   [D,Ar] = imregdemons(A,B,[32 16 8 4],'AccumulatedFieldSmoothing',2.5,'PyramidLevels',4,'DisplayWaitBar',false);
   m = (Ar/2+B/2);
   disp = {D zeros([size(A) 2])};
   idx0 = idx(1:floor(end/2)); % split dataset in two
   idx1 = idx(floor(end/2)+1 : end); % recursive alignment
   [A,D0] = sbxalign_nonrigid(fname,idx0);
   [B,D1] = sbxalign_nonrigid(fname,idx1);
   [D,Ar] = imregdemons(A,B,[32 16 8 4],'AccumulatedFieldSmoothing',2.5,'PyramidLevels',4,'DisplayWaitBar',false);
   m = (Ar/2+B/2);
   D0 = cellfun(@(x) (x+D),D0,'UniformOutput',false); % concatenate distortions
   disp = [D0 D1];

Once the displacement field is computed one can obtain the aligned images by applying the distortion to each frame individually using imwarp.

The example below shows the original (top) and stabilized sequences in one case where imaging at high magnification clearly leads to non-rigid deformations that need correction.

Calibrating the Optotune

If you have been using the Optotune, you must have noticed that the values displayed within the optotune panel for the setting of the slider and the parameters of the z-scanning waveform are unit-less and not very useful.

The latest update to Scanbox allows you to calibrate the Optotune and have all the units within the Optotune panel in micrometers.

A new configuration variable “optocal” in scanbox_config.m can be either empty or contain calibration data.  If optocal is empty, Scanbox behaves as it has done up to now — showing the unit-less values.

Start by setting optocal=[].

To calibrate simply take a green fluorescent slide and make a small marking on its surface.  Small dots with a fine, black sharpie should do.  Focus the microscope near the very top of the slide and pick one feature with the optotune slider set at zero.  Now, zero the position counters.  Our first point, corresponding to pairs of (values, depth) is (0,0).

Move the optotune slider up to a value near 200.  Then, compensate by lowering the objective (z-position) to bring back the feature you selected into focus. Once this is done, write down your next pair (value, depth), as the value of the slider and the depth you read in the position panel (or in Knobby). Continue this process, increasing the value of the optotune slider in steps of ~200, up to a value of ~2000.

Here is the set of measurements I obtained in my setup:


The curve saturates at a value of ~1760. This is because the current source has reached its maximum output voltage. The range of depths spanned by the optotune is ~340um (vertical axis). The smooth rising part of the curve, before saturation, is well approximated by a second-order polynomial (red curve).  To obtain the values of the coefficients use Matlab’s polyfit function by calling “polyfit(vals,depth,2)” (don’t include the saturating part of the curve!).  The coefficients of these polynomial are what the optocal variable should be set to.  In my case, I get the optimal coefficients to be [0.0001  0.0732 1.3162] and, therefore, I modified the config file so that optocal = [0.0001  0.0732  1.3162].


Once the optocal variable is set, you can restart Scanbox.  The result should be that all the values within the Optotune panel will read in microns (as shown on the left). This includes the slider and the parameters to the waveforms.  Moreover, as the function is slightly nonlinear, the actual waveforms will be linearized accordingly — so a linear ramp should very closely approximate a linear change in depth.


Scanbox Yeti (beta) has arrived!

Scanbox Yeti  is here!

Why Yeti?  I don’t really know… but it sounds nice.  Future versions will take the names of other mythical creatures as well.  We should credit Josh Trachtenberg for this intellectual contribution to the project.

We have been using Scanbox Yeti in the Lab for about a month without major problems and we are releasing this beta version so we can get feedback from other users, make sure it runs in all platforms, and find/fix potential problems.



Here is a list of the major changes:

Smaller file sizes: File sizes have been reduced by a factor of 6.3 with (effectively) no loss of information. Reading and writing files can be done quicker and you should experience noticable speedups.  Moreover, we have been streaming data directly to USB3 disks without problems, which means that people can bring the disks to the microscope, stream to them, and take the data to their offices for analysis.

Auto-stabilization mode: After collecting ~100 frames in a period of quiescence using the “Accumulate” mode under the “Image Display” panel, you can define a reference image in the “Real time processing” panel.  This will bring up small local windows that you should position around image areas that have two dimensional structure and do not change much over time (such as filled cells that do not show much activity or are close to saturation).  The number and size of these regions are stipulated by the variables nroi_auto and nroi_auto_size in the scanbox_config.m file.  Scanbox uses the motion estimates from these regions to compute a global, rigid translation that it uses to compensate to align the individual frames to the reference image.  This alignment is saved to an *.align file.  Auto-stabilization is engaged only after clicking the appropriate check-box.

Real-time signal extraction: By clicking the “Add” button under  the real-time processing panel one can add arbitrary polygon defining region of interests.  The average signal in these regions will be computed, displayed, and saved in a *real_time.mat file.  Once you set a reference image you can proceed to define ROIs by clicking on the image and defining the vertices of the polygon.  After the last vertex was placed just double-click to close the polygon.  The vertices can then be moved individually if you need further adjustment.  The new ROI will be placed on the left list of defined ROIs.  The right column contains the ROIs that are actually displayed when the microscope is collecting data.  You can move ROIs from one column to the other by selecting an entry and clicking the appropriate arrow between the columns.  Double arrows move all the ROIs in one column to the other.

Stimulus markers: A rectangle representing the onset and offset of a stimulus can be overlaid on top of the real-time traces.  All you need to do is to set up the signals as explained here and here, and click the “Stimulus marker” check box.

Network streaming: The real-time ROI data and stimulus markers can be streamed to the network for other computers to perform online processing or perform close-loop experiments.  A separate post will describe how set things up to achieve this.

Absolute position memory: You now have 4 absolute position memories for the microscope that include its 4 degrees of freedom (X, Y, Z and objective angle).  You can store the present position by clicking the ‘a’, ‘b’, ‘c’, ‘d’ buttons on the left column, and retrieve a position by clicking the matching buttons on the right column.  Remember that positions are reset if you power down the motor controller.

Automatic vertical alignment of the objective: The button XYZA=0 will make the position counters go to zero but will also move the objective to its true vertical position.  Additional wiring and programming of your motor control card is needed for this to work, so do not try to use before you do so.  A separate post will provide the necessary instructions. 

Force PMT gains to zero: A new button will force the PMT gains to zero even when the PMTs are disabled.

Laser power indicator: A circular counter with 32 steps indicates the relative position of the motor controlling the overall laser power.  It resets to zero when Scanbox restarts.

scanbox_config.m has a new home: The scanbox_config.m file now resides in the main scanbox/ directory, so this is the file you should edit to adapt the code to your environment.  Make sure to start by copying the same values you now have in C:\scanbox\scanbox_config.m to the new file, and just leave alone the variables that you cannot see in your file.

Bootloader: The card now starts up with a bootloader to allow the remote upgrading of its firmware.

Ready to download? Start by renaming your old scanbox directory (but do not remove it). Then download the new version from here (you need a password that should have been mailed to you) and unzip the contents into the $MATLAB/scanbox directory.  Make sure you clear all sub-directories within scanbox are in the Matlab path.

Problems/Questions?  Please leave a comment below so I can answer and others can have access to the information as well.  Please use the new set of processing functions within the sbx/ sub-directory to access and process data from Yeti, as the data format has changed.