two-photon imaging

A Plug-in Server

Scanbox has a memory mapped mechanism to share the incoming data stream with other processes that intend to consume the data in real-time.

Examples of simple uses of such data sharing include the calculation of rolling averages, displaying a real-time histogram, or generating a montage display for volumetric data.

Of course, different experiments may need to be processed by different plug-ins.  To simplify this process we now provide a single mechanism that allows the selection of a plug-in via a pull-down menu in the Scanbox GUI.

The configuration variable “plugin” can be used to list a number of different options to call.  In the example below, two possible options are listed.

sbconfig.plugin = {'rolling','montage'}; %plugin options

These must be names of Matlab scripts that will be called to process the data.

When Scanbox starts it will list these options in a pull down menu within the real-time processing panel, along with two check-boxes, one labeled “plugin” and another labeled “focus”.


The pull down menu allows the user to select one plugin among the available selections.

The plug-in checkbox indicates you want the data to be shared via the memory mapped mechanism, and the “focus” checkbox indicates whether you want the data to be shared not only during a grabbing operation, but also when you are focusing.

For example, adjusting the laser intensity by analyzing the histogram of values in the incoming data may be something you want to do ahead of an actual experiment, and something you will want to run while focusing.

Now, when you start Scanbox a new Matlab instance is opened and calls the sbxplugin_server.m script (assuming mmap=true in the config file).  This is a simple script that waits for new data to arrive and calls the selected plug-in.

% Scanbox plug in server script
% This code will run upon startup of Scanbox on a separate matlab instance if mmap=true
% in the scanbox_config file

% Open memory mapped file -- define just the header first

scanbox_config; % read the scanbox config structure

mmfile = memmapfile('scanbox.mmap','Writable',true, ...
 'Format', { 'int16' [1 16] 'header' } , 'Repeat', 1);
flag = 1;

disp('Plugin server ready');

% Process all incoming frames until Scanbox stops

running = false;


 disp('Waiting for imaging data stream');

 while ~running
 running = mmfile.Data.header(1)>=0;

 plugin_id = mmfile.Data.header(7); % get the plug in to be used

 fprintf('Plugin id=%d\n',plugin_id);
 fprintf('ROI=%d Expt=%d\n',mmfile.Data.header(8),mmfile.Data.header(9));

 while running

 running = (mmfile.Data.header(1) ~= -2); % no stop signal present

 if running && mmfile.Data.header(1)>=0 % if not stopped and frame present

 switch plugin_id

 case 1
 % replace the line below with your own script for
 % plugin with id #1
 fprintf('Frame: %05d Plugin: %d\n',mmfile.Data.header(1),plugin_id);

 case 2
 % replace the line below with your own script for
 % plugin with id #2
 fprintf('Frame: %05d Plugin: %d\n',mmfile.Data.header(1),plugin_id)

 disp('Invalid plugin id number');

 mmfile.Data.header(1) = -1; % signal Scanbox that frame has been consumed!


clear(mmfile); % close the memory mapped file
close all; % close all figures

The default server code simply lists number of of the incoming frame and the selected plug-in.  Just add an eval() call to the script name at the indicated locations.

The output of the script, as is, will look like:

Waiting for imaging data stream
Plugin id=2
ROI=0 Expt=0
Frame: 00000 Plugin: montage
Frame: 00001 Plugin: montage
Frame: 00002 Plugin: montage
Frame: 00003 Plugin: montage
Frame: 00004 Plugin: montage
Frame: 00005 Plugin: montage
Frame: 00006 Plugin: montage
Frame: 00007 Plugin: montage
Frame: 00008 Plugin: montage
Frame: 00009 Plugin: montage
Frame: 00010 Plugin: montage
Waiting for imaging data stream

The user can now easily switch between plugins from the Scanbox window by changing the selection in the pull down menu.


Latest Scanbox Release brings cool performance improvements

The latest Scanbox release brings a major performance improvement.  The main bottleneck in the acquisition loop consists of a function that re-arranges the incoming pixel stream into the images that are displayed and stored in real-time. This function now makes use of OpenMP to distribute the work among different physical cores available in your machine and thereby improving the performance in your machine.

Performance improvements should be reflected in your ability to scan at higher rates in both unidirectional and bidirectional modes without running into a buffer overflow fatal message.

The number of cores used in unidirectional and bidirectional modes are variables in the scanbox_config file.  By default they are both set to the actual number of physical cores in your machine:

sbconfig.phys_cores = uint16(feature('numCores')); % total number of physical cores
sbconfig.cores_uni = sbconfig.phys_cores; % number of cores in unidirectional scanning
sbconfig.cores_bi = sbconfig.phys_cores; % number of cores in bidirectional scanning

A second major change is the addition of more options for magnification, which now default to factors ranging from 1 to 16 in logarithmic steps, 4 steps per octave. Needless to say, be sure to reduce your laser power setting when using magnifications larger than 4.


These settings can be changed in the configuration file by means of two variables:

sbconfig.gain_galvo = logspace(log10(1),log10(16),13); % more options now!
sbconfig.gain_resonant = 1.4286 * sbconfig.gain_galvo;

If you want to customize your settings, please follow the instructions here.

The latest release requires the update of the firmware, which you can do by following the second part of the post here.

There are two associated variables, nroi_auto_size and ncolbi that need to be changed so they include entries for all of the magnification settings.

Finally, the PMT gain control panel has been simplified by removing the enable checkbox and zero gain button, as their behavior was causing some confusion. The behavior now is as follows.  If the microscope is not scanning the PMT gains are always zero.  Only when the microscope is scanning the PMT sliders are enabled and the PMT gains are dictated by their values.  Moreover, upon initialization and shutdown Scanbox enforces gains to be zero — as a defensive mechanism.

Note: Please save the latest working version of your Scanbox software so you can go back to it in case you run into any issues with this new distribution.  Also, please note that Scanbox no longer supports the 3D mouse. You have to use Knobby from now on.  Finally, make sure you install Visual C++ Redistributable 2015 which is required for this update as well.

A plug-in for real-time histograms in Scanbox

A Scanbox user requested to add the ability to display a histogram of image values in real-time.

Here, I use a the general mechanism of memory mapped files in Scanbox to show how easy is to add such “plug-ins” to provide additional functionality to the system.

The Matlab code necessary to add real-time histograms in Scanbox is very concise and commented below:

% Display real-time histogram in Scanbox

% Prepare the figure for display

close all;
h.MenuBar = 'None';
h.NumberTitle = 'off';
h.Name = 'Scanbox ChA real time histogram';

% Open memory-mapped file to have access to image stream

mmfile = memmapfile('scanbox.mmap','Writable',true, ...
'Format', { 'int16' [1 16] 'header' } , 'Repeat', 1);

% Some parameters for display

nframes = 5; % how often we want to display histogram
nbins = 128; % how many bins in the histogram
margin = 20; % size of the margins to crop around the central ROI

% Start processing until Scanbox stops...

k = 0;

flag = 1; % Necessary to set image format the first time


   while(mmfile.Data.header(1)<0) % wait for a new frame to arrive
      if(mmfile.Data.header(1) == -2) % exit if Scanbox finished acquiring data

   if(flag) % First time? Format chA according to lines/columns in header
      mmfile.Format = {'int16' [1 16] 'header' ; ...
      'uint16' double([mmfile.Data.header(2) mmfile.Data.header(3)]) 'chA'};
      flag = 0;

   if(mod(k,nframes==0)) % update histogram every k frames
      imhist(intmax('uint16')-mmfile.Data.chA(margin:end-margin,margin:end-margin),nbins); % display histogram

   mmfile.Data.header(1) = -1; % signal Scanbox that frame has been consumed!
   k = k+1;

clear(mmfile) % close memory-mapped file

That’s all…

Check the mmap configuration variable is 1 and start Scanbox.

Open a new Matlab process and run the script above.

Start scanning and you should see a real-time histogram being displayed, each of the frames will look like this:


You can add other features to this script, such as displaying the number of pixels saturated, and so on.

Importantly, notice that the memory-mapped mechanism can easily be accessed by plug-ins written in other languages, like Python, so you are not bound to use Matlab.

If you develop useful real-time plug-ins for Scanbox let us know and we will include them as part of the future releases.


Installing knobby

Many of you will be receiving Knobby in the mail soon.

Installing it is easy.  Just follow these steps:

  1. Connect knobby to a USB port
  2. Open the Device Manager and check which COM port # knobby was assigned to (it is the one displaying an Arduino Due board).
  3. Open Matlab, edit scanbox_config.m and set the tri_knob variable to the corresponding COM port.  Save.
  4. Start Scanbox.


As Scanbox starts up you will see a welcome message on the screen and Knobby will be initialized.  Test that all axes work properly. Finally, uninstall the 3Dconnexion software and reboot your machine.  You are done!

Let me know if you have questions…

Memory-mapped data-sharing for real-time processing

To enhance the real-time processing capabilities of Scanbox we now provide a general mechanism to share frames being collected with another application running concurrently on the same computer via a memory mapped file.

This general and flexible mechanism allows any user to extend the real-time processing of Scanbox beyond what is already offered.

As an example, this interface allows one to perform pixel-by-pixel image processing in ways that do not require prior definitions of ROIs. One such case would be the calculation of temporally encoded cortical maps by on-the-fly calculations of the magnitude/phase of each pixel.

All you have to do to enable this feature is to set the “mmap” variable to 1 in the configuration file and, in those imaging sessions you want to stream the data, enable the “Memory map” checkbox in the real-time processing panel:


The memory mapped file contains a 16 word header (int16s) that allows communication between Scanbox and another process, which is then followed by the imagery data.  Only the first 4 words of the header are used at the moment (the other are reserved for future use).

The first word is used as a semaphore between the applications to control access to the memory mapped file.  The value of this variable is -1 if the file can be used by Scanbox to stream data, it is -2 if Scanbox has stopped imaging, and it is a non-negative number that corresponds to a frame if Scanbox has copied the data to the memory mapped file.

The second and third variables represent the size of the frame, used by the application processing the data to interpret the organization of the file as an image.

The fourth variable is a copy of the TTL1, which is high during a stimulus presentation and low otherwise.

As an example of a Matlab process that consumes data from Scanbox and all it does is compute the average frame is listed below.

% Simple example of memory mapped processing in Scanbox

% Open memory mapped file -- just header first
mmfile = memmapfile('scanbox.mmap','Writable',true, ... 
 'Format', { 'int16' [1 16] 'header' } , 'Repeat', 1);
flag = 1;

% Process all incoming frames until Scanbox stops


   while(mmfile.Data.header(1)<0) % wait for a new frame...
     if(mmfile.Data.header(1) == -2) % exit if Scanbox stopped
 display(sprintf('Frame %06d',mmfile.Data.header(1))) % print frame# being processed
 if(flag) % first time? Format chA according to lines/columns in data
   mmfile.Format = {'int16' [1 16] 'header' ; ... 
   'uint16' double([mmfile.Data.header(2) mmfile.Data.header(3)]) 'chA'};
   mchA = double(mmfile.Data.chA);
   flag = 0;
 mchA = mchA + double(mmfile.Data.chA);
 % Here you can do whatever you want with
 % your data.... I am just accumulating.
 mmfile.Data.header(1) = -1; % signal Scanbox that frame has been consumed!


imagesc(mchA); % display mean image

The interface allows you to use any language that provides you access to memory mapped files, such as Python.

So, if you happen to write some neat real-time image processing module for Scanbox, feel free to share!

A set of more readable functions that encapsulate the basic functionality shown above will be provided soon to make the writing of real-time processing applications easier, and we may also adopt a double-buffering or ring-buffer mechanisms.

Adding an SLM path for optogenetics

As promised, there has been some progress in adding an SLM path for optogenetics to the microscope and things are looking good…

We are now calling on potential users to provide input into the features that they would need/want in a GUI interface, as this may impact how useful this addition will be to our own work.

At the moment here is our basic plan:

  • Provide a tool to align the SLM and two-photon imaging paths.  This will consist in the user using the camera port to acquire an image of the scan area onto a chroma slide.  The system will then flash a number of single spots on a different slide and acquire their positions to compute the best affine transformation between the SLM image plane and the tw0-photon image.  Calibrations will be required just before the beginning of each imaging session.
  • Once a sample is imaged, provide cell-selection tool to obtain the coordinates of N desired locations in the imaging plane.
  • After cell selection a Matlab-based server will use the calibration and and selected points to accept network commands that describe the desired intensity at each of the N locations and the duration of the pattern. After the optimal phase is computed on the GPU it will be presented to the SLM for the specified duration and the start/end time of the presentation will be logged by the microscope through an event line in Scanbox.

So present and future Scanbox users…  is this a reasonable starting point? As it is, the system would be limited to N (as large as you want) points of different intensities.

Please add  your comments and/or suggestions below.  This is your opportunity to have an input into the design of the SLM path….  don’t miss it!

Bidirectional scanning in Scanbox

We have implemented bidirectional scanning to Scanbox. To switch between bidirectional and unidirectional scanning all you have to do is hit the toggle button within the “Scanner” panel.

The defaubidirectionallt setting is “U”, standing for unidirectional scanning; hitting the button will switch it to “B”, indicating bidirectional scanning.  You can only switch between the two modes while the microscope is not scanning.

In bidirectional mode scanning samples are obtained while the laser scans in both directions of motion of the resonant mirror, and lines are advanced at both edges of the scan. On the plus side, this means that you can scan the same area twice as fast. You will notice the frame rate box doubling its value when you switch from unidirectional scanning to bidirectional scanning.

However, bidirectional scanning has some drawbacks as well.  First, you should be aware that that the Pockels’ cell will be active during both phases of the scan, while in unidirectional scanning it was active only in one direction. The result is that you will be delivering twice the amount of power onto the tissue, which may limit the length of your experiments due to potential photo-damage. Thus, use bidirectional scanning only if your indicators are fast enough and the scientific question you need to answer requires doubling the scanning speed.

Second, in bidirectional scanning even and odd lines are sampled in different directions, so they should be aligned properly to yield an image without spatial discontinuities. This alignment has to be fine tuned by the user in each setup, as there is variability in the resonant frequency of mirrors, the laser pulse frequency, and the phase of the sync pulse from the resonant mirror controller.

Scanbox allows for this fine tuning by changing a single variable in the configuration file named ‘ncolbi’ — a vector that allows alignment to be fined tuned separately for the three magnification factors.  This is necessary as well because the resonant frequency changes slightly with amplitude.

Once properly tuned, Scanbox performs compensation for non-uniform sampling and the alignment of even/odd lines during bidirectional scanning in real time.  Below is the mean image of 100 frames collected under bidirectional scanning.  You can see the alignment is very good by looking at the fine vertical processes in the image.

Alignment of even/odd lines in bidirectional scanning t 31 frames per second.

A final drawback of bidirectional scanning is that your file size will roughly double. This means that all other processing will also take longer — from aligning the images to extracting the signals.

Having said this, the faster rates achieved in bidirectional scanning will be helpful when coupled with the ability to do volumetric imaging by means of the optotune lens.

The bidirectional mode of Scanbox is now in beta testing.  I have not fully evaluated the extent to which other features of Scanbox, such as real time processing and image stabilization work in bidirectional mode. I welcome feedback from brave users who are willing to test bidirectional mode in their setups.

If you plan on giving bidirectional scanning a try let me know how it goes!  The updated code will be in Scanbox Github shortly.

Real-time, motion compensation in Scanbox

Our colleague Tobias Rose was recently asking about Scanbox’s ability to stabilize the motion of images and signals from regions-of-intrest (ROIs) in real-time.  The goal of such processing is to be able to do experiments in closed-loop, and do very quick analyses on the neuronal responses, such as computing tuning curves on the fly.

Below is a simple demonstration of automatic stabilization in Scanbox, where the microscope is tracking the activity on a few ROIs the size of a single cell body. In the first pass of the ROI signals displayed at the bottom motion compensation is off.  The traces are noisy as the cell bodies themselves move with respect to the defined ROI.  The second pass shows the same image sequence once image stabilization is engaged (when the checkbox on the lower left is clicked), the images appear much more stable in space and, correspondingly, the signals for the ROIs are smoother and the resulting SNR is higher.

Incidentally, the demo also shows another feature of Scanbox — the ability to play back already collected data in a loop, so people can  learn the features of the microscope and try things out without the need of a living, animal subject.

Real-time signal extraction, visualization and processing in scanbox 2.0

Here is a sneak preview of the new features to be released with Scanbox 2.0.

Some of the salient additions include:

Automatic stabilization: The system can automatically correct for rigid (x,y) translation in real time.

Selection of regions of interest (ROIs): Allows for the selection of regions of interest that need to be tracked in real-time.  After their definition, the polygons can be edited by translating them or individually moving their vertices.  The ROIs calculation can be used in conjunction with automatic stabilization.

Real-time calculation of ROI traces: Scanbox computes the mean image (other statistics also possible) in each ROI and displays its z-scored version it in real time on a rolling graph, where the vertical, dashed red line displays the present time.

Stimulus onset/offset marking: The beginning and end of stimuli are displayed on the trace graph by a background of a light blue color, allowing one to easily verify the cells are responding to a stimulus.

On-line processing: The ROI trace data can be mapped into analog output channels, streamed over the network, or streamed to the disk for further on-line processing.  So be ready to compute your tuning curves on the fly!

Here is a snapshot of how the GUI is looking at the moment:

Scanbox real time

Here is a movie without actual cells, but showing the ability to mark the location of the stimuli and the analog output.  The signal is generated by waving my finger near the front of the objective.


If you need any other features, please let your suggestion below (no, the system will not generate the figures and write the paper.)