real time processing

A Plug-in Server

Scanbox has a memory mapped mechanism to share the incoming data stream with other processes that intend to consume the data in real-time.

Examples of simple uses of such data sharing include the calculation of rolling averages, displaying a real-time histogram, or generating a montage display for volumetric data.

Of course, different experiments may need to be processed by different plug-ins.  To simplify this process we now provide a single mechanism that allows the selection of a plug-in via a pull-down menu in the Scanbox GUI.

The configuration variable “plugin” can be used to list a number of different options to call.  In the example below, two possible options are listed.

sbconfig.plugin = {'rolling','montage'}; %plugin options

These must be names of Matlab scripts that will be called to process the data.

When Scanbox starts it will list these options in a pull down menu within the real-time processing panel, along with two check-boxes, one labeled “plugin” and another labeled “focus”.

plugin

The pull down menu allows the user to select one plugin among the available selections.

The plug-in checkbox indicates you want the data to be shared via the memory mapped mechanism, and the “focus” checkbox indicates whether you want the data to be shared not only during a grabbing operation, but also when you are focusing.

For example, adjusting the laser intensity by analyzing the histogram of values in the incoming data may be something you want to do ahead of an actual experiment, and something you will want to run while focusing.

Now, when you start Scanbox a new Matlab instance is opened and calls the sbxplugin_server.m script (assuming mmap=true in the config file).  This is a simple script that waits for new data to arrive and calls the selected plug-in.

% Scanbox plug in server script
% This code will run upon startup of Scanbox on a separate matlab instance if mmap=true
% in the scanbox_config file

% Open memory mapped file -- define just the header first

scanbox_config; % read the scanbox config structure

mmfile = memmapfile('scanbox.mmap','Writable',true, ...
 'Format', { 'int16' [1 16] 'header' } , 'Repeat', 1);
flag = 1;

disp('Plugin server ready');

% Process all incoming frames until Scanbox stops

running = false;

while(true)

 disp('Waiting for imaging data stream');

 while ~running
 running = mmfile.Data.header(1)>=0;
 end

 plugin_id = mmfile.Data.header(7); % get the plug in to be used

 fprintf('Plugin id=%d\n',plugin_id);
 fprintf('ROI=%d Expt=%d\n',mmfile.Data.header(8),mmfile.Data.header(9));

 while running

 running = (mmfile.Data.header(1) ~= -2); % no stop signal present

 if running && mmfile.Data.header(1)>=0 % if not stopped and frame present

 switch plugin_id

 case 1
 % replace the line below with your own script for
 % plugin with id #1
 fprintf('Frame: %05d Plugin: %d\n',mmfile.Data.header(1),plugin_id);

 case 2
 % replace the line below with your own script for
 % plugin with id #2
 fprintf('Frame: %05d Plugin: %d\n',mmfile.Data.header(1),plugin_id)

 otherwise
 disp('Invalid plugin id number');
 end

 mmfile.Data.header(1) = -1; % signal Scanbox that frame has been consumed!

 end
 end
end

clear(mmfile); % close the memory mapped file
close all; % close all figures

The default server code simply lists number of of the incoming frame and the selected plug-in.  Just add an eval() call to the script name at the indicated locations.

The output of the script, as is, will look like:


Waiting for imaging data stream
Plugin id=2
ROI=0 Expt=0
Frame: 00000 Plugin: montage
Frame: 00001 Plugin: montage
Frame: 00002 Plugin: montage
Frame: 00003 Plugin: montage
Frame: 00004 Plugin: montage
Frame: 00005 Plugin: montage
Frame: 00006 Plugin: montage
Frame: 00007 Plugin: montage
Frame: 00008 Plugin: montage
Frame: 00009 Plugin: montage
Frame: 00010 Plugin: montage
Waiting for imaging data stream

The user can now easily switch between plugins from the Scanbox window by changing the selection in the pull down menu.

 

A rolling average plug-in for scanbox

Scanbox offers a memory-mapped interface mechanism to expose incoming data to other processes. This facilitates a wide rage of customization by users that may want to display their data in different ways or do some on-line processing not currently supported by Scanbox.

We previously provided one example of how a real-time histogram of the data can be generated. Today we offer a simple modification of this code to show a rolling average display, where frames are weighted with an exponential window, can be implemented.

These additional processes can run concurrently with the Scanbox live display if your computer is fast enough. To lighten the load on the computer you can now choose to disable the Scanbox display in the “Image Display” panel.

Remember that plug-ins run on a separate Matlab process and from the yeti/mmap directory.

The code is self-explanatory:


% Simply rolling average plug-in for Scanbox

% Open memory mapped file -- define just the header first

mmfile = memmapfile('scanbox.mmap','Writable',true, ...
    'Format', { 'int16' [1 16] 'header' } , 'Repeat', 1);
flag = 1;

% Define the forgetting factor  0 < delta <= 1

delta = 0.9;  % this will generate an exponential decaying memory window: lambda^n

% Process all incoming frames until Scanbox stops

while(true)
    
    while(mmfile.Data.header(1)<0) % wait for a new frame...
        if(mmfile.Data.header(1) == -2) % exit if Scanbox stopped
            return;
        end
    end
    
    display(sprintf('Frame %06d',mmfile.Data.header(1))); % print frame# being processed
    
    if(flag) % first time? Format chA according to lines/columns in data
        mmfile.Format = {'int16' [1 16] 'header' ; ...
            'uint16' double([mmfile.Data.header(2) mmfile.Data.header(3)]) 'chA'};
        mchA = double(intmax('uint16')-mmfile.Data.chA);
        flag = 0;
        ih = imagesc(mchA); % setup display
        axis off;           % remove axis
        colormap gray;      % use gray colormap
        truesize            % true image size
    else
        mchA = delta*mchA + (1-delta)*double(intmax('uint16')-mmfile.Data.chA);
        ih.CData = mchA;
    end
    
    mmfile.Data.header(1) = -1; % signal Scanbox that frame has been consumed!
    
    drawnow limitrate;
    
end

clear(mmfile); % close the memory mapped file
close all;     % close all figures

If you have any questions just post them below. Homework: Create a plug-in that shows areas of the image that are saturated in red (due date: Aug 8, 2016)

Memory-mapped data-sharing for real-time processing

To enhance the real-time processing capabilities of Scanbox we now provide a general mechanism to share frames being collected with another application running concurrently on the same computer via a memory mapped file.

This general and flexible mechanism allows any user to extend the real-time processing of Scanbox beyond what is already offered.

As an example, this interface allows one to perform pixel-by-pixel image processing in ways that do not require prior definitions of ROIs. One such case would be the calculation of temporally encoded cortical maps by on-the-fly calculations of the magnitude/phase of each pixel.

All you have to do to enable this feature is to set the “mmap” variable to 1 in the configuration file and, in those imaging sessions you want to stream the data, enable the “Memory map” checkbox in the real-time processing panel:

realtime

The memory mapped file contains a 16 word header (int16s) that allows communication between Scanbox and another process, which is then followed by the imagery data.  Only the first 4 words of the header are used at the moment (the other are reserved for future use).

The first word is used as a semaphore between the applications to control access to the memory mapped file.  The value of this variable is -1 if the file can be used by Scanbox to stream data, it is -2 if Scanbox has stopped imaging, and it is a non-negative number that corresponds to a frame if Scanbox has copied the data to the memory mapped file.

The second and third variables represent the size of the frame, used by the application processing the data to interpret the organization of the file as an image.

The fourth variable is a copy of the TTL1, which is high during a stimulus presentation and low otherwise.

As an example of a Matlab process that consumes data from Scanbox and all it does is compute the average frame is listed below.


% Simple example of memory mapped processing in Scanbox

% Open memory mapped file -- just header first
mmfile = memmapfile('scanbox.mmap','Writable',true, ... 
 'Format', { 'int16' [1 16] 'header' } , 'Repeat', 1);
flag = 1;

% Process all incoming frames until Scanbox stops

while(true)

   while(mmfile.Data.header(1)<0) % wait for a new frame...
     if(mmfile.Data.header(1) == -2) % exit if Scanbox stopped
     return;
     end
   end
 
 display(sprintf('Frame %06d',mmfile.Data.header(1))) % print frame# being processed
 
 if(flag) % first time? Format chA according to lines/columns in data
   mmfile.Format = {'int16' [1 16] 'header' ; ... 
   'uint16' double([mmfile.Data.header(2) mmfile.Data.header(3)]) 'chA'};
   mchA = double(mmfile.Data.chA);
   flag = 0;
 end
 
 mchA = mchA + double(mmfile.Data.chA);
 
 % Here you can do whatever you want with
 % your data.... I am just accumulating.
 
 mmfile.Data.header(1) = -1; % signal Scanbox that frame has been consumed!
end

clear(mmfile)

imagesc(mchA); % display mean image
truesize

The interface allows you to use any language that provides you access to memory mapped files, such as Python.

So, if you happen to write some neat real-time image processing module for Scanbox, feel free to share!

A set of more readable functions that encapsulate the basic functionality shown above will be provided soon to make the writing of real-time processing applications easier, and we may also adopt a double-buffering or ring-buffer mechanisms.