Page MenuHomePhabricator

Synchronization of individual render windows: camera position
Open, NormalPublic

Description

In order to synchronize the the camera position of multiple render windows, we need a synchronization concept.
Options that should be modifiable:

  • camera zoom
  • camera pan (x, y)
  • image slice walk (z)
  • view direction (axial, sagittal, coronal)

The view direction can be individually changed for each render window (has already been proven in the render window manager).

Since the camera position does not belong to a data node / image but to a render window, we will not use the property mechanism.
Instead, we want to use a event system.

What has to be considered concerning such an event system?

  • context information to provide group-specific events?
  • rules to define what happens on synchronization? (see T24274 and T24275)
    • std::function for arbitrary rule
  • how does the mouse event system work? (see T24273)

Event Timeline

kalali triaged this task as Normal priority.Feb 7 2018, 11:08 AM
kalali created this task.
kalali updated the task description. (Show Details)Feb 7 2018, 11:11 AM

Using the basic implementation (first working version) we can not zoom, pan or slice walk inside a single render window without the geometry planes.
Need to find the mechanism to navigate on a single render window.

kalali claimed this task.Feb 8 2018, 12:06 PM
kalali added a subscriber: metzger.
kalali added a comment.EditedFeb 8 2018, 2:55 PM

The MouseModeSwitcher was missing. Now the camera in each render window can be individually zoomed, panned and one can move through the image slices.
Using the render window manager one can change the view direction of each render window individually.
The slice-cross-hair is not visible anymore.

However, the mouse click (set new position) is still synchronized.
Next step is to find out if the mouse-event system can be synchronized / de-synchronized for different render windows.

kalali added a comment.EditedFeb 9 2018, 10:24 AM

Using different RenderingManager for each render window allows to de-synchronize the left-mouse-click synchronization.
However, now there is no single render window manager anymore that holds all visible render windows.

Using no RenderingManager may not work. On creation of QmitkRenderWindow, which is a mitk::RenderWindowBase, the function RenderWindowBase::Initialize is called. Here the RenderingManager is initialized with the global instance (singleton), if no RenderingMangager is given. Then the render window will be added to the RenderingManager.

kalali added a comment.EditedFeb 9 2018, 1:29 PM

Next step: Find out how to synchronize the view direction, camera zoom, camera pan and slice walk (see T24232).

kalali added a subscriber: floca.Feb 15 2018, 6:18 PM

The mitk::MouseModeSwitcher initializes the mitk::DisplayInteractor and registers it as a mitk::InteractionEventObserver service. However, we don't want a hard-coded display interactor at this point, since other parts of MITK will use a different display interactor. The service registration could be moved to the mitk::DisplayInteractor itself (self-registration on construction).
The mouse mode switcher also offers functionality to set the interaction scheme or the mouse mode (by setting .xml-files). We might not be able to use all known .xml-files in our own custom display interactor so we rather move the implementation of this functions to the display interactor.

Using two display interactors (and register both as a service) will lead to both interactors react on a mouse interaction. This needs to be considered.
In our scenario we want to replace the StdMultiWidget with our CustomMultiWidget so the classic DisplayInteractor of the mouse mode switcher of the StdMultiWidget will not be active.

kalali added a comment.EditedFeb 26 2018, 12:56 PM



kalali updated the task description. (Show Details)Feb 27 2018, 10:44 AM
kalali added a comment.EditedFeb 27 2018, 4:11 PM

For even looser coupling: The InteractionBroadcast class could be used as the event state machine and the interaction observer, whereas the classic display interactor (StdDisplayInteractor) now listens to the events from the broadcast class.
This way, each display action event listener can listen to specific action events without being forced to implement behavior for each event. One can easily change the event state machine / interaction config file without forcing display action event listener to react to new interactions.

A custom display action event (listener) can be used for our scenario, which connects the events from the broadcast class with custom std::functions.
The classic display interactor provides the same functions as before (Move, Zoom, ... in mitk::DisplayInteractor) and directly connects them internally with the received events.

According to the micro-service concept the broadcast class is notified about a render window interaction event (each InteractionEventObserver is notified). That means, the broadcast class will send its action events to the workbench, regardless of where the mouse interaction has happened (e.g. multiple render window editors).
Each render window editor can have its own specific action event listener that describes what to do on the events but we need a mechanism to filter the origin of the mouse interaction.

EDIT: Changed the naming to better reflect the idea behind the different classes.

kalali added a comment.EditedApr 6 2018, 4:26 PM

How the CustomDisplayActionEventHandler could look like:

#ifndef MITKCUSTOMDISPLAYACTIONEVENTHANDLER_H
#define MITKCUSTOMDISPLAYACTIONEVENTHANDLER_H

#include <MitkCoreExports.h>

#include "mitkDisplayActionEventBroadcast.h"
#include "mitkDisplayInteractor.h"
#include "mitkDisplayActionEvents.h"
#include "mitkStdFunctionCommand.h"

#include <mitkWeakPointer.h>

// c++
#include <functional>

namespace mitk
{
  class MITKCORE_EXPORT CustomDisplayActionEventHandler
  {
  public:
    /**
    * @brief Sets the display action event broadcast class that should be observed.
    *     This class receives events from the given broadcast class and triggers the "corresponding functions" to perform the custom actions.
    *     "Corresponding functions" are std::functions inside commands that observe the specific display action event.
    *
    * @post If the same broadcast class was already set, nothing changed
    * @post If a different broadcast class was already set, the observing commands are removed as observer
    *       #TODO: Do we want to have the current observing commands observe the new broadcast class?
    *
    * @par  observableBroadcast   The 'DisplayActionEventBroadcast'-class that should be observed.
    */
    void SetObservableBroadcast(mitk::DisplayActionEventBroadcast* observableBroadcast);

    /**
    * @brief Creates a 'StdFunctionCommand' and uses the given std::functions to customize the command.
    *     The display action event is used to define on which event the command should react.
    *     The display action event broadcast class member is then observed by the newly created command.
    *     A tag for the command is returned and stored in a member vector.
    *
    * @pre    The class' observable (the display action event broadcast) has to be set to connect display events.
    * @throw  mitk::Exception, if the class' observable is null.
    *
    * @par displayActionEvent   The 'DisplayActionEvent' on which the command should react.
    * @par actionFunction       A custom std::Function that will be executed if the command receives the correct event.
    * @par filterFunction       A custom std::Function that will be checked before the execution of the action function.
    *                           If the filter function is not specified, a default filter always returning 'true' will be used.
    *
    * @return   An unsigned long tag to identify, receive or remove the newly created 'StdFunctionCommand'.
    */
    unsigned long ConnectDisplayActionEvent(const mitk::DisplayActionEvent& displayActionEvent,
      const mitk::StdFunctionCommand::ActionFunction& actionFunction,
      const mitk::StdFunctionCommand::FilterFunction& filterFunction = [](const itk::EventObject& eventObject) { return true; });

    /**
    * @brief Uses the given observer tag to remove the corresponding custom StdFunctionCommand as an observer of the observed
    *     display action event broadcast class.
    *     If the given tag is not contained in the member vector of observer tags, nothing happens.
    *
    * @pre    The class' observable (the display action event broadcast) has to be set to connect display events.
    * @throw  mitk::Exception, if the class' observable is null.
    *
    * @par observerTag   The unsigned long tag to identify the 'StdFunctionCommand' observer.
    */
    void DisconnectObserver(unsigned long observerTag);

    const std::vector<unsigned long>& GetAllObserverTags() { return m_ObserverTags; };

  private:
    mitk::WeakPointer<mitk::DisplayActionEventBroadcast> m_ObservableBroadcast;
    std::vector<unsigned long> m_ObserverTags;

  };

} // end namespace mitk

#endif // MITKCUSTOMDISPLAYACTIONEVENTHANDLER_H
kalali added a comment.Apr 6 2018, 4:27 PM

The proposed class uses a custom command:

#ifndef MITKSTDFUNCTIONCOMMAND_H
#define MITKSTDFUNCTIONCOMMAND_H

#include <MitkCoreExports.h>

// itk
#include <itkCommand.h>

namespace mitk
{
  // define custom command to accept std::functions as "filter" and as "action"
  class MITKCORE_EXPORT StdFunctionCommand : public itk::Command
  {
  public:
    using Self =      StdFunctionCommand;
    using Pointer =   itk::SmartPointer<Self>;

    using FilterFunction = std::function<bool(const itk::EventObject&)>;
    using ActionFunction = std::function<void(const itk::EventObject&)>;

    /** Method for creation through the object factory. */
    itkNewMacro(Self);

    /** Run-time type information (and related methods). */
    itkTypeMacro(StdFunctionCommand, itk::Command);

    void SetCommandFilter(FilterFunction stdFunctionFilter)
    {
      m_StdFilterFunction = stdFunctionFilter;
    }

    void SetCommandAction(ActionFunction stdFunctionAction)
    {
      m_StdActionFunction = stdFunctionAction;
    }

    virtual void Execute(Object*, const itk::EventObject& event) override
    {
      if (m_StdFilterFunction && m_StdActionFunction)
      {
        if (m_StdFilterFunction(event))
        {
          m_StdActionFunction(event);
        }
      }
    }

    virtual void Execute(const Object*, const itk::EventObject& event) override
    {
      if (m_StdFilterFunction && m_StdActionFunction)
      {
        if (m_StdFilterFunction(event))
        {
          m_StdActionFunction(event);
        }
      }
    }

  protected:
    FilterFunction m_StdFilterFunction;
    ActionFunction m_StdActionFunction;

    StdFunctionCommand()
      : m_StdFilterFunction(nullptr)
      , m_StdActionFunction(nullptr)
    {}

    virtual ~StdFunctionCommand() {}

  private:
    ITK_DISALLOW_COPY_AND_ASSIGN(StdFunctionCommand);
  };

} // end namespace mitk

#endif // MITKSTDFUNCTIONCOMMAND_H

This is the current status:
The CustomDisplayActionEventHandler is informed by the DisplayActionEventBroadcast class that it observes about certain display action events. This automatically leads to a call of the ActionFunctions that where connected to this certain display action events. A user of the handler-class can use any known display action event to connects an action function (an std::function) to this event. The user can additionally specify a filter function (also an std::function) with any logic to use it as a filter / conditional check before executing the action function.

Both functions know the display action event, that triggered the function call. This event also includes the original interaction event, so that the sender can be retrieved (the sending renderer).

kalali added a comment.EditedApr 10 2018, 3:44 PM

The observed DisplayActionEventBroadcast class is now an EventStateMachine and an InteractionEventObserver, just like the original DisplayInteractor before.
However, only the geometric computation is performed. Instead of actually updating the render windows the corresponding event is invoked to inform its observer (e.g. the CustomDisplayActionEventHandler or the StdDisplayActionEventHandler (the former display interactor).

Now an action function looks like this:

void mitk::DisplayActionEventBroadcast::Move(StateMachineAction* stateMachineAction, InteractionEvent* interactionEvent)
{
  auto* positionEvent = static_cast<InteractionPositionEvent*>(interactionEvent);

  float invertModifier = -1.0;
  if (m_InvertMoveDirection)
  {
    invertModifier = 1.0;
  }

  // compute translation
  BaseRenderer* sender = interactionEvent->GetSender();
  Vector2D moveVector = (positionEvent->GetPointerPositionOnScreen() - m_LastDisplayCoordinate) * invertModifier;
  moveVector *= sender->GetScaleFactorMMPerDisplayUnit(); // #TODO: put here?

  // store new display coordinate
  m_LastDisplayCoordinate = positionEvent->GetPointerPositionOnScreen();

  // propagate move event with computed geometry values
  InvokeEvent(DisplayMoveEvent(interactionEvent, moveVector)); // NEW LINE: INSTEAD OF ACTUALLY UPDATING THE RENDER WINDOWS THE EVENT IS JUST PROPAGATED
}
floca added a comment.Apr 10 2018, 5:34 PM
       #TODO: Do we want to have the current observing commands observe the new broadcast class?
void SetObservableBroadcast(mitk::DisplayActionEventBroadcast* observableBroadcast);

No. I would keep it simple setting is resetting/clearing of all commands so far.

/**
* @brief Creates a 'StdFunctionCommand' and uses the given std::functions to customize the command.

StdFunctionCommand is an implementation detail. The Interface user must not know how we make it work.

unsigned long ConnectDisplayActionEvent(const mitk::DisplayActionEvent& displayActionEvent,

You should specify a own type for the tag and use it in the interface

using OberserverTagType = unsigned long;
ObserverTagType ConnectDisplayActionEvent(const mitk::DisplayActionEvent& displayActionEvent,

The rest looks good.

I think in the future we, should also move this design/architecture review to the code review/audit module. So just make a branch for this tasks and add your header propsals/drafts. Then we can comment/diff/refactor on/in that branch.
I think it is easier than making such an detailed inspection in the task comments itself.

I pushed the recent branch and added your suggestions.

However, there are some questions. I commented on them in the latest commit of T23760-Custom-multi-widget-editor: eac8b99e0769

kalali updated the task description. (Show Details)Apr 17 2018, 3:00 PM
kalali added a comment.May 8 2018, 9:45 AM

It is now possible to synchronize the mouse interaction (move, zoom, scroll, adjust level window, pan etc.) for all render windows of the custom multi widget via a toolbar-button. Another button allows to synchronize the view direction.

kalali added a comment.EditedMay 9 2018, 4:43 PM

Synchronization events should allow to compute / retrieve the new world coordinates as well as the delta (relative, absolute) of the corresponding change (e.g. slice-delta for the scroll event) (see T24810)

kalali removed kalali as the assignee of this task.Feb 13 2019, 2:18 PM
kalali added a project: Restricted Project.Feb 13 2019, 2:22 PM