Page MenuHomePhabricator

Provide filtering techniques for interaction events
Closed, ResolvedPublic

Assigned To
Authored By
kalali
Feb 27 2018, 6:55 PM
Referenced Files
F983998: StateMachine.png
Mar 14 2018, 3:04 PM
F984010: StateMachineMultiClick.png
Mar 14 2018, 3:04 PM
F974798: event_handling.png
Mar 8 2018, 1:58 PM
Subscribers

Description

We have a hierarchy of interaction events:
Qt mouse events (QMouseEvent) are received by a QmitkRenderWindow and transformed into an mitk::InteractionEvent (e.g. mitk::MouseMoveEvent).
This event is process by the renderer-specific dispatcher of the render window.
The dispatcher now retrieves all mitk::InteractionEventObserver using the micro-service approach. Thus all observer get notified, regardless of whether they are in any way connected to the sending render window or not.
A notified observer handles the event (e.g. the mitk::DisplayInteractor transform the event into a function call using its state machine configuration).
An example of this process for a mitk::DataInteractor can be seen here: http://docs.mitk.org/nightly/DataInteractionPage.html

event_handling.png (380×1 px, 30 KB)

Using our broadcast-class this would lead to another event being sent (via itk::Object::InvokeEvent). This is the concrete display interaction event that we want to process in a customized manner.

However, we want to include filtering techniques on different levels:

  1. Since the micro-service approach calls all interaction event observer, we want to filter if the interaction happened inside the MITK workbench part we are interested in (e.g. inside a specific multi widget editor).
  2. Since we might only want to react on specific render windows (e.g. only those that display the same images), we want to filter if the itk event is relevant for a certain observer.

Event Timeline

kalali triaged this task as Normal priority.Feb 27 2018, 6:55 PM
kalali created this task.

For 2.:
Looking at the suggested approach in T24274 we only accept std:.functions that need an itk::EventObject as parameter. This way we can easily provide the sending event for the std::function to work with. Since we design our own events (T24275) we can include all needed information for filtering (e.g. retrieving the sending render window (name, ID, ...) via the event).
The std::function has access to the information inside the event so it can use it for filtering.

Which information is needed for filtering techniques?

  • sending render window

For 2.:
Currently I don't see a way to decide if we want to call a command (that observes an observable object) on a specific event or not.
If we do something like

m_Observable->AddObserver(DisplayInteractorZoomEvent(nullptr, 0, Point2D()), command);

the command is processed on each DisplayintercatorZoomEvent. I see no possibility to filter at this point.

Using the approach from T24274 we could instead provide a filter for the observer instance that is checked when the Callback-function is called and tries to call the std::function:

void Callback(const itk::EventObject& eventObject) { if (m_Callback && Filter()) m_Callback(eventObject); }

This is similar to just putting the filter inside the callback std::function.
Here we have to think about different possibilities a particular Filter()-function gives us (variadic template function with arbitrary parameters?)

kalali updated the task description. (Show Details)

For 1.:
The sending QmitkRenderWindow does not know where it is included. It can only provide itself (or better: its renderer) as a sending object. This does not tell the receiver anything about the workbench part where the renderer is used.

The dispatcher informs each mitk::InteractionEventObserver that is registered as a micro-service. Therefore each mitk::InteractionEventObserver reacts and sends it concrete display interaction event.
So one dispatcher (from a certain renderer) triggers all observers.

kalali removed a subscriber: metzger.

For 1.:
Since the dispatcher informs each mitk::InteractionEventObserver that is registered as a micro-service and therefore the broadcast is always notified, it may be possible to filter here:
The dispatcher, which is specific to each renderer inside a render window, may get a list of concrete InteractionEventObserver which he notifies if he processes an incoming event (instead of notifying each registered observer). This way, e.g. a multi widget that creates several render windows could hold it's own interaction event observer and set this observer as the only observer to notify for its render windows.
If a multi widget (as an example of a workbench part) also wants to work with another interaction event observer (from another workbench part, e.g. a second multi widget) it has to retrieve this observer and add this to the list of known observer of the dispatcher.

The problem with this approach is, that it breaks the intention of the micro-service approach: It is an active implementation against the principle to notify all registered services.
Another problem is that the observer of other workbench parts are not known to the current workbench part. One could retrieve all interaction event observer via the micro-service approach and use the desired one. What does the desired one mean?
One has to be able to distinguish between different InteractionEventObserver-instances (from different workbench parts). This could be done by including the context (the workbench part that created the observer instance) in the observer. But if we do this, we could just simply filter by the context in the dispatcher and wouldn't need to set concrete observer instances that are valid and should be notified. The dispatcher could just hold multiple context and filter all interaction event observer itself with its valid context.

This approach would also just stop sending interaction events from a renderer inside a workbench part to renderer in a different workbench part (the renderer specific dispatcher would now only notify the valid (possibly only the local) interaction event observer). It would still receive events from other workbench parts, where the dispatcher still notifies all interaction event observer via the micro-service approach. With this approach a renderer can not actively stop listening.

Another idea is to move both filtering techniques (1. and 2.) to the same level:
If we know all our render windows in the corresponding workbench part (e.g. a multi widget knows all its render windows) we can just receive each interaction event and then check the sending render window (renderer) (as in 2.: react on specific render windows). The sending render window could be compared to a list of render windows that are currently held by the receiving multi widget. This way we would immediately see if the sending render window was a render window of our own workbench part (which we know and hold in the list) or if it is an unknown render window (from another workbench part).

For 1.: It is also possible to filter on the level of the broadcast class. Here we send events that have been transformed from mitk::InteractionEvent to our own custom mitk::DisplayInteractorEvent to inform the concrete handler classes. However, the broadcast class is also a state machine (see T24216). So first we have to handle the incoming InteractionEvent and find the appropriate action (which allows us to define a specific DisplayInteractorEvent that we can send). With the filter approach we don't need to access the state machine and handle the event, if we compare the sender to a list of valid render windows (renderer). Only if the sending renderer is valid, we handle the event. This allows to actively stop listening.

Compared to the aforementioned approaches:

  • we don't break the intention of the micro-service approach: still all registered services are notified (e.g. our broadcast class)
  • we don't stop sending events but stop listening
  • we can see if the event is coming from a render window of our own workbench part (which we know and hold in the list) or if it is an unknown render window (from another workbench part)

The question arose, if the EventStateMachine (e.g. the implementation of it in our broadcast-class) is able to distinguish between different senders, so let's look at the way the event state machine works.
The referenced page states: "The basic idea here is that each interaction can be described by states and transitions which in turn trigger actions." So what are the states?
Looking at the configuration of the MITK event state machine (DisplayInteraction.xml) we find several states: The start state start, into which the state machine is set in the beginning.
Each time the state machine handles an event it retrieves all possible transitions that are valid at the current state of the state machine. It then takes the first possible transition (possible means checking some transition conditions). Each transition leads to a state (next state) in which the event state machine moves if the corresponding action of the transition is performed.
In order to define the transition to take, so called event_variants are used: They further describe a transition to take, since the event_class is not sufficient to describe a transition. An event_class is the class of interaction events (such as InteractionPositionEvent).

If the state machine changes into another state due to a transition, other transitions (and actions) for this state are now possible.
The current default event state machine allows to get back to the start state from every other state. This is usually achieved by a receiving an event_variant that was triggered by a MouseReleaseEvent.

Keeping the idea of the state machine in mind, the states of the state machine describe the current mouse states and NOT the states of the render windows (or the renderer). The renderer are only used as the object that sent the event for a mouse state transition.
Note: Moving a mouse does also trigger an event variant. However, when the mouse is moved while a mouse button is clicked, the renderer that sent the event will not change. This means, that pressing the left mouse button in render window 1, moving the mouse with the pressed left mouse button to render window 2 and releasing the mouse button here will still perform all transitions / actions on render window 1. The QMouseEvent-documentation says: Qt automatically grabs the mouse when a mouse button is pressed inside a widget; the widget will continue to receive mouse events until the last mouse button is released..

So as a summary: If we interact with the render windows using the mouse, we start from a start state before the first mouse button has been pressed. We go back to this start state after a mouse button has been released. In between we are in specific mouse states, depending on the event variant that led from the start state to this specific mouse state. The sending render window is kept as the sender if a mouse button is kept pressed.

If we want to achieve something like a multi-click interaction, we need to define new states. In our current event state machine .xml-config file we go back to the start state after a mouse button has been released. However, we could also include a new state (e.g. clicked_once) that can be reached by releasing the mouse button after the mouse button has been clicked (e.g. left mouse button). In this state we might be able to perform the same transitions as in the original start state, with one exception: If the left mouse button is clicked (which is then the second click after the original start state) we move into a a new state (e.g. clicked_twice). here we perform our multi-click actions. From this state we are able to go back to the original start state.
Since the multi-click in our scenario has to be performed in a single render window, we must detect, if the first mouse-click happened inside render window 1 and the second mouse-click happened inside another render window. We could do this by including an event variant, that sets the state machine back into the original start state if the mouse left the render window of the first mouse click (see QmitkRenderWindow::leaveEvent). This will forget the first mouse-click since the mouse is no longer in the clicked_once state.

StateMachine.png (342×568 px, 20 KB)

The current state machine changes its states according to mouse events (left mouse button pressed = SetCrosshair, middle mouse button pressed = StartMove and right mouse button pressed = StartZoom).
On each corresponding mouse button release the state machine returns to its original start state. Holding the corresponding button and moving the mouse around results in the same state.

StateMachineMultiClick.png (369×1 px, 28 KB)

In this example, the left mouse button transformation path was changed: It now includes a new state that can be reached, if the pressed left mouse button has been released. In the new StateClickOnce the state machine now remembers a single left mouse button click. It is possible to change e.g. to the move state from here, when pressing the middle mouse button. However, when clicking the left mouse button again the state machine changes to the StateClickTwice state. (Whatever happens here is irrelevant for this example.)
When the mouse leaves the render window the state machine returns to its original start state, so that a multi click will only be accepted inside a single render window.

As a summary for the filtering techniques needed for 1. and 2. (see task description):

  • we want to actively stop listening to events sent from certain render windows
  • we don't want to actively stop sending events from certain render windows

listener are responsible to filter incoming events, not to filter outgoing events

  • we can define different levels on which we want to filter and thus "stop the interaction propagation process"
    • filter if an event should even be received
    • filter if a resulting action should be performed depending on the sender / the sender's properties
  • on each level we will use render windows / renderer (the "the event sender") for filtering

do we need different levels?
Discussion for the open question:
If we want to actively stop listening to interaction events the highest class where we could filter and and thus "stop the interaction propagation process" is on the level of the InteractionEventObserver. In our scenario this would be the DisplayActionEventBroadcast-class. Here we could check the sender of the interaction event before we use the state machine to figure out which action to take (i.e. before calling HandleEvent).
This would be achieved by setting a list of valid senders for the broadcast-class that can be compared to the sending object.
Pros:

  • it saves us function calls and computation since we stop the interaction propagation process early
  • we only need to define the valid senders at one location (for each broadcast-class)
  • it is easy to inspect and to maintain the list of valid senders (just add or remove valid senders to or from the list)

Cons:

  • since we only have one list we cannot selectively stop listening
    • selectively stop listening means that some render windows inside our multi widget should listen to the event whereas some render windows inside the same multi widgets should not

is this needed?

The next lower class where we could filter is on the level of the xyActionEventHandler. Here we already figured out which action to take and want to perform this action. Performing this action means either

  • calling the classic functions on the sender and other render windows to modify the render window (StdDisplayActionEventHandler)
  • calling the individual std::functions for each observer that is observing the sent action event (CustomDisplayActionEventHandler)

Since we are currently using the itk command pattern here (see T24382), we can further split this level:
The next sub-level is filtering inside the command before the callback-function is called. This can be done by using the CheckEvent-function of our custom action events (see T24275). The itk object that invokes an event (our broadcast class) will retrieve all its observers and their corresponding prototype event. The mentioned CheckEvent-function is called to check if the command (and thus the callback function) should be executed. Filtering here would give us the following pros and cons:
Pros:

  • we can define the list of valid senders individually for each observer

Cons:

  • we might compute values, needed for the action, although we later realize that we won't perform the action

How would a CheckEvent-function work?
The check event function has access to the incoming event object (that was invoked) and to the member variables of the prototype event. If the prototype event has a specific render window set, the function could check if the sender of the incoming event is the same as the prototype member render window. Since we can also create a prototype with different parameters we can also use a whole list of valid senders. The check event function then has to compare the sender to the list of valid senders. This is similar to the idea of the technique on the highest level. However, as stated above, we can individually define these lists.
However, if we use the prototype event idea we have another disadvantage:
Const:

  • we need to remove and then add the command (as observer) again to change this individual list

Therefore another approach is possible: Since we are creating an observer struct (see T24382) we could simply define a SetValidSenders-function inside the struct which takes a vector of valid render windows / renderer. The vector will be given from outside when using the ConnectDisplayInteractorZoomEvent-function (e.g. void ConnectDisplayInteractorZoomEvent(std::function<void(const itk::EventObject& eventObject)> callback, std::vector<mitk::BaseRenderer::Pointer>& validSenders)).
Each time the check event function is called and returns true it will execute the callback function (of our observer struct). In the observer struct we check if the callback function is actually set and can be executed (if (m_Callback) m_Callback(eventObject);). Here we could add another check that takes into account the set vector of valid senders and possibly stop before executing the callback. The benefit is that the user can add or remove valid senders during runtime. We will provide convenient access to modify this vector of valid senders inside an observer.

If we continue this idea, we could go as far as just adding a second std::function to our observer. Similar to the original std::function this could be set from outside and stored inside the observer struct. The benefit here is that this std::function can be arbitrary and can therefore contain more than just a vector of valid senders.
Pros

  • std::function allows arbitrary filtering techniques (not only relying on the sender itself)

Another benefit is that we eventually could use this kind of technique for both filter levels (see 1. and 2. in task description). We would not only define what a valid sender is but would be able to define how the valid sender's properties need to be which combines both filter levels.

Currently the latest approach is implemented: We provide a second std::function, a FilterFunction for the command that observes a certain event. This way, the command can use this filter function to check a condition before executing its ActionFunction. For this, a customized StdFunctionCommand-class has been implemented.
Using the customized command, the mentioned observer struct is not needed, since the command IS the observer itself.

With this approach, we are able to filter on the lowest level, which allows for the most individual filtering for each event-observer pair.

This idea is contained in T23760-Custom-multi-widget-editor. It will be used in the first version of a custom multi widget editor. This task remains as a look-up task for our decision process. For new ideas concerning filter-techniques new tasks should be created.