Feature caching not cleared

[giovannivolpe]

If a feature is called directly, its result is cached internally. This can affect how it behaves when reused in chained pipelines. For example:

stack_feature = dt.Stack(value=2)
_ = stack_feature(1) # Evaluate the feature and cache the output
(1 & stack_feature)()
[1, 1, 2]

To ensure consistent behavior when reusing a feature after calling it, reset its state using instead:

stack_feature = dt.Stack(value=2)
_ = stack_feature(1)
stack_feature.update() # clear cached state
(1 & stack_feature)()
[1, 2]

This should not happen and is probably a problem with the caching. The two calls should both produce [1, 2]

[giovannivolpe]

Comments from @BenjaminMidtvedt:

The issue can be reduced to

stack_feature = dt.Stack(value=2)
_ = stack_feature(3) # Evaluate the feature and cache the output
y = stack_feature()
print(y)

which should yield [2] but yields [2, 3]. This means that changing the input to the feature, here by removing the argument, has not invalidated the feature as it should.

But if you give a new input, like here, it correctly outputs [2, 2], meaning it has been invalidated.

stack_feature = dt.Stack(value=2)
_ = stack_feature(3) # Evaluate the feature and cache the output
y = stack_feature(2)
print(y)

This leads one to verify the logic for invalidating based on input in features.py:

           if (
               image_list is not None
               and not (isinstance(image_list, list) and len(image_list) == 0)
               and not (isinstance(image_list, tuple)
                       and any(isinstance(x, SourceItem) for x in image_list))
           ):
               self._input.set_value(image_list, _ID=_ID)

Here, we see that we only set the input value if it is not None, which is likely a bug. There's just not many features except Stack that can be called with out without input and give reasonable results, so it never was a problem.

Here, we should add a check, that if image_list is None, and self._input.current_value(_ID_ID) is not None, then we should set the input to be None. This should fix the issue at the source

[Pwhsky]

Right, the error was somewhere else than in dt.Stack.
Is this the fix you have in mind?

In features.Feature:

def __call__(self, image_list, _ID, **kwargs):
...

  with config.with_backend(self._backend):
    # If image_list is as Source, activate it.
    self._activate_sources(image_list)
    
    # Potentially fragile. Maybe a special variable dt._last_input instead?
    # If the input is not empty, set the value of the input.
    if (
        image_list is not None
        and not (isinstance(image_list, list) and len(image_list) == 0)
        and not (isinstance(image_list, tuple)
                and any(isinstance(x, SourceItem) for x in image_list))
    ):
        self._input.set_value(image_list, _ID=_ID)
    
    #Fix goes here ↓
    if image_list is None and self._input(_ID=_ID) is not None:
        self._input.set_value(None, _ID=_ID)

[BenjaminMidtvedt]

This is what I had in mind, yes. I think this snippet should be improved for clarity, but I'd have that as a separate issue.