Memory efficient spike encoding

[djsaunde]

At present, we generate input spikes prior to simulation. This results in tensors of shape [time, *input_shape]. When time and / or input_shape is large, this uses a lot of memory.

What we could do is generate spikes from encodings as-needed during simulation. The most obvious way to implement this, to me, is to create Nodes objects which maintain the variables needed to generate spikes according to their encoding function. For example, PoissonNodes would maintain rate parameters and generate spikes per timestep according to that rate. This would reduce memory usage down from [time, *input_shape] to just [*input_shape], which would be a big win especially for long simulations.

[Hananel-Hazan]

Good idea.
Two things that I think are relevant:

1. Need to check if generate input spike for one iteration is equivalent (same distribution and properties as to generate input spikes for time

2. I think that creating a special Nodes can cause a confusion with other nodes that are actually neurons. We can create a DataFeeder object that will be connected to the network like Monitors to feed the neuron. The object will be part of datasets that can maybe call Fetcher that use any encoding to fetch and feed the relevant neurons with one iteration of data

[djsaunde]

I'm not sure I agree with your second point. We already have two types of Nodes: those that subclass Nodes only, and those that subclass both Nodes and AbstractInput. I think it's a good abstraction! We can have another abstract class (maybe AbstractGenerator?) which would service things like PoissonGenerator, BernoulliGenerator, etc. I know this approach works because brian2 uses it, and it's nice and intuitive.

Another thing: Pre-computing the spike encoding is typically more efficient in terms of time, but not in terms of memory (as I wrote in the first post). So, there's a trade-off between using an Encoder vs. using a Generator (or whatever we want to call it), which the user will have to consider when writing their experiment code.

[Hananel-Hazan]

You are right, this trade offs is a good option to give the user to choose. On long runs Memory can be in short demand. You can use some buffering mechanism to generate couple of iteration a head of time to save CPU usage and not completely hung the memory.

The confusion start with the definition of Nodes. Basically, Node have input and output that change according to the input. You can take any node in Nodes (beside the Abstract) and place it everywhere in the network in any order of layer and have connections to other nodes. For instance, you can use McCullochPitts in the first layer (serve as input nodes) connected to LIF then connected to Input then connected to IzhikevichNodes as output of the network.

If the GeneratorNode that you suggested can change its output as some function to the input, and it can be place any where in the network, it can be serve as a node and should be under Nodes.

Else, If the purpose of the Generator is creating/generating output once it initialize. Its place is in datasets as another method to feed the network. Because it cannot be place anywhere in the network beside first layer and once it initialize, it will not change it behavior.

[djsaunde]

The confusion start with the definition of Nodes. Basically, Node have input and output that change according to the input. You can take any node in Nodes (beside the Abstract) and place it everywhere in the network in any order of layer and have connections to other nodes. For instance, you can use McCullochPitts in the first layer (serve as input nodes) connected to LIF then connected to Input then connected to IzhikevichNodes as output of the network.

Yeah, that's true, but if a layer connects to a nodes object subclassing AbstractInput, the input from that layer is simply discarded. Only user-passed data (in the inpts dictionary) is used by nodes subclassing AbstractInput. In fact, it might be good to throw a warning, or even an error, if this kind of connectivity exists. It doesn't make sense because it won't do anything.

In the same way, nodes subclassing AbstractGenerator would discard inputs from layers connecting to it. My idea is that generator nodes would have attributes which would determine their output activity. For example, for a PoissonGenerator, it might have an attribute rates which determine the firing rates, in Hertz, of the neurons in the layer. The user could set this prior to simulation, or between simulations:

import torch
from bindsnet.network import Network
from bindsnet.network.nodes import PoissonGenerator

network = Network()
input_layer = PoissonGenerator(rates=0, n=100, ...)  # Input layer of size 100.
network.add_layer(input_layer, name="I")
rates = 120 * torch.rand(100)  # Input data of size 100; specifies firing rates in [0Hz, 120Hz].
input_layer.rates = rates
network.run(inpts={}, time=1000)

In the above, the PoissonGenerator would generate spikes according to a Poisson process (sample x ~ Uniform(0, 1) and check if x < (rates / 1000) * dt).

Alternatively, we could pass inpts={"I": rates}, which would simply set the rates attribute of the PoissonGenerator object at the beginning of network.run.

[Hananel-Hazan]

You are right, because AbstractInput is only abstract, its behavior need to be corrected.
The rest of the nodes have the same behavior because Nodes are the Soma of neuron, they getting the sum of input, and outputting value according to function. That exactly the point, not to confuse the user with mixed and inconsistent behaviore.

The point of the framework to be general as possible, but to make sense. Even if the use case like you describe is not make any sense now, in the future, for someone for future use case it will make sense.

In the same way, nodes subclassing AbstractGenerator would discard inputs from layers connecting to it.

This is inconsistent to the definition of Node

I agree that the behavioral that you describe is desirable, but, it can be achieve using the current Node unit description, by feed the desire neuron with PoissonGenerator. Meaning, the Object feeder can be hook to any node in any place in the network, just by connecting it to it, just like you do with Monitor object.

In this way, Nodes object maintain is consistency with original description and behavioral of a Suma of the neuron that gets its input from other Nodes and can be place in any layer. In addition it can get an external input from Feeders / Generator.

[k-chaney]

One thing we can look towards is something like the AER type representation as input. Such as what is output from the DVS and other sensors like that. At the end of the day that would be equivalent to a sparse array representation. It should be pretty straight forward on the Input side as pytorch supports sparse tensors internally.

[djsaunde]

It should be pretty straight forward on the Input side as pytorch supports sparse tensors internally.

Yeah, mostly agree except that PyTorch's support for sparse tensor ops is not super strong. But it would be great to look into this.

[k-chaney]

I will look into this. There's some documentation on what exists for sparse and what doesn't exist for sparse. pytorch/pytorch#8853

[SimonInParis]

IMO, encoders should not be called on user side, except for specific needs.
What if the 'Network' was responsible for both spikes encoding and assigning ?
ex.:
model = Network(encoder='poisson', decoder = 'average_spikes', assignment='auto-assign')

[mahsa-ebrahimian]

decoder = 'average_spikes', assignment='auto-assign')

This is exactly my requirement, I need to use temporal encoding scheme for my SNN model to develop a supervised model, but I am unable to find any tool to handle this encoding part. I noticed that there are only three encoders in the repository.

[Hananel-Hazan]

@mahsa-ebrahimian, please use the encoders in BindsNET to convert static data like image to spike trains using the schemes here. You can find how to use them in the examples.

The open issue here is on the efficiency and memory utilization of the encoders only.