Hi there,
I came accross ExpandingHist, as it is also used inside Quantiles. Unfortunately it does not support merge and hence cannot be massively parallelized. That is a huge pity.
Looking at the algorithm, it seems like this can easily be generalized to support merge. We just need to change:
- have a fixed a priori known width which we always double (instead of currently the first width seen)
- have a fixed grid of starting points for every power of the width (instead of currently using the first point seen)
Then it should be possible to merge two different ExpandingHist, as all aggregations align over the same grid so that they can be merged with the same logic.
Have I overlooked something?
Quantiles are one the most frequently used "advanced" statistics at all. It would be so awesome if they support massive parallelization.
Hi there,
I came accross ExpandingHist, as it is also used inside Quantiles. Unfortunately it does not support merge and hence cannot be massively parallelized. That is a huge pity.
Looking at the algorithm, it seems like this can easily be generalized to support merge. We just need to change:
Then it should be possible to merge two different ExpandingHist, as all aggregations align over the same grid so that they can be merged with the same logic.
Have I overlooked something?
Quantiles are one the most frequently used "advanced" statistics at all. It would be so awesome if they support massive parallelization.