middleware / gnocchi

This simplifies the code base by just allowing to process new measures sack per
sack with the existing process_new_measures_for_sack() or to use the new
refresh_metrics() who takes a list of metrics. Since the list of metric must be
known, there's no need to use list_metric_with_measures_to_process() anymore,
so it can be removed.

Since some of the tests (test_rest) need to process all sacks now, reduce the
number of sack to a small number in order for each test not to take 10s.

Closes #600

Some of the tests in another scenario use those id already, making a potential
conflict.

SOURCES.txt generated by this version is wrong and broke bdist*
commands.

This change ignores these versions.

This allows in this first step to retrieve and store the unaggregated
timeseries all at once.

This user_id is also used in the resource-aggregation scenario, which might
conflict when tests are run at the same time.

This makes it safer to call it without taking care of the lock externally.

This adds a new method process_measures_for_sack to incoming storage driver. It
allows to read an entire sack for new measure rather than individual metrics.
This avoids doing 2 listing to process new metrics and only does one to load
the measures.

The process_new_measures(metrics) is kept for the refresh_metric() use case for
now. Some refactoring might be possible after this patch.

The S3 storage driver is a bit modified to store the incoming measures in
<sack>/<metric>/ rather than <sack><metric>/ so it's easier to list incoming
sacks and metrics.

This allows to return a dict that can be used by process_measure_for_metrics

This makes sure that all operations are batched, including the _list_split_key
that is now done only once for a metric!

Rather than emitting store and delete on each aggregation computing, this
batches them all at once for a metric by returning the operations from
`_add_measures` directly.

Whilst working on the Reproducible Builds effort [0], we noticed
that gnocchi could not be built reproducibly as it iterates over
a Python set when generating documentation.

This was originally filed in Debian as #892419.

 [0] https://reproducible-builds.org/
 [1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=892419

Signed-off-by: Chris Lamb <lamby@debian.org>

This factorize the call to _list_split_keys() so all aggregations that needs
the list of keys are retrieved in just one call.

This allows to batch storage and deletion of splits for a metric.

This is already what is passed as argument by gnocchi.storage, but there were
no test doing any assert. It does not really matter as the final
AggregatedTimeSerie is built from concatenating splits, but better be safe than
sorry.

All pbr interesting features (the seutp.cfg) have been implemented in
setuptools since version 30.3.0 (8 Dec 2016).
Also pbr will break us soon and force us to found a new solution
to generate things during setup.py steps.
https://github.com/gnocchixyz/gnocchi/issues/755

When you use pbr you cannot override most of the cmdclass. So we cannot
make our hooks working anymore.

Generation of ChangeLog and AUTHORS are done during egg_info.
gnocchi-api script is created on the fly with the right python header

Package file listing and package version is done by the pypa maintained
setuptools_scm.

This may impact downstream packaging as setuptools versions are a bit
old:
* xenial: 20.7
* zesty: 33.1.1
* centos7/rhel7: 0.9.8 ... really ?
* rdo queen: 22.0
* rdo rocky plan to 38.6

Closes-bug: #755

Splits are either retrieve for reading from:

- the REST API, where the splits are actually listed before being retrieved, we
  don't care about the metric existence or not, None is enough.

- on rewriting the split (WRITE_FULL) and in that case we know the metric
  exists

If data corruption occurs, the _get_splits_and_unserialize() method might not
return the same number of arguments that it received. It's not a big deal when
concatenating the splits to retrieve the data for reading. However, when
requesting a bunch of splits for rewrite, that's a problem because the list is
zipped with other data on iteration, possibly corrupting the results.

Raising an exception at this point is damageable as when multiple SplitKeys are
being fetched at the same time, only one being absent will cancel the call. The
caller is already solid and know show to handle None as a return value so just
use None to indicate the split is non-existing.

The list of keys passed as argument now embeds also its companion Aggregation
object so it'll be possible to retrieve splits from different aggregations in
one single call.

This decision is now available for the upper layer.

This patch also fixes the S3 driver which was not checking properly for metric
existence. Listing objects never returns a NoSuchKey error, at worse it returns
an empty content list. We therefore need an extra check with a HEAD request on
the unaggregated timeseries to be sure that the metric exists. It therefore
needs to be created at _get_or_create_unaggregated_timeseries_unbatched() like
the other drivers do.

This was not caught earlier because of the return empty by default mechanism,
but it is correctly tested.

Related to #784

The current code only delete one metric files :(

Closes #773

By default tox built a sdist with the python OS packages. We already
skip install/develop phase and manually use "pip install -e"
The sdist is useless and skipping it allow to run tox on old OS
that doesn't have a recent setuptools.

We use environment marker in requirement so we need at least
setuptools 20.6.8.

This must be a setup_requires because pbr use setuptools to read the
requirement.