mirror of
https://github.com/helm/chartmuseum.git
synced 2026-02-05 15:45:50 +01:00
0e9d2e36ea
Index regeneration before this PR was causing a concurrency pile-up when many requests were coming in at a fast pace. This was causing the time to return /index to always increase. The behavior was observed on a high latency connection to Google Cloud Storage, but would also occur with other network stores. With this PR, the regeneration time is constant and thus the serving of index.yaml is also constant. The implementation takes a general approach to first make initial network requests (to satisfy the object list from storage) and to pile up subsequent requests while the initial one is being completed. The same algorithm is used to update the in-memory cached index. Note that these requests need not be protected by a lock because they are only ever executing on their own. See server.getChartList() for the implementaion of this idea. A refactor was needed to separate the fetch from the diff calculation to allow separate calling of the network heavy operations. While doing so, we also removed redundant calls to storage file list update. Also made small low-hanging fruit style optimisations to index manipulations. Added request ID to all requests for better debugging. This will be visible with the --debug flag. This was indispensable to diagnose complex concurrent processing. To test the before and after state, we have added the use of the locusti.io loadtesting engine. A simple README in the loadtesting/ directory shows how to install locust (with pipenv) and loadtest chartmuseum. This will prove useful in the future. Fixes #18
Loadtesting is made with the excellent Python locust library.
To facilitate installation, this loadtesting subproject uses pipenv.
Install pipenv
pip install pipenv
Install chartmuseum locust loadtesting
cd loadtesting
pipenv install
Start chartmuseum.
Start locust:
# run locust on a running chartmuseum instance
pipenv run locust --host http://localhost:8080
Open your locust console in your browser at http://localhost:8089, and start a new loadtest.