mirror of
https://github.com/helm/chartmuseum.git
synced 2026-02-05 15:45:50 +01:00
0e9d2e36ea
Index regeneration before this PR was causing a concurrency pile-up when many requests were coming in at a fast pace. This was causing the time to return /index to always increase. The behavior was observed on a high latency connection to Google Cloud Storage, but would also occur with other network stores. With this PR, the regeneration time is constant and thus the serving of index.yaml is also constant. The implementation takes a general approach to first make initial network requests (to satisfy the object list from storage) and to pile up subsequent requests while the initial one is being completed. The same algorithm is used to update the in-memory cached index. Note that these requests need not be protected by a lock because they are only ever executing on their own. See server.getChartList() for the implementaion of this idea. A refactor was needed to separate the fetch from the diff calculation to allow separate calling of the network heavy operations. While doing so, we also removed redundant calls to storage file list update. Also made small low-hanging fruit style optimisations to index manipulations. Added request ID to all requests for better debugging. This will be visible with the --debug flag. This was indispensable to diagnose complex concurrent processing. To test the before and after state, we have added the use of the locusti.io loadtesting engine. A simple README in the loadtesting/ directory shows how to install locust (with pipenv) and loadtest chartmuseum. This will prove useful in the future. Fixes #18
40 lines
1.1 KiB
Python
40 lines
1.1 KiB
Python
from locust import HttpLocust, TaskSet
|
|
import tarfile
|
|
import io
|
|
|
|
patch_version = 1
|
|
chart_post_field_name = 'chart'
|
|
|
|
def index(l):
|
|
l.client.get("/index.yaml")
|
|
|
|
def post_new_chart(l):
|
|
global patch_version
|
|
|
|
# Create dummy 'chartmuseum-loadtest' chart package for which we only increment the patch version
|
|
chart_name = 'chartmuseum-loadtest'
|
|
chart_version = '0.0.%d' % patch_version
|
|
patch_version += 1
|
|
chart_fn = '%s-%s.tgz' % (chart_name, chart_version)
|
|
|
|
tgz_buf = io.BytesIO()
|
|
t = tarfile.open(mode = "w:gz", fileobj=tgz_buf)
|
|
chart_content = b'name: %s\nversion: %s\n' % (chart_name.encode('utf8'), chart_version.encode('utf8'))
|
|
tarinfo = tarfile.TarInfo('%s/Chart.yaml' % chart_name)
|
|
tarinfo.size = len(chart_content)
|
|
t.addfile(tarinfo=tarinfo, fileobj=io.BytesIO(chart_content))
|
|
t.close()
|
|
tgz_buf.seek(0)
|
|
|
|
l.client.post('/api/charts', files={chart_post_field_name: (chart_fn, tgz_buf)})
|
|
|
|
|
|
class UserBehavior(TaskSet):
|
|
tasks = {index: 10, post_new_chart: 1}
|
|
|
|
|
|
class WebsiteUser(HttpLocust):
|
|
task_set = UserBehavior
|
|
min_wait = 1000
|
|
max_wait = 3000
|