prometheus/docs
1
0
Fork 0
mirror of https://github.com/prometheus/docs.git synced 2026-02-06 00:46:09 +01:00
Code Issues Releases Activity

Fixes after consensus.

Browse Source 
Signed-off-by: bwplotka <bwplotka@gmail.com>
This commit is contained in:
bwplotka
2024-07-11 09:22:23 +01:00
parent 9b7f267f53
commit 125e2b6af3
1 changed files with 19 additions and 18 deletions
Download Patch File Download Diff File Expand all files Collapse all files

37
content/docs/specs/remote_write_spec_2_0.md
View File

@@ -11,7 +11,7 @@ sort_rank: 4
The Remote-Write specification, in general, is intended to document the standard for how Prometheus and Prometheus Remote-Write compatible senders send data to Prometheus or Prometheus Remote-Write compatible receivers.
This document is intended to define a second version of the [Prometheus Remote-Write](./remote_write_spec.md) API with minor changes to protocol and semantics. This second version adds a new Protobuf Message with new features enabling more use cases and wider adoption on top of performance and cost savings. The second version also deprecates the previous Protobuf Message from a [1.0 Remote-Write specification](./remote_write_spec.md#protocol) and adds mandatory [`X-Prometheus-Remote-Write-Accepted-` HTTP response headers](#response-headers) for reliability purposes. Finally, this spec outlines how to implement backwards-compatible senders and receivers (even under a single endpoint) using existing basic content negotiation request headers. More advanced, automatic content negotiation mechanisms might come in a future minor version if needed. For the rationales behind the 2.0 specification, see [the formal proposal](https://github.com/prometheus/proposals/pull/35).
This document is intended to define a second version of the [Prometheus Remote-Write](./remote_write_spec.md) API with minor changes to protocol and semantics. This second version adds a new Protobuf Message with new features enabling more use cases and wider adoption on top of performance and cost savings. The second version also deprecates the previous Protobuf Message from a [1.0 Remote-Write specification](./remote_write_spec.md#protocol) and adds mandatory [`X-Prometheus-Remote-Write-Written-*` HTTP response headers](#required-x-prometheus-remote-write-written--response-headers) for reliability purposes. Finally, this spec outlines how to implement backwards-compatible senders and receivers (even under a single endpoint) using existing basic content negotiation request headers. More advanced, automatic content negotiation mechanisms might come in a future minor version if needed. For the rationales behind the 2.0 specification, see [the formal proposal](https://github.com/prometheus/proposals/pull/35).
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC 2119](https://datatracker.ietf.org/doc/html/rfc2119).
@@ -41,7 +41,7 @@ In this document, the following definitions are followed:
* a `Protobuf Message` (or Proto Message) refers to the [content type](https://www.rfc-editor.org/rfc/rfc9110.html#name-content-type) definition of the data structure for this Protocol. Since the specification uses [Google Protocol Buffers ("protobuf")](https://protobuf.dev/) exclusively, the schema is defined in a ["proto" file](https://protobuf.dev/programming-guides/proto3/) and represented by a single Protobuf ["message"](https://protobuf.dev/programming-guides/proto3/#simple).
* a `Wire Format` is the format of the data as it travels on the wire (i.e. in a network). In the case of Remote-Write, this is always the compressed binary protobuf format.
* a `Sender` is something that sends Remote-Write data.
* a `Receiver` is something that receives Remote-Write data.
* a `Receiver` is something that receives (writes) Remote-Write data. The meaning of `writing` is up to the Receiver e.g. usually it means storing received data in a database, but also just validating, splitting or enhancing it.
* a `Sample` is a pair of (timestamp, value).
* a `Histogram` is a pair of (timestamp, [histogram value](https://github.com/prometheus/docs/blob/b9657b5f5b264b81add39f6db2f1df36faf03efe/content/docs/concepts/native_histograms.md)).
* a `Label` is a pair of (key, value).
@@ -130,33 +130,34 @@ Senders MUST include a user agent header that SHOULD follow [the RFC 9110 User-A
### Response
Receivers ingesting all Samples and Histograms successfully MUST return a [success 2xx HTTP status code](https://www.rfc-editor.org/rfc/rfc9110.html#name-successful-2xx). In such a successful case, the response body from the Receiver SHOULD be empty and the status code SHOULD be [204 HTTP No Content](https://www.rfc-editor.org/rfc/rfc9110.html#name-204-no-content); Senders MUST ignore the response body. The response body is RESERVED for future use.
Receivers that written all data successfully MUST return a [success 2xx HTTP status code](https://www.rfc-editor.org/rfc/rfc9110.html#name-successful-2xx). In such a successful case, the response body from the Receiver SHOULD be empty and the status code SHOULD be [204 HTTP No Content](https://www.rfc-editor.org/rfc/rfc9110.html#name-204-no-content); Senders MUST ignore the response body. The response body is RESERVED for future use.
Receivers MUST NOT return a 2xx HTTP status code if any of the Samples or Histograms were NOT written successfully (e.g. on a [partial write](#partial-write) or a full write rejection). In such a case, the Receiver MUST provide a human-readable error message in the response body. The Receiver's error SHOULD contain information about the amount of the samples being rejected and for what reasons. Senders MUST NOT try and interpret the error message and SHOULD log it as is.
Receivers MUST NOT return a 2xx HTTP status code if any of the pieces of sent data known to the Receiver (e.g. Samples, Histograms, Exemplars) were NOT written successfully (both [partial write](#partial-write) or full write rejection). In such a case, the Receiver MUST provide a human-readable error message in the response body. The Receiver's error SHOULD contain information about the amount of the samples being rejected and for what reasons. Senders MUST NOT try and interpret the error message and SHOULD log it as is.
The following subsections specify Sender and Receiver semantics around headers and different write error cases.
#### Response Headers
#### Required `X-Prometheus-Remote-Write-Written-*` Response Headers
<!---
Rationales: https://github.com/prometheus/prometheus/issues/14359
-->
Upon a successful content negotiation, Receivers ingest the received Samples, Histograms and Exemplars. In such cases, Receivers MUST track the total number of Samples, Histograms or Exemplars that were accepted or ingested. Receivers MUST share those statistics with the Sender as the HTTP response `X-Prometheus-Remote-Write-Accepted-*` headers specified below. Receivers MAY omit those when the value is zero.
Upon a successful content negotiation, Receivers process (write) the received batch of data. Once completed (with success or failure) for each important piece of data (currently Samples, Histograms and Exemplars) Receivers MUST send a dedicated HTTP `X-Prometheus-Remote-Write-Written-*` response header with the precise number of successfully written elements.
Each header value MUST be a single 64-byte integer. The header names MUST be as follows:
* `X-Prometheus-Remote-Write-Accepted-Samples <integer; count of all accepted Samples from this request>`
* `X-Prometheus-Remote-Write-Accepted-Histograms <integer; count of all accepted Histogram samples from this request>`
* `X-Prometheus-Remote-Write-Accepted-Exemplars <integer; count of all accepted Exemplars from this request>`
* `X-Prometheus-Remote-Write-Written-Samples <integer; count of all successfully written Samples from this request>`
* `X-Prometheus-Remote-Write-Written-Histograms <integer; count of all successfully Histogram samples from this request>`
* `X-Prometheus-Remote-Write-Written-Exemplars <integer; count of all successfully Exemplars from this request>`
Simple Senders MAY ignore this feature and rely on HTTP status codes only. However, more advanced Senders (e.g. Prometheus) focused on working with a variety of different Receivers, Protobuf Messages and designed for 100% durability SHOULD use those headers to confirm how many Samples, Histograms and Exemplars were accepted or successfully ingested by the Receiver. Such checks are primarily useful for:
Senders MAY use those headers to confirm which parts of data were successfully written by the Receiver. Common use cases:
* [Partial Write](#partial-write) failure situations: Senders MAY use those headers for more accurate client instrumentation and error handling.
* Additional check against miscommunication or implementation bugs exists between the Sender and Receiver, e.g. accidental decoding of the `io.prometheus.write.v2.Request` payload with `prometheus.WriteRequest` schema (no error on decoding and an empty result).
* Known and common bug in 1.0 Receiver implementations that do not check the content-type header.
* For example, Senders MAY assume [415 HTTP Unsupported Media Type](https://www.rfc-editor.org/rfc/rfc9110.html#name-415-unsupported-media-type) status code when the `io.prometheus.write.v2.Request` request with samples results in 2xx HTTP status code, but no `X-Prometheus-Remote-Write-Accepted-*` response headers from the Receiver (a common issue for the 1.0 Receiver without content-type header check). Senders MUST NOT assume the same for `prometheus.WriteRequest` (for compatibility reasons).
* Broken 1.0 Receiver implementations: Senders SHOULD assume [415 HTTP Unsupported Media Type](https://www.rfc-editor.org/rfc/rfc9110.html#name-415-unsupported-media-type) status code when sending the data using `io.prometheus.write.v2.Request` request and receiving 2xx HTTP status code, but none of the `X-Prometheus-Remote-Write-Written-*` response headers from the Receiver. This is a common issue for the 1.0 Receivers that do not check the `Content-Type` request header; accidental decoding of the `io.prometheus.write.v2.Request` payload with `prometheus.WriteRequest` schema results in empty result and no decoding errors.
* Unknown broken implementations or issues: Senders MAY use those headers to detect broken Sender and Receiver implementations or other problems.
Senders MUST treat the omission of any `X-Prometheus-Remote-Write-Accepted-*` header as either no Samples, Histograms or Exemplars were accepted by the Receiver or the Receiver only supports Remote Write 1.0 protocol.
On the 2xx HTTP status code, Senders CAN assume that a missing write response header means that no such a piece of data was written by the Receiver (count of `0`). value. One exception is that Senders MUST NOT assume the same (empty header means zero) when using the deprecated `prometheus.WriteRequest` Protobuf Message in the request.
Senders MUST NOT assume what Remote Write specification version the Receiver implements from the remote write response headers.
More (optional) headers might come in the future, e.g. when more entities or fields are added and worth confirming.
@@ -165,7 +166,7 @@ More (optional) headers might come in the future, e.g. when more entities or fie
<!---
Rationales: https://github.com/prometheus/proposals/blob/alexg/remote-write-20-proposal/proposals/2024-04-09_remote-write-20.md#partial-writes
-->
Senders SHOULD use Remote-Write to send samples for multiple series in a single request. As a result, Receivers MAY ingest valid samples within a write request that also contains some invalid or otherwise unwritten samples, which represents a partial write case. In such a case, the Receiver MUST return non-2xx status code following the [Invalid Samples](#invalid-samples) and [Retry on Partial Writes](#retries-on-partial-writes) sections.
Senders SHOULD use Remote-Write to send samples for multiple series in a single request. As a result, Receivers MAY write valid samples within a write request that also contains some invalid or otherwise unwritten samples, which represents a partial write case. In such a case, the Receiver MUST return non-2xx status code following the [Invalid Samples](#invalid-samples) and [Retry on Partial Writes](#retries-on-partial-writes) sections.
#### Unsupported Request Content
@@ -276,7 +277,7 @@ message TimeSeries {
// the accuracy of query results.
//
// Note that some receivers might require this and in return fail to
// ingest such samples within the Request.
// write such samples within the Request.
//
// For Go, see github.com/prometheus/prometheus/model/timestamp/timestamp.go
// for conversion from/to time.Time to Prometheus timestamp.
@@ -473,10 +474,10 @@ Because the 1.0 protocol does not use gRPC, breaking it would increase friction
If you use persistent HTTP/1.1 connections, they are pretty close to streaming. Of course, headers have to be re-sent, but that is less expensive than a new TCP set up.
**Why do we send samples in order?**
The in-order constraint comes from the encoding we use for time series data in Prometheus, the implementation of which is optimized for append-only workloads. However, this requirement is also shared across many other databases and vendors in the ecosystem. In fact, [Prometheus with OOO feature enabled](https://youtu.be/qYsycK3nTSQ?t=1321), allows out-of-order writes, but with the performance penalty, thus reserved for rare events. To sum up, Receivers may support out-of-order ingestion, though it is not permitted by the specification. In the future e.g. 2.x spec versions, we could extend content type to negotiate the out-of-order writes, if needed.
The in-order constraint comes from the encoding we use for time series data in Prometheus, the implementation of which is optimized for append-only workloads. However, this requirement is also shared across many other databases and vendors in the ecosystem. In fact, [Prometheus with OOO feature enabled](https://youtu.be/qYsycK3nTSQ?t=1321), allows out-of-order writes, but with the performance penalty, thus reserved for rare events. To sum up, Receivers may support out-of-order write, though it is not permitted by the specification. In the future e.g. 2.x spec versions, we could extend content type to negotiate the out-of-order writes, if needed.
**How can we parallelise requests with the in-order constraint?**
Samples must be in-order _for a given series_. However, even if a Receiver does not support out-of-order ingestion, the Remote-Write requests can be sent in parallel as long as they are for different series. Prometheus shards the samples by their labels into separate queues, and then writes happen sequentially in each queue. This guarantees samples for the same series are delivered in order, but samples for different series are sent in parallel - and potentially "out of order" between different series.
Samples must be in-order _for a given series_. However, even if a Receiver does not support out-of-order write, the Remote-Write requests can be sent in parallel as long as they are for different series. Prometheus shards the samples by their labels into separate queues, and then writes happen sequentially in each queue. This guarantees samples for the same series are delivered in order, but samples for different series are sent in parallel - and potentially "out of order" between different series.
**What are the differences between Remote-Write 2.0 and OpenTelemetry's OTLP protocol?**
[OpenTelemetry OTLP](https://github.com/open-telemetry/opentelemetry-proto/blob/a05597bff803d3d9405fcdd1e1fb1f42bed4eb7a/docs/specification.md) is a protocol for transporting of telemetry data (such as metrics, logs, traces and profiles) between telemetry sources, intermediate nodes and telemetry backends. The recommended transport involves gRPC with protobuf, but HTTP with protobuf or JSON are also described. It was designed from scratch with the intent to support a variety of different observability signals, data types and extra information. For [metrics](https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/metrics/v1/metrics.proto) that means additional non-identifying labels, flags, temporal aggregations types, resource or scoped metrics, schema URLs and more. OTLP also requires [the semantic convention](https://opentelemetry.io/docs/concepts/semantic-conventions/) to be used.