url: try local port on both IP stacks
Signed-off-by: Mathieu Tortuyaux <mtortuyaux@microsoft.com>
url: support both IPv4 and IPv6
This defines a wrapper that will try in paralell both IPv4 and IPv6 when
the provider declares those two IPs.
Signed-off-by: Mathieu Tortuyaux <mtortuyaux@microsoft.com>
url: support both IPv4 and IPv6
This defines a wrapper that will try in paralell both IPv4 and IPv6 when
the provider declares those two IPs.
Signed-off-by: Mathieu Tortuyaux <mtortuyaux@microsoft.com>
resource: add race condition fix using sync.Mutex
This aligns the aws-sdk-go-v2 version with what we used to do for aws-sdk-go,
although the error matching is string-based now because the SDK changed its
error reporting.
When we're not on EC2 we directly fetch with anonymous credentials, so there is
no point retrying. On errors other than EC2 role fetch, such as when key is
invalid, it also doesn't make sense to retry.
Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>
Update the code from aws-sdk-go to aws-sdk-go-v2, aws-sdk-go is out-of-support
since July 2025. I've tried to preserve the existing behavior as best I could:
- if we're running on EC2 we use ec2rolecreds first, with a fallback to anonymous
credentials
- use anonymous credentials when outside EC2
- use IMDS for region hints
One subtle change appears to be that the S3 code does not tolerate leading
slashes in key names any more, so we need to strip it out explicitly.
Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>
Those errors would be triggered after processing already happened so
they are unlikely and we should not act on them.
Fixes lint:
```
Error: internal/resource/http.go:319:19: Error return value of `resp.Body.Close` is not checked (errcheck)
Error: internal/resource/url.go:340:12: Error return value of `l.Close` is not checked (errcheck)
Error: internal/resource/url.go:380:24: Error return value of `dataReader.Close` is not checked (errcheck)
Error: internal/resource/url.go:620:33: Error return value of `downloadStream.Body.Close` is not checked (errcheck)
Error: internal/resource/url.go:654:26: Error return value of `decompressor.Close` is not checked (errcheck)
```
The linter found staticcheck and errorcheck issues. Deferred statements now use join to combine any new error with an existing one, preventing error information from being lost. Other minor error checks, like for printing text and flushing data have also been addressed.
When establishing a network connection random port selection from the
Kernel for local address is enough in 99.9% of the case.
For the 0.1%, let's add a way to customize the local port used.
Signed-off-by: Mathieu Tortuyaux <mtortuyaux@microsoft.com>
When running in GCE, we assumed that we should always perform
authenticated GS fetches. However, these can fail if the VM is not
associated with a service account, even if the object being fetched is
publicly readable:
error while reading content from ...: metadata: GCE metadata "instance/service-accounts/default/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdevstorage.read_only" not defined
Query the VM's service account scopes first, and if that query fails
(presumably because there is no service account), fall back to anonymous
access.
For S3 access point ARNs, we derive the access point region from the
region field in the ARN, allowing us to query the correct AWS partition.
For regular S3 object ARNs, the region field is not used, so we assume
the same partition Ignition is running in, or "aws" if running outside
AWS.
However, object ARNs include a valid partition field. Compute a region
hint from that, and use it to ask the partition for the correct bucket
region. That allows object ARNs to reference objects outside the current
partition (when running in AWS) or outside the "aws" partition (when
running outside AWS).
Support S3 access point URLs in ARN format as a source.
This allows valid, opaque S3 URLs such as
`s3:arn:aws:s3:us-west-2:123456789012:accesspoint/test/object`
Being able to use this format will allow S3 URLs on different
partitions and lays the foundation to potentially support
multi-region access points in the future.
Fixes https://github.com/coreos/ignition/issues/1091
Signed-off-by: Zeleena Kearney <zeleenak@lyft.com>
The value of initialBackoff is increased before the wait is ever
performed. This means that we never actually wait for the default
initialBackoff time. Move it to after the wait but also increase the
current default to mirror what was happening with the previous
functionality.
Fixes: https://github.com/coreos/ignition/issues/1117
Fix Ignition userdata fetch failure when IMDSv2 service is set to
required on AWS. By default, use IMDSv2 by making a PUT request to
the endpoint to fetch the session token. If the request fails on a
404, fall back to IMDSv1.
CAs are re-written into the fetched config as a base64 string to allow
cached reads to not require an additional fetch. When compression
support was added during the resource consolidation refactor for the
3.1.0 spec this interaction was missed.
The blackbox tests did not catch this as they were always clearing the
config cache on each run.
Right now Ignition will incorrectly report that it needs networking when
an empty scheme is given. Modify the `UrlNeedsNet` function to allow
both `data` & empty schemes.
A common case that could hit this is a config that writes a file with
empty contents.
Add a new `fetch-offline` stage which can optionally be run before the
`fetch` stage. The major difference between the two is that the former
tries to operate in offline mode: if it encounters any resource which
requires network access, it quietly exits, creating a stamp file on the
way out.
This allows OS integrators to make use of this to only bring up
networking if Ignition actually requires it. It does not solve the
harder problem of "partially" up networking, where some fetches might
succeed and some others might fail. However, it provides an incremental
step to get there by reusing the same signalling mechanism.
Dedupe the "remote resource" type from the spec. This came up too during
review of the new HTTP headers. They all had more or less the same
fields in common ("source", "verification", "compression", and more
recently "httpHeaders").
One noticeable spec level change from this is that some places now
support compression where they didn't before. Might seem odd to support
compression for e.g. certificate authorities, though it does make the
spec more consistent throughout.
Co-authored-by: Jonathan Lebon <jonathan@jlebon.com>
Set defaults first, then override them with caller-specified headers.
Fixes default Accept header causing 406 Not Acceptable when fetching
userdata on Packet.
Fixes: 75d3e2da74 ("Rework fetch/resource to automatically append required headers")
This commit adds an ability to use HTTP headers for config-reference,
ca-reference and file-contents structures. This can be useful when a
remote server requires authentication and we have to provide a token
in the header.
Rather than having each platform provider pass down `resource.ConfigHeaders`,
automatically inject it. Then each platform only needs to pass
additional headers, which in most cases is none.
Unexport `configHeaders` then.
This also avoids mutating a global variable (!). Queue obligatory
crying for 3 seconds that this code isn't Rust.
Prep for further work around headers.
We shouldn't write a tmpfile when trying to fetch directly into memory.
Rework FetchToBuffer to be like Fetch instead of calling it. This allows
use to handle the special case of s3 (which needs a WriterAt +
ReadSeeker) separately from the other cases which only need a Writer.
Most of the fetchFrom* functions don't need more than the Writer
interface, s3 being the exception. Specify only the needed interfaces
instead of *os.File.
