The GetBuilds function, available on both HCL2 and legacy JSON
configuration objects, used to return the Build interface.
This typing by interface is not useful in this instance, since all the
uses of `GetBuilds' are self-contained within Packer, and we're never
using any other implementation for it than `*CoreBuild`.
We've been relying on the dynamic type for all the builds being
*CoreBuild in some places of the code, so to avoid potential surprises
in the future, we'll change the signature now so it returns only
concrete types.
In the current state, a Packer build that succeeds but fails to push its
metadata to HCP for reasons other than a lack of artifact will always
succeed from the perspective of a user invoking `packer build`.
This can be a bit misleading, as users may expect their artifacts to
appear on HCP Packer if their build succeeded on Packer Core, so this
commit changes this behaviour, instead reporting HCP errors as a real
error if the build failed, so packer returns a non-zero error code if
this happens.
As we're trying to move away from gob for serialising data over the
wire, this commit adds the capability for Packer to pick dynamically
between gob or protobuf for the serialisation format to communicate with
plugins.
As it stands, if all the plugins discovered are compatible with
protobuf, and we have not forced gob usage, protobuf will be the
serialisation format picked.
If any plugin is not compatible with protobuf, gob will be used for
communicating with all the plugins that will be used over the course of
a command.
For all the commands that call Initialise, we introduce a new flag:
UseSequential.
This disables DAG scheduling for evaluating datasources and locals as a
fallback to the newly introduced DAG scheduling approach.
`hcl2_upgrade` is a special case here, as the template is always JSON,
there cannot be any datasource, so the DAG in this case becomes
meaningless, and is not integrated in this code path.
Since we added the capability for the plugin discovery to ignore
installed pre-releases of plugins, we give that capacity to the validate
and build commands, through a new command-line flag: release-only.
When running a build with HCP Packer enabled, Packer attempts to push
the build status to HCP.
If the build fails, we update the status to BUILD_FAILED, and that's the
end of it.
If however the build succeeds, Packer attempts to get the HCP artifact
from the builder, which will only succeed if the builder supports it.
Otherwise, we'll get either nil, or an artifact type that is not
compatible with what is expected for HCP support.
When either of those happens, we warn that the builder may not support
HCP Packer at all, so users are aware of the problem.
However, when the error was introduced, it only looked at the fact that
an error was produced, independently of the type of error. This caused
legitimate errors while building to be reported as potential
incompatibility between the builder and HCP, which was confusing to
users.
This commit changes this by introducing a new error type, only produced
when the artifact either is nil, or failed to be deserialised into a HCP
artifact, which lets us produce the incompatibility warning with more
accuracy.
With the removal of vendored_plugins we are no longer interested in tacking bundled plugins usage.
For plugins such as file, null, or anything bultin into Packer we don't track because there is no
way to install them outside of Packer, for now.
The enumeration for FlagSetFlags, which presumably was added when the
Meta structure was introduced, aims to pre-populate the flagset for a
subcommand with a series of arguments.
However, despite it being documented, it is actually not used, and
therefore can safely be removed from the codebase.
* Updating the license from MPL to Business Source License
Going forward, this project will be licensed under the Business Source License v1.1. Please see our blog post for more details at https://hashi.co/bsl-blog, FAQ at https://hashi.co/license-faq, and details of the license at www.hashicorp.com/bsl.
* Update copyright file headers to BUSL-1.1
---------
Co-authored-by: hashicorp-copywrite[bot] <110428419+hashicorp-copywrite[bot]@users.noreply.github.com>
Since bundled plugins will be removed in an upcoming version of Packer,
this commit adds a new warning message whenever a template uses one such
plugin.
This warning has been implemented on build, validate, console and the
inspect subcommands.
In addition to warning about the upcoming change and potential issue
this will cause, this warning message proposes solutions to the user so
they know what they'll have to do in order not to rely on those bundled
plugins later.
Fingerprints are how we link a packer build to an iteration on HCP.
These are computed automatically from the Git SHA in the current state,
and are unique to the bucket/iteration.
The main problem with this approach is that while sound in theory, it
quickly falls apart when users want to run the same build configuration
twice, but expect a new image to be created.
With the current model, this fails, as the iteration with the current
SHA already exists.
While this is solvable through environment variables, or by committing a
change to the repository, we think this is not clear enough, and causes
an extra step to what should otherwise be a simple process.
Therefore, to lower the barrier of entry into HCP, we change this
behaviour with this commit.
Now, fingerprints are randomly generated ULIDs instead of a git SHA, and
a new one is always generated, unless one is already specified in the
environment.
This makes continuation of an existing iteration a conscious choice
rather than something automatic, and virtually eliminates conflicts such
as the ones described above.
This commit irons out one of the pain points of the HCP rework by
introducing a HCPPublisher interface, implemented both by the JSON Core,
and the HCL2 PackerConfig, which keeps a map of the build names used by
Packer to the build names pushed on HCP.
This in turn lets us go back to the old implementation of the GetBuilds
function, which returns a list of (filtered) builds, and eventually an
error if something went wrong while processing.
* Update validation options for undeclared variables
In an effort to help users move from JSON to HCL2 templates the support for
variable definitions files are being updated to ignore undeclared
variable warnings on build execution. For legacy JSON templates builds no
warnings are displayed when var-files contain undeclared variables.
Since preferred mode HCL2 templates is to be explicit with variable
declarations - they must be declared to be used - validation for
undeclared variables still warns when running `packer validate`. A new
flag has been added to the validate command that can be used to disable
undeclared variable warnings.
* Update validation test for unused variables
Example Run
```
~> go run . validate -no-warn-undeclared-var -var-file
command/test-fixtures/validate/var-file-tests/undeclared.pkrvars.hcl
command/test-fixtures/validate/var-file-tests/basic.pkr.hcl
The configuration is valid.
~> go run . validate -var-file
command/test-fixtures/validate/var-file-tests/undeclared.pkrvars.hcl
command/test-fixtures/validate/var-file-tests/basic.pkr.hcl
Warning: Undefined variable
The variable "unused" was set but was not declared as an input variable.
To declare variable "unused" place this block in one of your .pkr.hcl
files,
such as variables.pkr.hcl
variable "unused" {
type = string
default = null
}
The configuration is valid.
~> go run . build -var-file
command/test-fixtures/validate/var-file-tests/undeclared.pkrvars.hcl
command/test-fixtures/validate/var-file-tests/basic.pkr.hcl
file.chocolate: output will be in this color.
Build 'file.chocolate' finished after 744 microseconds.
==> Wait completed after 798 microseconds
==> Builds finished. The artifacts of successful builds are:
--> file.chocolate: Stored file: chocolate.txt
```
* Rename Strict field to WarnOnUndeclaredVar
The field name Strict is a bit vague since it is only used for
checking against undeclared variables within a var-file definition.
To mitigate against potential overloading of this field it is
being renamed to be more explicit on its usage.
* command/build: Add warn-on-undeclared-var flag
Now that the default behaviour is to not display warnings for undeclared variables
an optional flag has been added to toggle the old behaviour.
```
~> go run . build -warn-on-undeclared-var -var-file command/test-fixtures/validate/var-file-tests/undeclared.pkrvars.hcl command/test-fixtures/validate/var-file-tests/basic.pkr.hcl
Warning: Undefined variable
The variable "unused" was set but was not declared as an input variable.
To declare variable "unused" place this block in one of your .pkr.hcl files,
such as variables.pkr.hcl
variable "unused" {
type = string
default = null
}
file.chocolate: output will be in this color.
Build 'file.chocolate' finished after 762 microseconds.
==> Wait completed after 799 microseconds
==> Builds finished. The artifacts of successful builds are:
--> file.chocolate: Stored file: chocolate.txt
```
This commit reorganises the code for both the registry/API and the
Orchestrator/Registry.
The main difference with the previous version is how stuff is exposed.
Now we only expose a Registry interface to the outside (previously named
Orchestrator), which has several implementations: null is the default,
and is returned if HCP is not enabled.
The other implementations being HCL/JSON, both private to the hcp
sub-package.
The api (previously `registry') is the set of functionality that
abstracts and calls the HCP API.
It was meant to be merged with the `hcp' package, but because of a
dependency loop with the datasources, both are separated for now.
As part of the work to displace everything related to HCP from scattered
places around the Packer code, we move it all to an hcp package.
This in turn reduces the amount of code that the commands have to
integrate, and leaves the HCP details to its own enclave.
To be able to track HCP builds during a Packer build, we need to
propagate the name used for registering the build to HCP.
This works out-of-the-box for JSON templates, as the build's name is
always the builder's, so there's no confusion on that.
For HCP templates however, there's a possibility for a template to
define a name for the build, which is returned by a CoreBuild's Name()
function, in addition to the source's Type/Name.
The builds are registered to HCP with the source.String() function
however, which does not contain the build's name. This prevents any
build containing a name to work, as their name/source.String does not
match.
The name we registered the build with is stored in a CoreBuild as the
Type attribute, but cannot be safely accessed, as builds are types
within the command as packersdk.Build, which only exposes Name() for
this purpose.
In order to circumvent this problem, and as a way to present a prototype
of solution that will likely have to be discussed before we merge it,
this commit hotfixes the issue.
The HCP error messages were sometimes a bit terse in terms of the
information it bubbles up to the user.
This made them useful for people who already knew what to look for, but
newcomers would almost certainly be lost because of the lack of
information in those.
To improve on this situation, we reword most of those error messages in
this commit.
When a template with some builds to run ends its GetBuilds with an
error, and no builds produced, we can exit immediately without printing
more errors.
When a `packer build' is run on a config that does not define anything
to be built, the process would run normally, and only state that no
artifacts were produced after builds were run successfully.
This is misleading as we reach the end of the command without having a
proper warning that nothing was even attempted to be built.
This commit adds a warning when that happens, and leaves immediately
after.
Before this commit, some HCP-related logic was embedded in the parser,
making all of it common to all commands.
This complexifies the logic as some parts rely on the template being in
a HCP-valid environment, and the datasources need to be evaluated in
order for some operations to be executed.
To simplify this and avoid those pitfalls, we move this logic to its own
set of functions that will be performed after the parsing is done, on
any valid HCL2 or JSON template.
Since the GetConfig methods are common to all the commands of Packer
that rely on a config to work, we move them from build.go to meta.go in
order to make the intent clearer.
* Add working registry pkg
* Add custom error for handling the loading of PAR environment variables
* Working Publish to Build, with proper error handling for bucket names
* Update hcp-sdk-go to use branch instead of mod replace directive
* Update Packer build status configuration
* Add support for HCP_PACKER_BUILD_FINGERPRINT env
* Add support for publishing one or more PARtifacts from a single build
* add git shas to this branch
* Add ability to set provider name if available
* Add working RegistryBuilder type
* Add RegistryPostProcessor as wrapper post-processor
* When in PAR mode a empty RegistryPostProcessor is added to the end of
the post-processor list to publish all final image data.
* Add support for updating a build from PAR that is not in a DONE state
* Fix a small issue with creation the initial builds for an empty
iteration.
* Add PAR URL to post-processor display
* Implement hcp_packer_registry block (#11168)
* Update vendored Amazon plugin to v1.0.1-dev
* Fix panic when running a Packer registry build in a clean directory
* Remove the publishing of post-processor metadata from the registry
post-processor.
* Remove metadata add from registry_builder
* Update registry builder to skip a build that was found to be DONE
Co-authored-by: Megan Marsh <megan@hashicorp.com>
Co-authored-by: Sylvia Moss <moss@hashicorp.com>
This adds the new `required_plugins` block to be nested under the packer block.
Example:
```hcl
packer {
required_plugins {
aws = {
version = ">= 2.7.0"
source = "azr/aws"
}
azure = ">= 2.7.0"
}
}
```
For example on darwin_amd64 Packer will install those under :
* "${PACKER_HOME_DIR}/plugin/github.com/azr/amazon/packer-plugin-amazon_2.7.0_x5.0_darwin_amd64"
* "${PACKER_HOME_DIR}/plugin/github.com/hashicorp/azure/packer-plugin-azure_2.7.0_x5.0_darwin_amd64_x5"
+ docs
+ tests
The initialization of packer core in JSON also validates that `null` variables were set, except in the case of `packer validate --syntax-only` , but after the refactor to allow to have all commands work with HCL2 and JSON this subtlety was lost.
This refactors the initialisation of the core in order to allow to have `packer validate --syntax-only` not error in case a variable is not set. Since these calls are refactored this works for HCL2 too.
fix #9478
* refactor so that json and hcl2 templates are both prepared in the same place in the build call, to make code easier to reason about. Remove overly verbose error output which isn't useful in vast majority of cases
* fix tests
* check err msg
* hcl2template.PackerConfig.GetBuilds: raise a diagnostic in case the packer core build perpare call errors
Co-authored-by: Adrien Delorme <adrien.delorme@icloud.com>
