getsops/sops
1
0
Fork 0
mirror of https://github.com/getsops/sops.git synced 2026-02-05 12:45:21 +01:00
Code Issues Activity

Add and use rotate, edit, and set subcommands.

Browse Source 
Signed-off-by: Felix Fontein <felix@fontein.de>
This commit is contained in:
Felix Fontein
2023-11-05 17:36:46 +01:00
parent 2ae16f5457
commit c3abac2a1d
3 changed files with 449 additions and 44 deletions
Download Patch File Download Diff File Expand all files Collapse all files

67
README.rst
View File

@@ -96,12 +96,12 @@ separated, in the **SOPS_PGP_FP** env variable.
Note: you can use both PGP and KMS simultaneously.
Then simply call ``sops`` with a file path as argument. It will handle the
Then simply call ``sops edit`` with a file path as argument. It will handle the
encryption/decryption transparently and open the cleartext file in an editor
.. code:: sh
$ sops mynewtestfile.yaml
$ sops edit mynewtestfile.yaml
mynewtestfile.yaml doesn't exist, creating it.
please wait while an encryption key is being generated and stored in a secure fashion
file written to mynewtestfile.yaml
@@ -164,7 +164,7 @@ Given that, the only command a SOPS user needs is:
.. code:: sh
$ sops <file>
$ sops edit <file>
`<file>` will be opened, decrypted, passed to a text editor (vim by default),
encrypted if modified, and saved back to its original location. All of these
@@ -184,7 +184,7 @@ the example files and pgp key provided with the repository::
$ git clone https://github.com/getsops/sops.git
$ cd sops
$ gpg --import pgp/sops_functional_tests_key.asc
$ sops example.yaml
$ sops edit example.yaml
This last step will decrypt ``example.yaml`` using the test private key.
@@ -480,35 +480,33 @@ separated list.
SOPS will prompt you with the changes to be made. This interactivity can be
disabled by supplying the ``-y`` flag.
Command Line
************
``rotate`` command
******************
Command line flag ``--add-kms``, ``--add-pgp``, ``--add-gcp-kms``, ``--add-azure-kv``,
``--rm-kms``, ``--rm-pgp``, ``--rm-gcp-kms`` and ``--rm-azure-kv`` can be used to add
and remove keys from a file.
These flags use the comma separated syntax as the ``--kms``, ``--pgp``, ``--gcp-kms``
and ``--azure-kv`` arguments when creating new files.
The ``rotate`` command generates a new data encryption key and reencrypt all values
with the new key. At te same time, the command line flag ``--add-kms``, ``--add-pgp``,
``--add-gcp-kms``, ``--add-azure-kv``, ``--rm-kms``, ``--rm-pgp``, ``--rm-gcp-kms``
and ``--rm-azure-kv`` can be used to add and remove keys from a file. These flags use
the comma separated syntax as the ``--kms``, ``--pgp``, ``--gcp-kms`` and ``--azure-kv``
arguments when creating new files.
Note that ``-r`` or ``--rotate`` is mandatory in this mode. Not specifying
rotate will ignore the ``--add-*`` options. Use ``updatekeys`` if you want to
add a key without rotating the data key.
Use ``updatekeys`` if you want to add a key without rotating the data key.
.. code:: sh
# add a new pgp key to the file and rotate the data key
$ sops -r -i --add-pgp 85D77543B3D624B63CEA9E6DBC17301B491B3F21 example.yaml
$ sops rotate -i --add-pgp 85D77543B3D624B63CEA9E6DBC17301B491B3F21 example.yaml
# remove a pgp key from the file and rotate the data key
$ sops -r -i --rm-pgp 85D77543B3D624B63CEA9E6DBC17301B491B3F21 example.yaml
$ sops rotate -i --rm-pgp 85D77543B3D624B63CEA9E6DBC17301B491B3F21 example.yaml
Direct Editing
**************
Alternatively, invoking ``sops`` with the flag **-s** will display the master keys
Alternatively, invoking ``sops edit`` with the flag **-s** will display the master keys
while editing. This method can be used to add or remove ``kms`` or ``pgp`` keys under the
``sops`` section. Invoking ``sops`` with the **-i** flag will perform an in-place edit
instead of redirecting output to ``stdout``.
``sops`` section.
For example, to add a KMS master key to a file, add the following entry while
editing:
@@ -620,7 +618,7 @@ When creating a new file, you can specify the encryption context in the
.. code:: sh
$ sops --encryption-context Environment:production,Role:web-server test.dev.yaml
$ sops edit --encryption-context Environment:production,Role:web-server test.dev.yaml
The format of the Encrypt Context string is ``<EncryptionContext Key>:<EncryptionContext Value>,<EncryptionContext Key>:<EncryptionContext Value>,...``
@@ -651,13 +649,16 @@ Key Rotation
~~~~~~~~~~~~
It is recommended to renew the data key on a regular basis. ``sops`` supports key
rotation via the ``-r`` flag. Invoking it on an existing file causes ``sops`` to
reencrypt the file with a new data key, which is then encrypted with the various
rotation via the ``rotate`` command. Invoking it on an existing file causes ``sops``
to reencrypt the file with a new data key, which is then encrypted with the various
KMS and PGP master keys defined in the file.
Add the ``-i`` option to write the rotated file back, instead of printing it to
stdout.
.. code:: sh
$ sops -r example.yaml
$ sops rotate example.yaml
Using .sops.yaml conf to select KMS, PGP and age for new files
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -741,7 +742,7 @@ Creating a new file with the right keys is now as simple as
.. code:: sh
$ sops <newfile>.prod.yaml
$ sops edit <newfile>.prod.yaml
Note that the configuration file is ignored when KMS or PGP parameters are
passed on the SOPS command line or in environment variables.
@@ -847,7 +848,7 @@ For example:
.. code:: sh
$ sops --shamir-secret-sharing-threshold 2 example.json
$ sops edit --shamir-secret-sharing-threshold 2 example.json
Alternatively, you can configure the Shamir threshold for each creation rule in the ``.sops.yaml`` config
with ``shamir_threshold``:
@@ -880,7 +881,7 @@ with ``shamir_threshold``:
- pgp:
- fingerprint5
And then run ``sops example.json``.
And then run ``sops edit example.json``.
The threshold (``shamir_threshold``) is set to 2, so this configuration will require
master keys from two of the three different key groups in order to decrypt the file.
@@ -1348,7 +1349,7 @@ The command below creates a new file with a data key encrypted by KMS and PGP.
.. code:: sh
$ sops --kms "arn:aws:kms:us-west-2:927034868273:key/fe86dd69-4132-404c-ab86-4269956b4500" --pgp C9CAB0AF1165060DB58D6D6B2653B624D620786D /path/to/new/file.yaml
$ sops edit --kms "arn:aws:kms:us-west-2:927034868273:key/fe86dd69-4132-404c-ab86-4269956b4500" --pgp C9CAB0AF1165060DB58D6D6B2653B624D620786D /path/to/new/file.yaml
Encrypting an existing file
~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -1448,12 +1449,12 @@ Set a sub-part in a document tree
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
SOPS can set a specific part of a YAML or JSON document, by providing
the path and value in the ``--set`` command line flag. This is useful to
set specific values, like keys, without needing an editor.
the path and value in the ``set`` command. This is useful to set specific
values, like keys, without needing an editor.
.. code:: sh
$ sops --set '["app2"]["key"] "app2keystringvalue"' ~/git/svc/sops/example.yaml
$ sops set ~/git/svc/sops/example.yaml '["app2"]["key"]' '"app2keystringvalue"'
The tree path syntax uses regular python dictionary syntax, without the
variable name. Set to keys by naming them, and array elements by
@@ -1461,13 +1462,13 @@ numbering them.
.. code:: sh
$ sops --set '["an_array"][1] "secretuser2"' ~/git/svc/sops/example.yaml
$ sops set ~/git/svc/sops/example.yaml '["an_array"][1]' '"secretuser2"'
The value must be formatted as json.
.. code:: sh
$ sops --set '["an_array"][1] {"uid1":null,"uid2":1000,"uid3":["bob"]}' ~/git/svc/sops/example.yaml
$ sops set ~/git/svc/sops/example.yaml '["an_array"][1]' '{"uid1":null,"uid2":1000,"uid3":["bob"]}'
Showing diffs in cleartext in git
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -1742,7 +1743,7 @@ when creating a new file:
.. code:: sh
$ sops --pgp "E60892BB9BD89A69F759A1A0A3D652173B763E8F,84050F1D61AF7C230A12217687DF65059EF093D3,85D77543B3D624B63CEA9E6DBC17301B491B3F21" mynewfile.yaml
$ sops edit --pgp "E60892BB9BD89A69F759A1A0A3D652173B763E8F,84050F1D61AF7C230A12217687DF65059EF093D3,85D77543B3D624B63CEA9E6DBC17301B491B3F21" mynewfile.yaml
Threat Model
------------

400
cmd/sops/main.go
View File

@@ -898,6 +898,406 @@ func main() {
return toExitError(err)
},
},
{
Name: "rotate",
Usage: "generate a new data encryption key and reencrypt all values with the new key",
ArgsUsage: `file`,
Flags: append([]cli.Flag{
cli.BoolFlag{
Name: "in-place, i",
Usage: "write output back to the same file instead of stdout",
},
cli.StringFlag{
Name: "output",
Usage: "Save the output after decryption to the file specified",
},
cli.StringFlag{
Name: "input-type",
Usage: "currently json, yaml, dotenv and binary are supported. If not set, sops will use the file's extension to determine the type",
},
cli.StringFlag{
Name: "output-type",
Usage: "currently json, yaml, dotenv and binary are supported. If not set, sops will use the input file's extension to determine the output format",
},
cli.StringFlag{
Name: "encryption-context",
Usage: "comma separated list of KMS encryption context key:value pairs",
},
cli.StringFlag{
Name: "add-gcp-kms",
Usage: "add the provided comma-separated list of GCP KMS key resource IDs to the list of master keys on the given file",
},
cli.StringFlag{
Name: "rm-gcp-kms",
Usage: "remove the provided comma-separated list of GCP KMS key resource IDs from the list of master keys on the given file",
},
cli.StringFlag{
Name: "add-azure-kv",
Usage: "add the provided comma-separated list of Azure Key Vault key URLs to the list of master keys on the given file",
},
cli.StringFlag{
Name: "rm-azure-kv",
Usage: "remove the provided comma-separated list of Azure Key Vault key URLs from the list of master keys on the given file",
},
cli.StringFlag{
Name: "add-kms",
Usage: "add the provided comma-separated list of KMS ARNs to the list of master keys on the given file",
},
cli.StringFlag{
Name: "rm-kms",
Usage: "remove the provided comma-separated list of KMS ARNs from the list of master keys on the given file",
},
cli.StringFlag{
Name: "add-hc-vault-transit",
Usage: "add the provided comma-separated list of Vault's URI key to the list of master keys on the given file ( eg. https://vault.example.org:8200/v1/transit/keys/dev)",
},
cli.StringFlag{
Name: "rm-hc-vault-transit",
Usage: "remove the provided comma-separated list of Vault's URI key from the list of master keys on the given file ( eg. https://vault.example.org:8200/v1/transit/keys/dev)",
},
cli.StringFlag{
Name: "add-age",
Usage: "add the provided comma-separated list of age recipients fingerprints to the list of master keys on the given file",
},
cli.StringFlag{
Name: "rm-age",
Usage: "remove the provided comma-separated list of age recipients from the list of master keys on the given file",
},
cli.StringFlag{
Name: "add-pgp",
Usage: "add the provided comma-separated list of PGP fingerprints to the list of master keys on the given file",
},
cli.StringFlag{
Name: "rm-pgp",
Usage: "remove the provided comma-separated list of PGP fingerprints from the list of master keys on the given file",
},
cli.StringFlag{
Name: "filename-override",
Usage: "Use this filename instead of the provided argument for loading configuration, and for determining input type and output type",
},
cli.StringFlag{
Name: "decryption-order",
Usage: "comma separated list of decryption key types",
EnvVar: "SOPS_DECRYPTION_ORDER",
},
}, keyserviceFlags...),
Action: func(c *cli.Context) error {
if c.Bool("verbose") {
logging.SetLevel(logrus.DebugLevel)
}
if c.NArg() < 1 {
return common.NewExitError("Error: no file specified", codes.NoFileSpecified)
}
warnMoreThanOnePositionalArgument(c)
if c.Bool("in-place") && c.String("output") != "" {
return common.NewExitError("Error: cannot operate on both --output and --in-place", codes.ErrorConflictingParameters)
}
fileName, err := filepath.Abs(c.Args()[0])
if err != nil {
return toExitError(err)
}
if _, err := os.Stat(fileName); os.IsNotExist(err) {
if c.String("add-kms") != "" || c.String("add-pgp") != "" || c.String("add-gcp-kms") != "" || c.String("add-hc-vault-transit") != "" || c.String("add-azure-kv") != "" || c.String("add-age") != "" ||
c.String("rm-kms") != "" || c.String("rm-pgp") != "" || c.String("rm-gcp-kms") != "" || c.String("rm-hc-vault-transit") != "" || c.String("rm-azure-kv") != "" || c.String("rm-age") != "" {
return common.NewExitError("Error: cannot add or remove keys on non-existent files, use the `edit` subcommand instead.", codes.CannotChangeKeysFromNonExistentFile)
}
}
fileNameOverride := c.String("filename-override")
if fileNameOverride == "" {
fileNameOverride = fileName
}
inputStore := inputStore(c, fileNameOverride)
outputStore := outputStore(c, fileNameOverride)
svcs := keyservices(c)
order, err := decryptionOrder(c.String("decryption-order"))
if err != nil {
return toExitError(err)
}
rotateOpts, err := getRotateOpts(c, fileName, inputStore, outputStore, svcs, order)
if err != nil {
return toExitError(err)
}
output, err := rotate(rotateOpts)
if err != nil {
return toExitError(err)
}
// We open the file *after* the operations on the tree have been
// executed to avoid truncating it when there's errors
if c.Bool("in-place") {
file, err := os.Create(fileName)
if err != nil {
return common.NewExitError(fmt.Sprintf("Could not open in-place file for writing: %s", err), codes.CouldNotWriteOutputFile)
}
defer file.Close()
_, err = file.Write(output)
if err != nil {
return toExitError(err)
}
log.Info("File written successfully")
return nil
}
outputFile := os.Stdout
if c.String("output") != "" {
file, err := os.Create(c.String("output"))
if err != nil {
return common.NewExitError(fmt.Sprintf("Could not open output file for writing: %s", err), codes.CouldNotWriteOutputFile)
}
defer file.Close()
outputFile = file
}
_, err = outputFile.Write(output)
return toExitError(err)
},
},
{
Name: "edit",
Usage: "edit an encrypted file",
ArgsUsage: `file`,
Flags: append([]cli.Flag{
cli.StringFlag{
Name: "kms, k",
Usage: "comma separated list of KMS ARNs",
EnvVar: "SOPS_KMS_ARN",
},
cli.StringFlag{
Name: "aws-profile",
Usage: "The AWS profile to use for requests to AWS",
},
cli.StringFlag{
Name: "gcp-kms",
Usage: "comma separated list of GCP KMS resource IDs",
EnvVar: "SOPS_GCP_KMS_IDS",
},
cli.StringFlag{
Name: "azure-kv",
Usage: "comma separated list of Azure Key Vault URLs",
EnvVar: "SOPS_AZURE_KEYVAULT_URLS",
},
cli.StringFlag{
Name: "hc-vault-transit",
Usage: "comma separated list of vault's key URI (e.g. 'https://vault.example.org:8200/v1/transit/keys/dev')",
EnvVar: "SOPS_VAULT_URIS",
},
cli.StringFlag{
Name: "pgp, p",
Usage: "comma separated list of PGP fingerprints",
EnvVar: "SOPS_PGP_FP",
},
cli.StringFlag{
Name: "age, a",
Usage: "comma separated list of age recipients",
EnvVar: "SOPS_AGE_RECIPIENTS",
},
cli.StringFlag{
Name: "input-type",
Usage: "currently json, yaml, dotenv and binary are supported. If not set, sops will use the file's extension to determine the type",
},
cli.StringFlag{
Name: "output-type",
Usage: "currently json, yaml, dotenv and binary are supported. If not set, sops will use the input file's extension to determine the output format",
},
cli.StringFlag{
Name: "unencrypted-suffix",
Usage: "override the unencrypted key suffix.",
},
cli.StringFlag{
Name: "encrypted-suffix",
Usage: "override the encrypted key suffix. When empty, all keys will be encrypted, unless otherwise marked with unencrypted-suffix.",
},
cli.StringFlag{
Name: "unencrypted-regex",
Usage: "set the unencrypted key regex. When specified, only keys matching the regex will be left unencrypted.",
},
cli.StringFlag{
Name: "encrypted-regex",
Usage: "set the encrypted key regex. When specified, only keys matching the regex will be encrypted.",
},
cli.StringFlag{
Name: "encryption-context",
Usage: "comma separated list of KMS encryption context key:value pairs",
},
cli.IntFlag{
Name: "shamir-secret-sharing-threshold",
Usage: "the number of master keys required to retrieve the data key with shamir",
},
cli.BoolFlag{
Name: "show-master-keys, s",
Usage: "display master encryption keys in the file during editing",
},
cli.BoolFlag{
Name: "ignore-mac",
Usage: "ignore Message Authentication Code during decryption",
},
cli.StringFlag{
Name: "decryption-order",
Usage: "comma separated list of decryption key types",
EnvVar: "SOPS_DECRYPTION_ORDER",
},
}, keyserviceFlags...),
Action: func(c *cli.Context) error {
if c.Bool("verbose") {
logging.SetLevel(logrus.DebugLevel)
}
if c.NArg() < 1 {
return common.NewExitError("Error: no file specified", codes.NoFileSpecified)
}
warnMoreThanOnePositionalArgument(c)
fileName, err := filepath.Abs(c.Args()[0])
if err != nil {
return toExitError(err)
}
if _, err := os.Stat(fileName); os.IsNotExist(err) {
return common.NewExitError("Error: cannot operate on non-existent file", codes.NoFileSpecified)
}
inputStore := inputStore(c, fileName)
outputStore := outputStore(c, fileName)
svcs := keyservices(c)
order, err := decryptionOrder(c.String("decryption-order"))
if err != nil {
return toExitError(err)
}
var output []byte
_, statErr := os.Stat(fileName)
fileExists := statErr == nil
opts := editOpts{
OutputStore: outputStore,
InputStore: inputStore,
InputPath: fileName,
Cipher: aes.NewCipher(),
KeyServices: svcs,
DecryptionOrder: order,
IgnoreMAC: c.Bool("ignore-mac"),
ShowMasterKeys: c.Bool("show-master-keys"),
}
if fileExists {
output, err = edit(opts)
if err != nil {
return toExitError(err)
}
} else {
// File doesn't exist, edit the example file instead
encConfig, err := getEncryptConfig(c, fileName)
if err != nil {
return toExitError(err)
}
output, err = editExample(editExampleOpts{
editOpts: opts,
encryptConfig: encConfig,
})
if err != nil {
return toExitError(err)
}
}
// We open the file *after* the operations on the tree have been
// executed to avoid truncating it when there's errors
file, err := os.Create(fileName)
if err != nil {
return common.NewExitError(fmt.Sprintf("Could not open in-place file for writing: %s", err), codes.CouldNotWriteOutputFile)
}
defer file.Close()
_, err = file.Write(output)
if err != nil {
return toExitError(err)
}
log.Info("File written successfully")
return nil
},
},
{
Name: "set",
Usage: `set a specific key or branch in the input document. value must be a json encoded string. eg. '/path/to/file ["somekey"][0] {"somevalue":true}'`,
ArgsUsage: `file index value`,
Flags: append([]cli.Flag{
cli.StringFlag{
Name: "input-type",
Usage: "currently json, yaml, dotenv and binary are supported. If not set, sops will use the file's extension to determine the type",
},
cli.StringFlag{
Name: "output-type",
Usage: "currently json, yaml, dotenv and binary are supported. If not set, sops will use the input file's extension to determine the output format",
},
cli.IntFlag{
Name: "shamir-secret-sharing-threshold",
Usage: "the number of master keys required to retrieve the data key with shamir",
},
cli.BoolFlag{
Name: "ignore-mac",
Usage: "ignore Message Authentication Code during decryption",
},
cli.StringFlag{
Name: "decryption-order",
Usage: "comma separated list of decryption key types",
EnvVar: "SOPS_DECRYPTION_ORDER",
},
}, keyserviceFlags...),
Action: func(c *cli.Context) error {
if c.Bool("verbose") {
logging.SetLevel(logrus.DebugLevel)
}
if c.NArg() != 3 {
return common.NewExitError("Error: no file specified, or index and value are missing", codes.NoFileSpecified)
}
fileName, err := filepath.Abs(c.Args()[0])
if err != nil {
return toExitError(err)
}
inputStore := inputStore(c, fileName)
outputStore := outputStore(c, fileName)
svcs := keyservices(c)
path, err := parseTreePath(c.Args()[1])
if err != nil {
return common.NewExitError("Invalid set index format", codes.ErrorInvalidSetFormat)
}
value, err := jsonValueToTreeInsertableValue(c.Args()[2])
if err != nil {
return toExitError(err)
}
order, err := decryptionOrder(c.String("decryption-order"))
if err != nil {
return toExitError(err)
}
output, err := set(setOpts{
OutputStore: outputStore,
InputStore: inputStore,
InputPath: fileName,
Cipher: aes.NewCipher(),
KeyServices: svcs,
DecryptionOrder: order,
IgnoreMAC: c.Bool("ignore-mac"),
Value: value,
TreePath: path,
})
if err != nil {
return toExitError(err)
}
// We open the file *after* the operations on the tree have been
// executed to avoid truncating it when there's errors
file, err := os.Create(fileName)
if err != nil {
return common.NewExitError(fmt.Sprintf("Could not open in-place file for writing: %s", err), codes.CouldNotWriteOutputFile)
}
defer file.Close()
_, err = file.Write(output)
if err != nil {
return toExitError(err)
}
log.Info("File written successfully")
return nil
},
},
}
app.Flags = append([]cli.Flag{
cli.BoolFlag{

26
functional-tests/src/lib.rs
View File

@@ -267,9 +267,10 @@ bar: baz",
"sops didn't exit successfully"
);
let output = Command::new(SOPS_BINARY_PATH)
.arg("--set")
.arg(r#"["a"] {"aa": "aaa"}"#)
.arg("set")
.arg(file_path.clone())
.arg(r#"["a"]"#)
.arg(r#"{"aa": "aaa"}"#)
.output()
.expect("Error running sops");
assert!(output.status.success(), "sops didn't exit successfully");
@@ -310,9 +311,10 @@ bar: baz",
"sops didn't exit successfully"
);
let output = Command::new(SOPS_BINARY_PATH)
.arg("--set")
.arg(r#"["c"] {"cc": "ccc"}"#)
.arg("set")
.arg(file_path.clone())
.arg(r#"["c"]"#)
.arg(r#"{"cc": "ccc"}"#)
.output()
.expect("Error running sops");
assert!(output.status.success(), "sops didn't exit successfully");
@@ -357,9 +359,10 @@ b: ba"#
"sops didn't exit successfully"
);
let output = Command::new(SOPS_BINARY_PATH)
.arg("--set")
.arg(r#"["a"] {"aa": "aaa"}"#)
.arg("set")
.arg(file_path.clone())
.arg(r#"["a"]"#)
.arg(r#"{"aa": "aaa"}"#)
.output()
.expect("Error running sops");
assert!(output.status.success(), "sops didn't exit successfully");
@@ -404,9 +407,10 @@ b: ba"#
"sops didn't exit successfully"
);
let output = Command::new(SOPS_BINARY_PATH)
.arg("--set")
.arg(r#"["c"] {"cc": "ccc"}"#)
.arg("set")
.arg(file_path.clone())
.arg(r#"["c"]"#)
.arg(r#"{"cc": "ccc"}"#)
.output()
.expect("Error running sops");
assert!(output.status.success(), "sops didn't exit successfully");
@@ -452,10 +456,10 @@ b: ba"#
);
assert!(
Command::new(SOPS_BINARY_PATH)
.arg("--set")
.arg(r#"["a"] "aaa""#)
.arg("-i")
.arg("set")
.arg(file_path.clone())
.arg(r#"["a"]"#)
.arg(r#""aaa""#)
.output()
.expect("Error running sops")
.status