* feat: initial adding of vualt transit backend to sops
initial work on integration
feat(vault): added cli coomands working for vualt"
fix(vault): fixed config with correct tests
fix(vault): added vault to keygroup and to keyservice server
fixed metadata load
* feat(docs): added docs in README.md and in command help
fix(doc): fix rst formatting"
fix(doc): fix rst formatting
* fix(vault): addressed typos and fixes from autrilla
feat(cli): moved vault to hc-vault naming
* fix(test): typo while rebasing
* fix typos and imporve error messages for vault kms
* rename package from vault to hcvault
* refactor vault keysource url validation
* add negative test cases for vault keysource
* add hc vault transit config option via objects
additional to URIs
* remove vault_example.yml
* streamline key name to snake case
* rename `BackendPath` to `EnginePath` for hc vault
* correction in hc-vault-transit commands
Signed-off-by: vnzongzna <github@vaibhavk.in>
* resolving conflict
Signed-off-by: vnzongzna <github@vaibhavk.in>
* Apply suggestions from code review
Co-Authored-By: Adrian Utrilla <adrianutrilla@gmail.com>
* allowing only hc_vault_transit_uri as input
Co-Authored-By: gitirabassi
Co-Authored-By: ldue
Signed-off-by: vnzongzna <github@vaibhavk.in>
Co-authored-by: gitirabassi <giacomo@tirabassi.eu>
Co-authored-by: ldue <larsduennwald@gmail.com>
Co-authored-by: Vaibhav Kaushik <vaibhavkaushik@vaibhavka-ltm1.internal.salesforce.com>
Co-authored-by: Adrian Utrilla <adrianutrilla@gmail.com>
Add support for decoding JSON arrays of arrays by handling, during
slice decoding, when the next token is an array opening. This produces
nested []interface{} slices.
Closes #640.
Rationale
=========
The dotenv store as it exists right now performs splitting on newlines
to determine where a new key-value pair or comment begins. This works
remarkably well, up until you need to handle values that contain
newlines.
While I couldn't find an offical dotenv file format spec, I sampled a
number of open-source dotenv parsers and it seems that they typically
apply the following rules:
Comments:
* Comments may be written by starting a line with the `#` character.
Newline handling:
* If a value is unquoted or single-quoted and contains the character
sequence `\n` (`0x5c6e`), it IS NOT decoded to a line feed (`0x0a`).
* If a value is double-quoted and contains the character sequence `\n`
(`0x5c6e`), it IS decoded to a line feed (`0x0a`).
Whitespace trimming:
* For comments, the whitespace immediately after the `#` character and any
trailing whitespace is trimmed.
* If a value is unquoted and contains any leading or trailing whitespace, it
is trimmed.
* If a value is either single- or double-quoted and contains any leading or
trailing whitespace, it is left untrimmed.
Quotation handling:
* If a value is surrounded by single- or double-quotes, the quotation marks
are interpreted and not included in the value.
* Any number of single-quote characters may appear in a double-quoted
value, or within a single-quoted value if they are escaped (i.e.,
`'foo\'bar'`).
* Any number of double-quote characters may appear in a single-quoted
value, or within a double-quoted value if they are escaped (i.e.,
`"foo\"bar"`).
Because single- and double-quoted values may contain actual newlines,
we cannot split our input data on newlines as this may be in the middle
of a quoted value. This, along with the other rules around handling
quoted values, prompted me to try and implement a more robust parsing
solution. This commit is my first stab at that.
Special Considerations
======================
This is _not_ a backwards-compatible change:
* The `dotenv` files produced by this version of SOPS _cannot_ be read
by an earlier version.
* The `dotenv` files produced by an earlier version of SOPS _can_ be
read by this version, with the understanding that the semantics around
quotations and newlines have changed.
Examples
========
The below examples show how double-quoted values are passed to the
running environment:
```console
$ echo 'FOO="foo\\nbar\\nbaz"' > plaintext.env
$ sops -e --output ciphertext.env plaintext.env
$ sops exec-env ciphertext.env 'env | grep FOO | xxd'
00000000: 464f 4f3d 666f 6f5c 6e62 6172 5c6e 6261 FOO=foo\nbar\nba
00000010: 7a0a z.
```
```console
$ echo 'FOO="foo\nbar\nbaz"' > plaintext.env
$ sops -e --output ciphertext.env plaintext.env
$ sops exec-env ciphertext.env 'env | grep -A2 FOO | xxd'
00000000: 464f 4f3d 666f 6f0a 6261 720a 6261 7a0a FOO=foo.bar.baz.
```
When reading and writing dotenv files, we need to make sure to
encode/decode newline characters. SOPS does not currently do this, as
can be seen from the below:
```console
$ echo '{"foo": "foo\nbar\nbaz"}' > plaintext.json
$ sops -e --output ciphertext.json plaintext.json
$ sops -d --output-type dotenv ciphertext.json
foo=foo
bar
baz
```
This output, is invalid and cannot even be fed back into SOPS:
```console
$ sops -d --output-type dotenv --output plaintext.env ciphertext.json
$ sops -e plaintext.env
Error unmarshalling file: invalid dotenv input line: bar
```
This commit fixes the issue, such that the final `sops -d ...` command
above produces the correct output:
```console
$ sops -d --output-type dotenv ciphertext.json
foo=foo\nbar\nbaz
```
This eliminates use of 3rd party tool but greatly simplifies supported versions.
MAJOR.MINOR.PTACH
Minor & Patch may be omitted ("v3", "v3.2", "v3.2.1")
At the moment, the examples produce warnings, due to the very old format
they use.
This commit re-encrypts the example files to eliminate the warnings that
are occurring from the use of the very old sops format.
Commit e9b9f7aeef generated new PGP keys
for this repository, but failed to update the keys used by the examples.
As a result, the documentation for testing with the dev pgp key does not
work.
This commit rekeys the examples using the newly generated examples,
which allows the testing to work again.
