This avoids quoting problems, fixes #784, and also better handles
various problems that can arise, like '=' in keys and non-string
keys and values.
Signed-off-by: Felix Fontein <felix@fontein.de>
Fixes #1353
Consolidate the common json marshalling and \n handing to stores/flatten.go
Make existing functions unexported.
Add casts for MACOnlyEncrypted
Signed-off-by: Lance Rushing <lance@lancerushing.com>
this will allow for setting of parameters specific to each store, such
as indentation level for YAML
Co-authored-by: Bastien Wermeille <bastien.wermeille@gmail.com>
Signed-off-by: James J. Goodhouse <jgoodhouse@newrelic.com>
Makes sure that the 'data' key refers to strings. Also
improves error messages, and on CLI hints at the --output-type
option.
Signed-off-by: Felix Fontein <felix@fontein.de>
Deprecation of `io/ioutil`, removal of unused functions, possible nil
pointer dereference, and other tiny nits.
There are (many) more, but these would require their own (commit)
context.
Signed-off-by: Hidde Beydals <hidde@hhh.computer>
This commit renames the Go module from `go.mozilla.org/sops/v3` to
`github.com/getsops/sops/v3` without a major version bump, to align
with new stewardship.
For more information around this change, refer to
https://github.com/getsops/sops/issues/1246.
For a one-liner to change the `go.mod` and any import paths in your
Go project making use of this module, run:
```
find /path/to/repo -type f \( -name "*.go" -o -name "go.mod" \) -exec sed -i 's|go.mozilla.org/sops/v3|github.com/getsops/sops/v3|g' {} \;
find /path/to/repo -type f \( -name "*.go" -o -name "go.mod" \) -exec sed -i '' 's|go.mozilla.org/sops/v3|github.com/getsops/sops/v3|g' {} \;
```
Signed-off-by: Hidde Beydals <hidde@hhh.computer>
* Add another test (that currently fails).
* First shot at using yaml.v3 for reading YAML files with comments.
* Allow parsing multi-document YAML files.
* Use Decoder to parse multi-part documents.
* Use yaml.v3 for config and audit.
* First step of serializing YAML using yaml.v3.
* Always serialize with yaml.v3.
* Remove debug prints.
* Remove traces of github.com/mozilla-services/yaml.
* Improve serialization of documents consisting only of comments.
* Improve handling of some empty documents.
* Adjust to latest changes in go-yaml/yaml#684.
* Bump yaml.v3 version, temporarily disable failing tests.
* Run go mod tidy.
* Fix CI.
* feat: initial adding of vualt transit backend to sops
initial work on integration
feat(vault): added cli coomands working for vualt"
fix(vault): fixed config with correct tests
fix(vault): added vault to keygroup and to keyservice server
fixed metadata load
* feat(docs): added docs in README.md and in command help
fix(doc): fix rst formatting"
fix(doc): fix rst formatting
* fix(vault): addressed typos and fixes from autrilla
feat(cli): moved vault to hc-vault naming
* fix(test): typo while rebasing
* fix typos and imporve error messages for vault kms
* rename package from vault to hcvault
* refactor vault keysource url validation
* add negative test cases for vault keysource
* add hc vault transit config option via objects
additional to URIs
* remove vault_example.yml
* streamline key name to snake case
* rename `BackendPath` to `EnginePath` for hc vault
* correction in hc-vault-transit commands
Signed-off-by: vnzongzna <github@vaibhavk.in>
* resolving conflict
Signed-off-by: vnzongzna <github@vaibhavk.in>
* Apply suggestions from code review
Co-Authored-By: Adrian Utrilla <adrianutrilla@gmail.com>
* allowing only hc_vault_transit_uri as input
Co-Authored-By: gitirabassi
Co-Authored-By: ldue
Signed-off-by: vnzongzna <github@vaibhavk.in>
Co-authored-by: gitirabassi <giacomo@tirabassi.eu>
Co-authored-by: ldue <larsduennwald@gmail.com>
Co-authored-by: Vaibhav Kaushik <vaibhavkaushik@vaibhavka-ltm1.internal.salesforce.com>
Co-authored-by: Adrian Utrilla <adrianutrilla@gmail.com>
Add support for decoding JSON arrays of arrays by handling, during
slice decoding, when the next token is an array opening. This produces
nested []interface{} slices.
Closes #640.
Rationale
=========
The dotenv store as it exists right now performs splitting on newlines
to determine where a new key-value pair or comment begins. This works
remarkably well, up until you need to handle values that contain
newlines.
While I couldn't find an offical dotenv file format spec, I sampled a
number of open-source dotenv parsers and it seems that they typically
apply the following rules:
Comments:
* Comments may be written by starting a line with the `#` character.
Newline handling:
* If a value is unquoted or single-quoted and contains the character
sequence `\n` (`0x5c6e`), it IS NOT decoded to a line feed (`0x0a`).
* If a value is double-quoted and contains the character sequence `\n`
(`0x5c6e`), it IS decoded to a line feed (`0x0a`).
Whitespace trimming:
* For comments, the whitespace immediately after the `#` character and any
trailing whitespace is trimmed.
* If a value is unquoted and contains any leading or trailing whitespace, it
is trimmed.
* If a value is either single- or double-quoted and contains any leading or
trailing whitespace, it is left untrimmed.
Quotation handling:
* If a value is surrounded by single- or double-quotes, the quotation marks
are interpreted and not included in the value.
* Any number of single-quote characters may appear in a double-quoted
value, or within a single-quoted value if they are escaped (i.e.,
`'foo\'bar'`).
* Any number of double-quote characters may appear in a single-quoted
value, or within a double-quoted value if they are escaped (i.e.,
`"foo\"bar"`).
Because single- and double-quoted values may contain actual newlines,
we cannot split our input data on newlines as this may be in the middle
of a quoted value. This, along with the other rules around handling
quoted values, prompted me to try and implement a more robust parsing
solution. This commit is my first stab at that.
Special Considerations
======================
This is _not_ a backwards-compatible change:
* The `dotenv` files produced by this version of SOPS _cannot_ be read
by an earlier version.
* The `dotenv` files produced by an earlier version of SOPS _can_ be
read by this version, with the understanding that the semantics around
quotations and newlines have changed.
Examples
========
The below examples show how double-quoted values are passed to the
running environment:
```console
$ echo 'FOO="foo\\nbar\\nbaz"' > plaintext.env
$ sops -e --output ciphertext.env plaintext.env
$ sops exec-env ciphertext.env 'env | grep FOO | xxd'
00000000: 464f 4f3d 666f 6f5c 6e62 6172 5c6e 6261 FOO=foo\nbar\nba
00000010: 7a0a z.
```
```console
$ echo 'FOO="foo\nbar\nbaz"' > plaintext.env
$ sops -e --output ciphertext.env plaintext.env
$ sops exec-env ciphertext.env 'env | grep -A2 FOO | xxd'
00000000: 464f 4f3d 666f 6f0a 6261 720a 6261 7a0a FOO=foo.bar.baz.
```
When reading and writing dotenv files, we need to make sure to
encode/decode newline characters. SOPS does not currently do this, as
can be seen from the below:
```console
$ echo '{"foo": "foo\nbar\nbaz"}' > plaintext.json
$ sops -e --output ciphertext.json plaintext.json
$ sops -d --output-type dotenv ciphertext.json
foo=foo
bar
baz
```
This output, is invalid and cannot even be fed back into SOPS:
```console
$ sops -d --output-type dotenv --output plaintext.env ciphertext.json
$ sops -e plaintext.env
Error unmarshalling file: invalid dotenv input line: bar
```
This commit fixes the issue, such that the final `sops -d ...` command
above produces the correct output:
```console
$ sops -d --output-type dotenv ciphertext.json
foo=foo\nbar\nbaz
```
