Discussion:
[gentoo-portage-dev] [PATCH 0/5] webrsync: support sync-openpgp-key-path (bug 661838)
Zac Medico
2018-07-27 07:56:40 UTC
Permalink
Add repos.conf settings that enable sync-openpgp-key-path support for
both emerge-webrsync and emerge-delta-webrsync:

sync-webrsync-delta = true|false
Use app-portage/emerge-delta-webrsync to minimize
bandwidth. Defaults to false.

sync-webrsync-keep-snapshots = true|false
Keep snapshots in DISTDIR (do not delete). Defaults to false.

sync-webrsync-verify-signature = true|false
Require the detached tarball signature to contain a good OpenPGP
signature. This uses the OpenPGP key(ring) specified by the
sync-openpgp-key-path setting. Defaults to false.

Zac Medico (5):
emerge-webrsync: exit early for signature problem (bug 661838)
webrsync: support sync-openpgp-key-path (bug 661838)
webrsync: support sync-webrsync-keep-snapshots
emerge-delta-webrsync: exit early for signature problem (bug 661838)
webrsync: support emerge-delta-webrsync (bug 661838)

bin/emerge-webrsync | 22 ++++++-
lib/portage/sync/modules/webrsync/__init__.py | 6 +-
lib/portage/sync/modules/webrsync/webrsync.py | 84 ++++++++++++++++++++++++---
man/portage.5 | 12 ++++
misc/emerge-delta-webrsync | 23 +++++++-
5 files changed, 135 insertions(+), 12 deletions(-)
--
2.16.4
Zac Medico
2018-07-27 07:56:41 UTC
Permalink
Exit early after signature verification failure, since it's
typically inappropriate to try other mirrors in this case
(it may indicate a keyring problem).

Bug: https://bugs.gentoo.org/661838
---
bin/emerge-webrsync | 7 +++++++
1 file changed, 7 insertions(+)

diff --git a/bin/emerge-webrsync b/bin/emerge-webrsync
index 560dd0236..b135567b7 100755
--- a/bin/emerge-webrsync
+++ b/bin/emerge-webrsync
@@ -191,6 +191,13 @@ check_file_signature() {
fi
done <<< "${gnupg_status}"
fi
+ if [[ ${r} -ne 0 ]]; then
+ # Exit early since it's typically inappropriate to
+ # try other mirrors in this case (it may indicate
+ # a keyring problem).
+ eecho "signature verification failed"
+ exit 1
+ fi
else
eecho "cannot check signature: gpg binary not found"
exit 1
--
2.16.4
Zac Medico
2018-07-27 07:56:43 UTC
Permalink
Add a repos.conf sync-webrsync-keep-snapshots setting that enables
the emerge-webrsync --keep option, which keeps snapshots in DISTDIR
instead of deleting them.
---
lib/portage/sync/modules/webrsync/__init__.py | 1 +
man/portage.5 | 3 +++
2 files changed, 4 insertions(+)

diff --git a/lib/portage/sync/modules/webrsync/__init__.py b/lib/portage/sync/modules/webrsync/__init__.py
index 1e09d1a47..118e752de 100644
--- a/lib/portage/sync/modules/webrsync/__init__.py
+++ b/lib/portage/sync/modules/webrsync/__init__.py
@@ -46,6 +46,7 @@ module_spec = {
},
'validate_config': CheckSyncConfig,
'module_specific_options': (
+ 'sync-webrsync-keep-snapshots',
'sync-webrsync-verify-signature',
),
},
diff --git a/man/portage.5 b/man/portage.5
index 8ebf980f5..4cb1b0b34 100644
--- a/man/portage.5
+++ b/man/portage.5
@@ -1128,6 +1128,9 @@ when 0. Defaults to disabled.
Require the repository to contain a signed MetaManifest and verify
it using \fBapp\-portage/gemato\fR. Defaults to no.
.TP
+.B sync\-webrsync\-keep\-snapshots = true|false
+Keep snapshots in \fBDISTDIR\fR (do not delete). Defaults to false.
+.TP
.B sync\-webrsync\-verify\-signature = true|false
Require the detached tarball signature to contain a good OpenPGP
signature. This uses the OpenPGP key(ring) specified by the
--
2.16.4
Zac Medico
2018-07-27 07:56:42 UTC
Permalink
Add repos.conf sync-webrsync-verify-signature = true|false setting that
enables sync-openpgp-key-path support like in the rsync and git sync
modules. This is disabled by default, in order to avoid interference
with legacy manual PORTAGE_GPG_DIR configuration.

When sync-webrsync-verify-signature = true is set in repos.conf,
if the PORTAGE_GPG_DIR has not been exported, emerge-webrsync will
assume that it has been called directly and it will output an error
message advising the user to instead call emerge --sync or emaint sync.

Bug: https://bugs.gentoo.org/661838
---
bin/emerge-webrsync | 15 +++++-
lib/portage/sync/modules/webrsync/__init__.py | 4 +-
lib/portage/sync/modules/webrsync/webrsync.py | 74 +++++++++++++++++++++++----
man/portage.5 | 5 ++
4 files changed, 87 insertions(+), 11 deletions(-)

diff --git a/bin/emerge-webrsync b/bin/emerge-webrsync
index b135567b7..841f01614 100755
--- a/bin/emerge-webrsync
+++ b/bin/emerge-webrsync
@@ -10,7 +10,14 @@
# - all output should prob be converted to e* funcs
# - add support for ROOT

+# repos.conf configuration for use with emerge --sync and emaint sync
+# using keyring from app-crypt/openpgp-keys-gentoo-release:
+# [gentoo]
+# sync-type = webrsync
+# sync-webrsync-verify-signature = true
+# sync-openpgp-key-path = /usr/share/openpgp-keys/gentoo-release.asc
#
+# Alternative (legacy) PORTAGE_GPG_DIR configuration:
# gpg key import
# KEY_ID=0x96D8BF6D
# gpg --homedir /etc/portage/gnupg --keyserver subkeys.pgp.net --recv-keys $KEY_ID
@@ -67,7 +74,13 @@ do_verbose=0
do_debug=0
keep=false

-if has webrsync-gpg ${FEATURES} ; then
+if has $(__repo_attr "${repo_name}" sync-webrsync-verify-signature) true yes; then
+ if [[ ! -d ${PORTAGE_GPG_DIR} ]]; then
+ eecho "Do not call ${argv0##*/} directly, instead call emerge --sync or emaint sync."
+ exit 1
+ fi
+ WEBSYNC_VERIFY_SIGNATURE=1
+elif has webrsync-gpg ${FEATURES}; then
WEBSYNC_VERIFY_SIGNATURE=1
else
WEBSYNC_VERIFY_SIGNATURE=0
diff --git a/lib/portage/sync/modules/webrsync/__init__.py b/lib/portage/sync/modules/webrsync/__init__.py
index dc7def20c..1e09d1a47 100644
--- a/lib/portage/sync/modules/webrsync/__init__.py
+++ b/lib/portage/sync/modules/webrsync/__init__.py
@@ -45,7 +45,9 @@ module_spec = {
'exists and is a valid repository',
},
'validate_config': CheckSyncConfig,
- 'module_specific_options': (),
+ 'module_specific_options': (
+ 'sync-webrsync-verify-signature',
+ ),
},
}
}
diff --git a/lib/portage/sync/modules/webrsync/webrsync.py b/lib/portage/sync/modules/webrsync/webrsync.py
index 3d79f4557..1b4c08e65 100644
--- a/lib/portage/sync/modules/webrsync/webrsync.py
+++ b/lib/portage/sync/modules/webrsync/webrsync.py
@@ -1,17 +1,25 @@

'''WebRsync module for portage'''

+import io
import logging

import portage
from portage import os
from portage.util import writemsg_level
+from portage.util.futures import asyncio
from portage.output import create_color_func
good = create_color_func("GOOD")
bad = create_color_func("BAD")
warn = create_color_func("WARN")
from portage.sync.syncbase import SyncBase

+try:
+ from gemato.exceptions import GematoException
+ import gemato.openpgp
+except ImportError:
+ gemato = None
+

class WebRsync(SyncBase):
'''WebRSync sync class'''
@@ -39,15 +47,63 @@ class WebRsync(SyncBase):
for var in ['uid', 'gid', 'groups']:
self.spawn_kwargs.pop(var, None)

- exitcode = portage.process.spawn_bash("%s" % \
- (self.bin_command),
- **self.spawn_kwargs)
- if exitcode != os.EX_OK:
- msg = "!!! emerge-webrsync error in %s" % self.repo.location
- self.logger(self.xterm_titles, msg)
- writemsg_level(msg + "\n", level=logging.ERROR, noiselevel=-1)
- return (exitcode, False)
- return (exitcode, True)
+ verbose = '--verbose' in self.options['emerge_config'].opts
+ quiet = '--quiet' in self.options['emerge_config'].opts
+ openpgp_env = None
+ try:
+ if self.repo.module_specific_options.get(
+ 'sync-webrsync-verify-signature', 'false').lower() in ('true', 'yes'):
+
+ if not self.repo.sync_openpgp_key_path:
+ writemsg_level("!!! sync-openpgp-key-path is not set\n",
+ level=logging.ERROR, noiselevel=-1)
+ return (1, False)
+
+ if not os.path.isfile(self.repo.sync_openpgp_key_path):
+ writemsg_level("!!! sync-openpgp-key-path file not found: %s\n" %
+ self.repo.sync_openpgp_key_path, level=logging.ERROR, noiselevel=-1)
+ return (1, False)
+
+ if gemato is None:
+ writemsg_level("!!! Verifying against specified key requires gemato-11.0+ installed\n",
+ level=logging.ERROR, noiselevel=-1)
+ return (1, False)
+
+ openpgp_env = gemato.openpgp.OpenPGPEnvironment()
+
+ out = portage.output.EOutput(quiet=quiet)
+ try:
+ out.einfo('Using keys from %s' % (self.repo.sync_openpgp_key_path,))
+ with io.open(self.repo.sync_openpgp_key_path, 'rb') as f:
+ openpgp_env.import_key(f)
+ self._refresh_keys(openpgp_env)
+ self.spawn_kwargs["env"]["PORTAGE_GPG_DIR"] = openpgp_env.home
+ except (GematoException, asyncio.TimeoutError) as e:
+ writemsg_level("!!! Verification impossible due to keyring problem:\n%s\n"
+ % (e,),
+ level=logging.ERROR, noiselevel=-1)
+ return (1, False)
+
+ webrsync_cmd = [self.bin_command]
+ if verbose:
+ webrsync_cmd.append('-v')
+ elif quiet:
+ webrsync_cmd.append('-q')
+
+ if self.repo.module_specific_options.get(
+ 'sync-webrsync-keep-snapshots', 'false').lower() in ('true', 'yes'):
+ webrsync_cmd.append('-k')
+
+ exitcode = portage.process.spawn(webrsync_cmd, **self.spawn_kwargs)
+ if exitcode != os.EX_OK:
+ msg = "!!! emerge-webrsync error in %s" % self.repo.location
+ self.logger(self.xterm_titles, msg)
+ writemsg_level(msg + "\n", level=logging.ERROR, noiselevel=-1)
+ return (exitcode, False)
+ return (exitcode, True)
+ finally:
+ if openpgp_env is not None:
+ openpgp_env.close()


class PyWebRsync(SyncBase):
diff --git a/man/portage.5 b/man/portage.5
index a57531d44..8ebf980f5 100644
--- a/man/portage.5
+++ b/man/portage.5
@@ -1127,6 +1127,11 @@ when 0. Defaults to disabled.
.B sync\-rsync\-verify\-metamanifest = yes|no
Require the repository to contain a signed MetaManifest and verify
it using \fBapp\-portage/gemato\fR. Defaults to no.
+.TP
+.B sync\-webrsync\-verify\-signature = true|false
+Require the detached tarball signature to contain a good OpenPGP
+signature. This uses the OpenPGP key(ring) specified by the
+sync\-openpgp\-key\-path setting. Defaults to false.

.RE
--
2.16.4
Zac Medico
2018-07-27 07:56:44 UTC
Permalink
Exit early after signature verification failure, since it's
typically inappropriate to try other mirrors in this case
(it may indicate a keyring problem).

Bug: https://bugs.gentoo.org/661838
---
misc/emerge-delta-webrsync | 7 +++++++
1 file changed, 7 insertions(+)

diff --git a/misc/emerge-delta-webrsync b/misc/emerge-delta-webrsync
index 868c6a347..ebaa616f9 100755
--- a/misc/emerge-delta-webrsync
+++ b/misc/emerge-delta-webrsync
@@ -283,6 +283,13 @@ check_file_signature() {
fi
done <<< "${gnupg_status}"
fi
+ if [[ ${r} -ne 0 ]]; then
+ # Exit early since it's typically inappropriate to
+ # try other mirrors in this case (it may indicate
+ # a keyring problem).
+ eecho "signature verification failed"
+ exit 1
+ fi
else
eecho "cannot check signature: gpg binary not found"
exit 1
--
2.16.4
Zac Medico
2018-07-27 07:56:45 UTC
Permalink
Add a repos.conf sync-webrsync-delta setting that makes the webrsync
module call emerge-delta-webrsync, so that emerge-delta-webrsync users
can benefit from sync-openpgp-key-path support in the webrsync module.

Bug: https://bugs.gentoo.org/661838
---
lib/portage/sync/modules/webrsync/__init__.py | 1 +
lib/portage/sync/modules/webrsync/webrsync.py | 10 ++++++++++
man/portage.5 | 4 ++++
misc/emerge-delta-webrsync | 16 +++++++++++++++-
4 files changed, 30 insertions(+), 1 deletion(-)

diff --git a/lib/portage/sync/modules/webrsync/__init__.py b/lib/portage/sync/modules/webrsync/__init__.py
index 118e752de..a413553a1 100644
--- a/lib/portage/sync/modules/webrsync/__init__.py
+++ b/lib/portage/sync/modules/webrsync/__init__.py
@@ -46,6 +46,7 @@ module_spec = {
},
'validate_config': CheckSyncConfig,
'module_specific_options': (
+ 'sync-webrsync-delta',
'sync-webrsync-keep-snapshots',
'sync-webrsync-verify-signature',
),
diff --git a/lib/portage/sync/modules/webrsync/webrsync.py b/lib/portage/sync/modules/webrsync/webrsync.py
index 1b4c08e65..609ba0be2 100644
--- a/lib/portage/sync/modules/webrsync/webrsync.py
+++ b/lib/portage/sync/modules/webrsync/webrsync.py
@@ -34,6 +34,16 @@ class WebRsync(SyncBase):
def __init__(self):
SyncBase.__init__(self, 'emerge-webrsync', '>=sys-apps/portage-2.3')

+ @property
+ def has_bin(self):
+ if (self._bin_command != 'emerge-delta-webrsync' and
+ self.repo.module_specific_options.get(
+ 'sync-webrsync-delta', 'false').lower() in ('true', 'yes')):
+ self._bin_command = 'emerge-delta-webrsync'
+ self.bin_command = portage.process.find_binary(self._bin_command)
+ self.bin_pkg = '>=app-portage/emerge-delta-webrsync-3.7.5'
+
+ return super(WebRsync, self).has_bin

def sync(self, **kwargs):
'''Sync the repository'''
diff --git a/man/portage.5 b/man/portage.5
index 4cb1b0b34..cd9d5036d 100644
--- a/man/portage.5
+++ b/man/portage.5
@@ -1128,6 +1128,10 @@ when 0. Defaults to disabled.
Require the repository to contain a signed MetaManifest and verify
it using \fBapp\-portage/gemato\fR. Defaults to no.
.TP
+.B sync\-webrsync\-delta = true|false
+Use \fBapp\-portage/emerge\-delta\-webrsync\fR to minimize bandwidth.
+Defaults to false.
+.TP
.B sync\-webrsync\-keep\-snapshots = true|false
Keep snapshots in \fBDISTDIR\fR (do not delete). Defaults to false.
.TP
diff --git a/misc/emerge-delta-webrsync b/misc/emerge-delta-webrsync
index ebaa616f9..5ade2708b 100755
--- a/misc/emerge-delta-webrsync
+++ b/misc/emerge-delta-webrsync
@@ -4,7 +4,15 @@
# Author: Brian Harring <***@gentoo.org>, ***@gentoo.org originally.
# Rewritten from the old, Perl-based emerge-webrsync script

+# repos.conf configuration for use with emerge --sync and emaint sync
+# using keyring from app-crypt/openpgp-keys-gentoo-release:
+# [gentoo]
+# sync-type = webrsync
+# sync-webrsync-delta = true
+# sync-webrsync-verify-signature = true
+# sync-openpgp-key-path = /usr/share/openpgp-keys/gentoo-release.asc
#
+# Alternative (legacy) PORTAGE_GPG_DIR configuration:
# gpg key import
# KEY_ID=0x96D8BF6D
# gpg --homedir /etc/portage/gnupg --keyserver subkeys.pgp.net --recv-keys $KEY_ID
@@ -106,7 +114,13 @@ if [[ ! -d $STATE_DIR ]]; then
exit -2
fi

-if has webrsync-gpg ${FEATURES} ; then
+if has $(__repo_attr "${repo_name}" sync-webrsync-verify-signature) true yes; then
+ if [[ ! -d ${PORTAGE_GPG_DIR} ]]; then
+ eecho "Do not call ${argv0##*/} directly, instead call emerge --sync or emaint sync."
+ exit 1
+ fi
+ WEBSYNC_VERIFY_SIGNATURE=1
+elif has webrsync-gpg ${FEATURES}; then
WEBSYNC_VERIFY_SIGNATURE=1
else
WEBSYNC_VERIFY_SIGNATURE=0
--
2.16.4
Brian Dolbec
2018-07-27 15:25:35 UTC
Permalink
On Fri, 27 Jul 2018 00:56:40 -0700
Post by Zac Medico
Add repos.conf settings that enable sync-openpgp-key-path support for
sync-webrsync-delta = true|false
Use app-portage/emerge-delta-webrsync to minimize
bandwidth. Defaults to false.
sync-webrsync-keep-snapshots = true|false
Keep snapshots in DISTDIR (do not delete). Defaults to false.
sync-webrsync-verify-signature = true|false
Require the detached tarball signature to contain a good
OpenPGP signature. This uses the OpenPGP key(ring) specified by the
sync-openpgp-key-path setting. Defaults to false.
emerge-webrsync: exit early for signature problem (bug 661838)
webrsync: support sync-openpgp-key-path (bug 661838)
webrsync: support sync-webrsync-keep-snapshots
emerge-delta-webrsync: exit early for signature problem (bug 661838)
webrsync: support emerge-delta-webrsync (bug 661838)
bin/emerge-webrsync | 22 ++++++-
lib/portage/sync/modules/webrsync/__init__.py | 6 +-
lib/portage/sync/modules/webrsync/webrsync.py | 84
++++++++++++++++++++++++---
man/portage.5 | 12 ++++
misc/emerge-delta-webrsync | 23 +++++++- 5 files
changed, 135 insertions(+), 12 deletions(-)
this series looks good. I'm also surprised to see nearly all the
changes were on the python side. Barely any changes to the bash
scripts.
Zac Medico
2018-07-31 16:46:48 UTC
Permalink
Post by Brian Dolbec
On Fri, 27 Jul 2018 00:56:40 -0700
Post by Zac Medico
Add repos.conf settings that enable sync-openpgp-key-path support for
sync-webrsync-delta = true|false
Use app-portage/emerge-delta-webrsync to minimize
bandwidth. Defaults to false.
sync-webrsync-keep-snapshots = true|false
Keep snapshots in DISTDIR (do not delete). Defaults to false.
sync-webrsync-verify-signature = true|false
Require the detached tarball signature to contain a good
OpenPGP signature. This uses the OpenPGP key(ring) specified by the
sync-openpgp-key-path setting. Defaults to false.
emerge-webrsync: exit early for signature problem (bug 661838)
webrsync: support sync-openpgp-key-path (bug 661838)
webrsync: support sync-webrsync-keep-snapshots
emerge-delta-webrsync: exit early for signature problem (bug 661838)
webrsync: support emerge-delta-webrsync (bug 661838)
bin/emerge-webrsync | 22 ++++++-
lib/portage/sync/modules/webrsync/__init__.py | 6 +-
lib/portage/sync/modules/webrsync/webrsync.py | 84
++++++++++++++++++++++++---
man/portage.5 | 12 ++++
misc/emerge-delta-webrsync | 23 +++++++- 5 files
changed, 135 insertions(+), 12 deletions(-)
this series looks good. I'm also surprised to see nearly all the
changes were on the python side. Barely any changes to the bash
scripts.
Thanks, merged:

https://gitweb.gentoo.org/proj/portage.git/commit/?id=f810f8694f78dd87172e38d942580532017db4fe
--
Thanks,
Zac
Loading...