Skip to content

auth: skip strict revision validation for verify-only JWT providers#21909

Open
tkren wants to merge 2 commits into
etcd-io:mainfrom
tkren:jwt-avoid-revision-validation-verify-only-token-provider
Open

auth: skip strict revision validation for verify-only JWT providers#21909
tkren wants to merge 2 commits into
etcd-io:mainfrom
tkren:jwt-avoid-revision-validation-verify-only-token-provider

Conversation

@tkren

@tkren tkren commented Jun 4, 2026

Copy link
Copy Markdown

Description

Support for verify-only JWT providers (added in #9883) allows etcd to entrust an external token management system with handling the lifecycle and signing of client tokens, without exposing the private key to etcd. As an example, setting only pub-key in --auth-token 'jwt,pub-key=/path/to/jwt_ecdsa_pub.pem,sign-method=ES256' is enough to initialize a verify-only JWT provider.

Currently, these verify-only providers enforce the same strict auth store revision validation as standard JWT providers that include the priv-key option. This introduces severe operational issues in environments where etcd clients do not use username/password authentication (Authenticate() RPC) and rely solely on externally minted tokens.

The Problem

The current implementation surfaces two major friction points:

  • Token generation overhead: external token management systems must query etcd to fetch the current auth store revision number before they can mint a valid token.

  • Brittle token invalidation: if the auth store revision is bumped (e.g., creating a new user or changing permissions independent of the token's user), all active JWT tokens referencing the older revision are instantly rejected with ErrAuthOldRevision (etcdserver: revision of auth store is old). Because these clients lack passwords, recovering requires an emergency mass-minting and distribution of new tokens to all disconnected clients.

Proposed Solution

Since verify-only JWT providers reject Authenticate() requests with ErrVerifyOnly (auth: token signing attempted with verify-only key), a password race is structurally impossible. Therefore, strict revision checking is unnecessary.

This PR modifies verify-only JWT providers to ignore the incoming revision token claim and instead dynamically use the current auth store revision passed into authInfoFromToken(). This aligns exactly with how TLS client certificate authentication handles identity verification in AuthInfoFromTLS().

Key Advantages

  • Eliminates etcd lookups for minting JWTs: external token management systems no longer need to fetch the current revision. They can simply hardcode 1 (or any value) in the revision token claim.

  • Prevents sudden client disconnections: changes to the auth store will no longer invalidate active tokens. Tokens remain valid until their actual expiration time, regardless of cluster configuration changes.

  • Enables gateway proxies: Unlike TLS common-name authentication, this approach allows the use of gRPC-proxy and gRPC-gateway with externally signed JWTs.

@k8s-ci-robot

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: tkren
Once this PR has been reviewed and has the lgtm label, please assign jmhbnz for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot

Copy link
Copy Markdown

Hi @tkren. Thanks for your PR.

I'm waiting for a etcd-io member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

This modifies the behaviour of verify-only JWT providers to ignore the
revision claim of the JWT token and instead dynamically use the current
auth store revision in authInfoFromToken(). This matches the existing
behavior of TLS client certificate authentication (AuthInfoFromTLS).
Revision validation is not required for verify-only JWT providers as
they cannot handle `Authenticate()` requests.

Signed-off-by: Thomas Krennwallner <teakay@aiven.io>
@tkren tkren force-pushed the jwt-avoid-revision-validation-verify-only-token-provider branch from 924d440 to 1bec50e Compare June 4, 2026 20:55
@tkren tkren marked this pull request as ready for review June 4, 2026 20:55
@tkren tkren marked this pull request as draft June 7, 2026 14:57
Add TestAuthJWTOnlySkipRevision (based on etcd-io#16803) for testing that
directly set JWT tokens (which are signed externally) stay valid after
an auth revision bump, but will be denied access if permissions change
or the user was deleted.

Signed-off-by: Thomas Krennwallner <teakay@aiven.io>
@tkren

tkren commented Jun 7, 2026

Copy link
Copy Markdown
Author

I added an integration test for testing externally signed JWT tokens and their behaviour with permission changes or user deletion.

@tkren tkren marked this pull request as ready for review June 7, 2026 16:05
@tkren

tkren commented Jun 10, 2026

Copy link
Copy Markdown
Author

Hi Etcd maintainers, could you have a look at this PR here? Does it make sense to you, is something missing? I'm looking for feedback in case that I may have missed something here. Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Development

Successfully merging this pull request may close these issues.

2 participants