Turn on the "Allow npm dist-tag" permission and drop the token you kept just for tags.
Since September 30, npm trusted publishing can manage dist-tags with short-lived OIDC credentials. It's an opt-in "Allow npm dist-tag" permission, so you can drop the token you kept just for tags.