Read & Write for issues is set in my integration. Creating an issue works just fine. However setting the labels is not working. According to the docs I need push access to the repo to set the labels. This makes no logical sense in a permission settings.
Push access would allow the integration to push to the repo. However I only want to work with the issues and comments to issues to limit the access of the integration. Given that if I make a new integration with Push access this would leave a potential security hole in an integration only designed to manipulate the issues.