Python Packages in Artifact Registry (Updated)

Update: Google released an update to Cloud Functions on February 16th, 2022, that makes it very easy to use Artifact Registry, so the content below regarding Cloud Functions is no longer accurate. Check out the Cloud Functions documentation for using private dependencies from Artifact Registry.

Today, Google announced a public preview of support for Python packages (using the Pypi repository format) in Artifact Registry. With this feature, you can create a private repository in your project where you can publish your packages, instead of pushing them to the public repository at Pypi.org.

Uploading Python Packages with Cloud Build

Google recommends authenticating to Artifact Registry PyPi repositories using the Python keyring library. Here is an example of how you can do this in your cloudbuild.yaml :

---
steps:
  - name: python:3.9
    entrypoint: /bin/bash
    dir: package
    args:
      - -c
      - |
        pip install --upgrade pip setuptools twine wheel
        pip install keyring
        pip install keyrings.google-artifactregistry-auth
        mkdir -p ~/.config/pip
        printf "[distutils]\nindex-servers = \n    ${_REPO}\n\n[${_REPO}]\nrepository: https://${_REGION}-pypi.pkg.dev/${PROJECT_ID}/${_REPO}/\n" > ~/.pypirc
        printf "[global]\nindex-url = https://${_REGION}-pypi.pkg.dev/${PROJECT_ID}/${_REPO}/simple/\n" > ~/.config/pip/pip.conf
        python setup.py sdist bdist_wheel
        twine upload --repository-url https://${_REGION}-pypi.pkg.dev/${PROJECT_ID}/${_REPO}/ dist/*

This Cloud Build config handles the following steps:

• install setuptools, twine, wheel, keyring and the Artifacts Registry backend
• create the ~/.config/pip directory
• populate the pip.conf
• build the wheels and sdists
• upload to the registry

and it assumes the PROJECT_ID of the Pypi repository is the same project where Cloud Build is running and depends on the following substitutions:

• _REPO: the name of the Pypi repository (mypypi for example)
• _REGION: the region

Your Cloud Build GitHub Trigger could look something like this:

---
description: Upload Package to Artifact Registry PyPi Repository
filename: package/cloudbuild.yaml
github:
  name: my-package
  owner: my-org
  push:
    branch: ^main$

includedFiles:
  - package/setup.py
name: upload-package
substitutions:
  _REGION: us-east4
  _REPO: mypypi

This way, when you push a new commit to the main branch in GitHub with changes to the package/setup.py file (since you need to update the version anytime you push to a repository), the Cloud Build job will run and will update the package in the Artifact Registry Repository.

Using Python Packages with Cloud Functions

What if you want to use these private Artifact Registry packages in your Cloud Functions? Google provides instructions for how to install a package, so you could use that to install the package in your local environment or to include it in one of container images which you could then use with Cloud Run. But what about Cloud Functions?

The first undocumented trick you need to know is there is support for authenticating with an oauth2accesstoken by adding the following line to the beginning of your requirements.txt.

--extra-index-url https://oauth2accesstoken:{{ACCESS_TOKEN}}@{{REGION}}-pypi.pkg.dev/{{PROJECT}}/{{REPO}}/simple/

where {{ACCESS_TOKEN}} is replaced by the output of:

gcloud auth print-access-token

and {{REGION}}, {{PROJECT}} and {{REPO}} describe the location of your Artifact Registry Pypi repository.

In order to get this working in a cloudbuild.yaml , I did a couple of other things.

1. Renamed my requirements.txt to requirements.txt.tmpl and added the templated line above with the {{}}’s .
2. Added requirements.txt to my .gitignore so that the file with the access token never gets committed to GitHub.
3. Added !requirements.txt to my .gcloudignore so that the requirements.txt file does get pushed when I run gcloud functions deploy.
4. Added a shell script to prepare the requirements.txt file called prepare-requirements.sh (see below).
5. Added a step to the cloudbuild.yaml that deploys my Cloud Function

Here is an example of the prepare-requirements.sh shell script that turns the requirements.txt.tmpl into the requirements.txt with the --extra-index-url populated.

#!/bin/bash
ACCESS_TOKEN="$(gcloud auth print-access-token)"
REGION="${REGION:="us-east4"}"
REPO="${REPO:="mypypi"}"
PROJECT="${PROJECT:="broad-gaia-karlsson"}"
sed -e "s/{{ACCESS_TOKEN}}/${ACCESS_TOKEN}/" \
    -e "s/{{PROJECT}}/${PROJECT}/" \
    -e "s/{{REGION}}/${REGION}/" \
    -e "s/{{REPO}}/${REPO}/" \
    requirements.txt.tmpl > requirements.txt
echo "Created requirements.txt from requirements.txt.tmpl"

Then, in my cloudbuild.yaml for deploying the Cloud Function, I added a step at the beginning to prepare the requirements.txt.

---
steps:
  - name: gcr.io/google.com/cloudsdktool/cloud-sdk:alpine
    dir: functions/my-fancy-function
    entrypoint: bash
    args:
      - ../../scripts/prepare-requirements.sh
    env:
      - PROJECT=${PROJECT_ID}
      - REGION=${_REGION}
      - REPO=${_REPO}
- name: gcr.io/google.com/cloudsdktool/cloud-sdk:alpine
    dir: functions/my-fancy-function
    entrypoint: gcloud
    args:
      - functions
      - deploy
      - my-fancy-function
      - --memory=1024MB
      - --project=${PROJECT_ID}
      - --region=us-east4
      - --runtime=python38
      - --timeout=300
      - --trigger-http
      - -q

Finally, I created a Cloud Build GitHub Trigger to deploy the Cloud Function:

---
description: Deploy my-fancy-function
filename: functions/my-fancy-function/cloudbuild.yaml
github:
  name: my-package
  owner: my-org
  push:
    branch: ^main$

includedFiles:
  - functions/my-fancy-function/**
name: deploy-my-fancy-function
substitutions:
  _REGION: us-east4
  _REPO: mypypi

Now, when I push a new commit to the main branch in GitHub, if it includes changes to this Cloud Function, the Cloud Build Job will run and will populate the requirements.txt and deploy the function.

Conclusion

Support for Pypi packages in Artifact Registry is currently in the Preview launch stage, so it is very possible that this process will change, almost certainly to make it easier to do than the current keychain approach. In the meantime, you should be able to start using private Pypi repositories with your Cloud Functions today!

Check back here for updates as things change!