Project Specific Buckets on GCP

Introduction

This tutorial will walk you through how to configure a Google Cloud Storage bucket and service account for use with Tator on Google Cloud Platform (GCP), how to register the bucket with your organization in Tator, and how to use the bucket in a specific Tator project.

To begin this tutorial, you will need GCP storage admin privileges on a GCP project, the gsutil command line utility, and administrative permissions for your organization in Tator.

Create and configure a GCS bucket on GCP

1. From the GCP storage browser, click Create Bucket.

2. Set a bucket name (we will use my-tator-bucket), region, default storage class, and access control:

   

3. Click CREATE

4. Enable bucket versioning from the command line with gsutil:

   gsutil versioning set on gs://my-tator-bucket

5. Create a file containing the lifecycle policies for the bucket (we'll call it lifecycle_config.json):

   {
     "rule": [
       {
         "action": {
           "storageClass": "COLDLINE",
           "type": "SetStorageClass"
         },
         "condition": {
           "daysSinceNoncurrentTime": 30,
           "isLive": false
         }
       },
       {
         "action": {
           "type": "Delete"
         },
         "condition": {
           "daysSinceNoncurrentTime": 365,
           "isLive": false
         }
       }
     ]
   }

6. Set the lifecycle policies using gsutil (using the filename and bucket name from before)

   gsutil lifecycle set lifecycle_config.json gs://my-tator-bucket

7. Create a file containing the CORS configuration (we'll call it cors.json):

   [
     {
       "maxAgeSeconds": 3600,
       "method": ["GET", "PUT", "POST", "HEAD"],
       "origin": ["https://www.tatorapp.com", "https://cloud.tator.io"],
       "responseHeader": ["*"]
     }
   ]

8. Set CORS configuration with gsutil (using the filename and bucket name from before)

   gsutil cors set cors.json gs://my-tator-bucket

Your bucket is now ready to use with your project.

Create and configure an IAM service account on GCP

1. Follow the instructions for creating a service account
2. Give it the role of "Storage Admin"
3. Navigate to service accounts and click on the newly created service account
4. Click on the Keys tab
5. Click ADD KEY > Create new key
6. Use key type JSON (the default)
7. Click CREATE
8. Make note of the filename (we will use tator-keys.json) and location. If this file is lost, its contents cannot be recreated and a new key must be created.

The IAM account is now ready to use with Tator.

Create a Bucket and Project in Tator

We will use tator-py to register the bucket to our organization, then create a project that will use that bucket. You can also change the bucket property of an existing project.

1. Use django shell to create a Bucket object, loading the GCP service account info from the file created in the previous steps:

import json
with open("tator-keys.json", "r") as fp:
    key_info = json.load(fp)

b=Bucket.objects.create(organization=Organization.objects.get(pk=ORG_ID), name=BUCKET-NAME, config=key_info, store_type=ObjectStore.GCP, archive_sc='STANDARD')

2. Use the bucket ID to create a project:

   p=Project.objects.get(pk=Foo)
   p.bucket = b
   p.save()

The project is now ready to use.