Manage Secrets

NeMo Microservices provides an interface to manage user-defined secrets within the NeMo Microservices platform. This allows you to securely store API keys to integrate with other providers. For example, you might want to store a Hugging Face API token as a secret to access models, or a Weights & Biases API key for experiment tracking.

Concepts

A secret is a named resource that holds sensitive information, such as API keys or tokens.

• Secrets are unique within a workspace, meaning they exist within a specific workspace in the NeMo Microservices platform. Operations which utilize secrets must specify the workspace in which the secret resides.

• All secrets are encrypted at rest. Once a secret is created, its value cannot be retrieved through the API—not even by workspace Admins. Users can view secret metadata (name, description, timestamps) but can only update or delete the secret, never read its value back.

• Only Platform Administrators can retrieve secret values. This restricted access ensures that sensitive credentials remain protected even from users with full workspace administrative privileges.

• Platform services access secrets internally as needed to facilitate user-requested operations, for example, when a job requires access to a Hugging Face model, or when reporting metrics to Weights & Biases.

• Secrets referenced by other resources (e.g., customization configs) are referenced by their name and workspace.

  • A reference to a secret by another resource is stored in the format workspace/secret_name.

Access Control

When authentication is enabled, secrets follow role-based access control:

Role	List	View Metadata	Create	Update	Delete	Read Value
Viewer	✓	✓	✗	✗	✗	✗
Editor	✓	✓	✓	✓	✓	✗
Admin	✓	✓	✓	✓	✓	✗
Platform Admin	✓	✓	✓	✓	✓	✓

Important

Even workspace Admins cannot read secret values. This is by design—secrets contain sensitive credentials that should not be exposed through the API. Only Platform Administrators have the elevated permissions required to retrieve secret values when absolutely necessary.

Naming Requirements

Secret names must follow these conventions:

• Allowed characters: Letters (a-z, A-Z), numbers (0-9), hyphens (-), underscores (_), and dots (.)

• Maximum length: 255 characters

• Must not be empty

Tip

Best practice: Use lowercase letters, numbers, and hyphens only (e.g., my-api-key). This ensures compatibility with Kubernetes naming conventions if your secrets are used in job environments.

Example valid names:

• hf-token

• nim-api-key

• wandb-key-production

Understanding Workspaces

In NeMo Microservices, a workspace is used to organize and isolate resources. Workspaces function similarly to namespaces - they provide logical separation between different sets of resources.

• The default workspace is named default

• Most API operations require specifying a workspace parameter

• Secrets, model providers, jobs, and other resources exist within a workspace

• Resources in one workspace cannot directly access resources in another workspace

Tip

If you’re just getting started, use the default workspace. You can create additional workspaces later to organize resources by team, project, or environment.

Operations

To manage secrets, you can use the NeMo Microservices SDK. Below are examples of how to create, list, update, and delete secrets.

To work with secrets, you can use the following import and initialization:

from nemo_microservices import NeMoMicroservices
sdk = NeMoMicroservices(base_url="http://localhost:8080")

Creating Secrets

Creating a secret involves specifying a name, workspace, and the initial secret data.

secret = sdk.secrets.create(
    name="my-secret",
    workspace="default",
    data="<your-secret-data>"
)

Listing Secrets

You can list all secrets within a specific workspace. This returns metadata only.

# List all secrets in the "default" workspace
secrets = sdk.secrets.list(workspace="default")
for secret in secrets.data:
    print(secret.name)

Retrieving a Secret

To retrieve a single secret by name, use the retrieve method. This returns the secret’s metadata only.

# Get a specific secret's metadata
secret = sdk.secrets.retrieve(
    name="my-secret",
    workspace="default"
)
print(f"Secret: {secret.name}, Created: {secret.created_at}")

Updating Secrets

To update a secret, you can change it’s data field.

# Update the content of the secret
updated_secret = sdk.secrets.update(
    name="my-secret",
    workspace="default",
    data="<your-new-secret-data>"
)

Deleting Secrets

Deleting a secret will remove the secret from the platform.

# Delete a secret
sdk.secrets.delete(
    name="my-secret",
    workspace="default"
)

Use Cases

Hugging Face API Tokens

A Hugging Face API token secret allows NeMo Microservices to authenticate with the Hugging Face Hub for accessing models and repositories.

How are Hugging Face API Tokens Used?

NeMo Microservices uses Hugging Face api tokens to authenticate requests to the Hugging Face Hub. This enables users to download private models and upload customized models to their Hugging Face repositories.

Creating a Hugging Face Secret

To create a Hugging Face API key secret, use the NeMo Microservices SDK to create a new secret with your Hugging Face token.

from nemo_microservices import NeMoMicroservices
sdk = NeMoMicroservices(base_url="http://localhost:8080")

hf_secret = sdk.secrets.create(
    name="my-hugging-face-token",
    workspace="default",
    data="<your-hugging-face-token>"
)

Weights & Biases Keys

NeMo Microservices users may integrate with Weight & Biases by providing their API key as a secret. This allows the NeMo Customizer service to report training metrics to the user’s W&B account.

How are Weights & Biases Keys Used?

When a Weight and Biases key is provided, the NeMo Customizer service sends telemetry data to W&B including:

• Job ID

• Training loss

• Validation loss

• Other relevant metrics

All metrics are stored under the user’s nvidia-nemo-customizer project in their W&B account.

Creating a Weights & Biases Secret

To create a W&B secret, use the NeMo Microservices SDK to create a new secret with your W&B API key.

from nemo_microservices import NeMoMicroservices
sdk = NeMoMicroservices(base_url="http://localhost:8080")

wandb_secret = sdk.secrets.create(
    name="my-wandb-key",
    workspace="default",
    data="<your-wandb-api-key>"
)

Encryption Configuration

Important

For production deployments, it is critical to configure secure encryption for secrets at rest. The default local encryption provider is intended for initial deployment or evaluation.

Rotating or Migrating Encryption Keys

NeMo Microservices supports rotation of encryption keys used to secure secrets at rest. Key rotation can be performed by Platform Administrators to comply with security policies or to migrate between encryption providers.

To rotate or migrate encryption keys:

1. Add a new provider configuration to the platform config’s secrets encryption settings. You may name the new provider as desired (e.g., v2), and set it as the current_provider.

   secrets:
     encryption:
       current_provider: new_provider_name
       providers:
         secret_key:
           # Keep the existing provider for decryption of old secrets
           v1:
             value: "<old-encryption-key>"
           # Add the new provider for encrypting new secrets
           v2:
             value: "<new-encryption-key>"
             # from_env: "NEW_ENCRYPTION_KEY_ENV_VAR" # Optional alternative to load from environment variable
   

2. Re-deploy the platform services to apply the new configuration.

3. As the platform administrator role, call the /v2/secrets/rotate-encryption-keys endpoint to initiate the rotation process. This will re-encrypt all existing secrets using the new provider. You may also initiate the rotation using the NeMo Microservices SDK:

   from nemo_microservices import NeMoMicroservices
   sdk = NeMoMicroservices(base_url="http://localhost:8080")
   
   # Initiate the key rotation process
   sdk.secrets.admin.rotate_encryption_keys()
   

   This operation may take some time depending on the number of secrets stored. During this process, both the old and new encryption providers must be available to ensure seamless access to secrets. Do not remove the old provider until rotation is confirmed completed.

   If any errors occur during rotation, they will be logged for review. You can retry the rotation operation as needed until all secrets are successfully re-encrypted.

4. To validate the rotation completed successfully, you can check the logs for the following confirmation message:

   Completed rotation of encryption keys for all secrets
   

5. Once rotation is confirmed, you may optionally remove the old provider configuration from the platform settings to prevent its further use.