Skip to content

Signed in as

Sign in to Curious Dev Learn

Sign in to verify labs, save your progress and get a certificate. Lessons stay open without an account.

Module 051 h 45 minLab

Releases without fear

Ship a new version to a tagged URL first, move traffic in steps you control from Pulumi config, and roll back in seconds without a rebuild.

Someone renamed url to targetUrl in the link response because it read better. The tests were updated in the same pull request, so they passed. The web client was redeployed the same afternoon. The mobile app, which lives in a store and updates on users’ schedule, was not. For 40 minutes every link tapped in the app failed, and the “rollback” was a revert commit that went through the whole pipeline again: 12 minutes of building an image that already existed.

Two things went wrong, and they are separate. The release model was all-or-nothing: the moment the new version was ready, it got 100 % of users. And the change itself was breaking: two versions of the API could not serve the same clients. This module fixes both. You will make “deploy” and “release” two different actions, put the traffic split in Pulumi config, and learn which API changes are safe to roll out gradually.

Revisions are immutable, traffic is separate

Section titled “Revisions are immutable, traffic is separate”

Every time you change anything in a Cloud Run service’s template (image, env vars, probes, concurrency, the secret volume), Cloud Run creates a new revision. A revision is a frozen snapshot: one image digest plus one configuration. You cannot edit a revision. You can only create a new one.

The service has a second, independent piece of configuration: traffic. Traffic is a list of targets, each saying “this revision gets this percentage of requests”. Creating a revision and sending traffic to it are two different operations that happen to look like one, because of a default.

That default is TRAFFIC_TARGET_ALLOCATION_TYPE_LATEST: “send 100 % to whatever the latest ready revision is”. Since module 1 your service has been running in this mode, because you never set Traffics at all and Cloud Run filled it in. In this mode a new revision gets all traffic as soon as its startup probe passes. That is exactly the incident above.

The alternative is TRAFFIC_TARGET_ALLOCATION_TYPE_REVISION: “send N % to the revision with this exact name”. When every target is pinned like this, a new revision is created, becomes ready and gets zero traffic until you change the traffic list. Deploy becomes cheap and safe; release becomes a deliberate, separate step.

Because revisions are immutable and Cloud Run keeps old ones around, a rollback is not a rebuild. The old revision still exists, with its old image and its old env. Rolling back means pointing traffic at it again, which takes seconds.

Have a look at what you have right now:

Terminal window
export PROJECT_ID=your-project-id
export REGION=europe-west1
# All revisions of the service, newest first
gcloud run revisions list --service sniplink-api --region $REGION
# The current traffic configuration (read-only)
gcloud run services describe sniplink-api --region $REGION \
--format="yaml(spec.traffic,status.traffic)"

You will see auto-generated names like sniplink-api-00007-zuf and a single traffic entry with latestRevision: true. Both are about to change.

Auto-generated names are fine for machines and bad for humans. “Send 10 % to sniplink-api-00008-qix” is a sentence nobody can review in a pull request. So the first change is to name revisions explicitly, through Template.Revision.

The rule: a revision name must start with the service name followed by a dash. sniplink-api-v1 is valid; v1 or sniplink-v1 is rejected by the API. gcloud run deploy --revision-suffix=v1 hides this by prepending the service name for you; the API and therefore Pulumi do not, so you write the full name.

Names are also unique per service, forever, or at least for as long as the revision exists. Once sniplink-api-v1 has been created, you cannot create another sniplink-api-v1 with a different image. I come back to this in the gotchas, because it is the first error everyone hits.

At the same time, give the app a version it can report. LabKit already returns version from /_lab/info, reading the APP_VERSION env var; until now you never set it, so it was empty. From now on, the revision name and the version move together, and both come from stack config.

Here is the design I use. The Pulumi program does not hard-code any percentages. It reads five config values and builds the Traffics list from them:

Key Example What it means
sniplink:release v2 Suffix of the revision the template will create
sniplink:version 2.0.0 Value of APP_VERSION in that revision
sniplink:stableRevision sniplink-api-v1 The revision most users get
sniplink:canaryRevision sniplink-api-v2 Optional; the revision behind the canary tag
sniplink:canaryPercent 10 Share of traffic the canary gets (0 to 100)

Every stage of a release is then a pulumi config set and a pulumi up. The diff is two or three lines in Pulumi.dev.yaml, which you can review, revert and read in git history six months later. That is the whole point of doing this in code rather than in the console’s traffic slider.

infra/Program.cs
using System;
using System.Collections.Generic;
using System.Linq;
using Pulumi;
using Gcp = Pulumi.Gcp;
using Pulumi.Gcp.CloudRunV2;
using Pulumi.Gcp.CloudRunV2.Inputs;
return await Deployment.RunAsync(() =>
{
const string serviceName = "sniplink-api";
var config = new Config("sniplink");
var region = new Config("gcp").Require("region");
var image = config.Require("image");
var release = config.Require("release"); // "v1", "v2", ...
var version = config.Require("version"); // "1.0.0", "2.0.0", ...
var stableRevision = config.Require("stableRevision"); // "sniplink-api-v1"
var canaryRevision = config.Get("canaryRevision"); // null when no canary
var canaryPercent = config.GetInt32("canaryPercent") ?? 0;
var studentId = config.Require("studentId");
// Fail at preview time, not halfway through an update.
if (canaryPercent is < 0 or > 100)
throw new ArgumentException("canaryPercent must be 0..100");
if (canaryRevision is null && canaryPercent != 0)
throw new ArgumentException("canaryPercent > 0 needs canaryRevision");
if (canaryRevision == stableRevision)
throw new ArgumentException("canary and stable must differ");
var traffics = new InputList<ServiceTrafficArgs>
{
new ServiceTrafficArgs
{
Type = "TRAFFIC_TARGET_ALLOCATION_TYPE_REVISION",
Revision = stableRevision,
Percent = 100 - canaryPercent,
},
};
if (canaryRevision is not null)
{
traffics.Add(new ServiceTrafficArgs
{
Type = "TRAFFIC_TARGET_ALLOCATION_TYPE_REVISION",
Revision = canaryRevision,
Percent = canaryPercent,
Tag = "canary",
});
}
// … repo from module 1; runtimeSa, labSecret and IAM from module 3 (unchanged)
var service = new Service(serviceName, new ServiceArgs
{
Name = serviceName,
Location = region,
DeletionProtection = false,
Ingress = "INGRESS_TRAFFIC_ALL",
Template = new ServiceTemplateArgs
{
Revision = $"{serviceName}-{release}",
ServiceAccount = runtimeSa.Email,
MaxInstanceRequestConcurrency = 10,
// … Scaling from module 4, Volumes from module 3 (unchanged)
Containers =
{
new ServiceTemplateContainerArgs
{
Image = image,
Envs =
{
new ServiceTemplateContainerEnvArgs
{
Name = "LABKIT_STUDENT_ID",
Value = studentId,
},
new ServiceTemplateContainerEnvArgs
{
Name = "APP_VERSION",
Value = version,
},
},
// … ports, resources, probes and volume mounts from modules 1, 3 and 4
},
},
},
Traffics = traffics,
}, new CustomResourceOptions { DependsOn = { repo, labKeyAccess, labKeyVersion } });
// … public invoker binding from module 1 (unchanged)
return new Dictionary<string, object?>
{
["url"] = service.Uri,
["canaryUrl"] = service.TrafficStatuses.Apply(statuses =>
statuses.FirstOrDefault(s => s.Tag == "canary")?.Uri ?? ""),
["latestRevision"] = service.LatestReadyRevision,
};
});

A few lines deserve a reason each:

  • Revision = $"{serviceName}-{release}". The prefix rule lives in one place, so config can never produce an invalid name.
  • APP_VERSION next to the revision name. A revision you cannot identify from the outside is a revision you cannot verify. The lab relies on this.
  • Both targets are ..._REVISION. No target points at “latest”, so nothing moves unless you move it.
  • The guard clauses. Percentages that do not add up to 100 or a canary pointing at the stable revision are rejected by the API anyway, but only after Pulumi has started the update. Failing in pulumi preview is cheaper.
  • canaryUrl from TrafficStatuses. The tagged URL is assigned by Cloud Run, not by you. The status block reports it once the tag exists.

Apart from the config reads, the revision name, one env var and the Traffics list, this is the module 4 program unchanged. Leave Traffics out and Cloud Run falls back to “100 % to latest”, which is why I always set it explicitly once a service has users.

Before you can canary anything, the current version needs a name. Nothing changes in the app here; you only rename what is running and pin traffic to it.

Terminal window
cd infra
pulumi config set sniplink:release v1
pulumi config set sniplink:version 1.0.0
pulumi config set sniplink:stableRevision sniplink-api-v1
pulumi config rm sniplink:canaryRevision # no-op if not set
pulumi config set sniplink:canaryPercent 0
pulumi up

The preview shows one update on the service: template.revision goes from empty to sniplink-api-v1, an APP_VERSION env var appears, and traffics changes from “latest 100” to “sniplink-api-v1 100”. Because the template changed, Cloud Run creates a new revision, and because the traffic list names that revision, it gets all traffic in the same update.

Check it:

Terminal window
SERVICE_URL=$(pulumi stack output url)
curl -s $SERVICE_URL/_lab/info | jq '{service, revision, version}'
# { "service": "sniplink-api", "revision": "sniplink-api-v1", "version": "1.0.0" }

Commit Pulumi.dev.yaml and Program.cs. From now on, the stack config file is your release log.

Now the actual v2. The incident was caused by a breaking change, so before writing it, the rule that makes a canary possible at all:

During a canary, two versions serve the same clients at the same time. A user may create a link through v2 and read it through v1 a second later. A mobile app built against v1 will hit v2 for some of its requests. If the two versions disagree on the contract, the canary does not reduce risk; it just spreads the breakage to 10 % of requests, randomly, which is harder to debug than 100 %.

So v2 of Sniplink makes an additive change: links get an optional expiresAt. Clients that do not know about it ignore it; clients that do can set it. Nothing existing is renamed, removed, or made required.

src/Sniplink.Api/Links.cs
using System.Text.Json.Serialization;
namespace Sniplink.Api;
public sealed record CreateLinkRequest(string Url, DateTimeOffset? ExpiresAt = null);
public sealed record LinkResponse(
string Slug,
string Url,
[property: JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
DateTimeOffset? ExpiresAt = null);

Until now the links lived in the ConcurrentDictionary<string, string> from the starter, which has no room for an expiry. Move it behind a small store; it is still in memory, still per instance:

src/Sniplink.Api/LinkStore.cs
using System.Collections.Concurrent;
using System.Security.Cryptography;
namespace Sniplink.Api;
public sealed record Link(string Slug, string Url, DateTimeOffset? ExpiresAt)
{
public bool IsExpired(DateTimeOffset now) => ExpiresAt is { } exp && exp <= now;
}
public interface ILinkStore
{
Link Add(string url, DateTimeOffset? expiresAt);
bool TryGet(string slug, out Link link);
}
// In-memory on purpose, per instance. Persistence is the next course.
public sealed class InMemoryLinkStore : ILinkStore
{
private readonly ConcurrentDictionary<string, Link> _links = new();
public Link Add(string url, DateTimeOffset? expiresAt)
{
while (true)
{
var slug = RandomNumberGenerator.GetString("abcdefghijkmnpqrstuvwxyz23456789", 6);
var link = new Link(slug, url, expiresAt);
if (_links.TryAdd(slug, link)) return link;
}
}
public bool TryGet(string slug, out Link link) => _links.TryGetValue(slug, out link!);
}

Register it with builder.Services.AddSingleton<ILinkStore, InMemoryLinkStore>();, delete the old dictionary and the starter’s CreateLink record, and change the two endpoints that create and resolve links. GET /api/links/{slug} moves to the store the same way and returns a LinkResponse. Everything else in Program.cs (the PORT binding, /health, the module 4 health checks and shutdown settings, LabKit) stays as it is.

src/Sniplink.Api/Program.cs
// …
app.MapPost("/api/links", (CreateLinkRequest req, ILinkStore store) =>
{
if (!Uri.TryCreate(req.Url, UriKind.Absolute, out var uri) || uri.Scheme is not ("http" or "https"))
return Results.ValidationProblem(new Dictionary<string, string[]> { ["url"] = ["Must be an absolute http or https URL."] });
if (req.ExpiresAt is { } exp && exp <= DateTimeOffset.UtcNow)
return Results.ValidationProblem(new Dictionary<string, string[]> { ["expiresAt"] = ["Must be in the future."] });
var link = store.Add(req.Url, req.ExpiresAt);
return Results.Created($"/api/links/{link.Slug}",
new LinkResponse(link.Slug, link.Url, link.ExpiresAt));
});
app.MapGet("/{slug}", (string slug, ILinkStore store) =>
store.TryGet(slug, out var link) && !link.IsExpired(DateTimeOffset.UtcNow)
? Results.Redirect(link.Url)
: Results.NotFound());
// …

Why each choice:

  • Optional in the request, with a default. A v1 client posting { "url": "…" } gets exactly the v1 behaviour.
  • Omitted from the response when null. Most JSON clients tolerate unknown fields, but a few strict ones (some generated clients, some mobile decoders) do not tolerate an unexpected null where they expected nothing. Omitting it means a v1 client sees a byte-for-byte v1 response unless it opted in.
  • Expired links return 404, not a new status. A v1 client already handles 404 for an unknown slug. Introducing 410 Gone would be more precise and also a contract change for every client that switches on status codes. That can come later, behind a deliberate decision.
  • Only http and https URLs. Uri.TryCreate with UriKind.Absolute alone accepts /foo on Linux, where it parses as file:///foo, and a redirect to that goes nowhere useful. This is stricter than v1, but no working client relies on creating a link that cannot be followed.

Sometimes you really do want targetUrl instead of url, or a different shape altogether. Two patterns keep that safe:

  • A new route for a new contract. /api/v2/links returns the new shape; /api/links keeps the old one, backed by the same code. Old clients keep working until you can prove nobody calls the old route, from request logs, not from hope.
  • Expand and contract. For a field rename: release one version that writes both url and targetUrl and reads either (expand). Move every client to targetUrl. Only then release a version that drops url (contract). Each step is backward compatible with the one before it, so each step can be canaried.

Expand-and-contract is slower: three releases instead of one. That is the actual cost of having clients you do not deploy yourself, and the incident above is what skipping it looks like.

Build and push v2 with the Docker commands from module 2, using the version as the image tag:

Terminal window
docker build --platform linux/amd64 -t $REGION-docker.pkg.dev/$PROJECT_ID/sniplink/api:2.0.0 .
docker push $REGION-docker.pkg.dev/$PROJECT_ID/sniplink/api:2.0.0

Now deploy v2 without releasing it. The template moves to v2; traffic stays on v1; the new revision gets a tag so you can reach it directly.

Terminal window
pulumi config set sniplink:image $REGION-docker.pkg.dev/$PROJECT_ID/sniplink/api:2.0.0
pulumi config set sniplink:release v2
pulumi config set sniplink:version 2.0.0
pulumi config set sniplink:canaryRevision sniplink-api-v2
pulumi config set sniplink:canaryPercent 0
# stableRevision stays sniplink-api-v1
pulumi up

Read the preview carefully, because this is where the model clicks: the template changes (new image, new revision name, new version), and the traffic list gains a second entry with percent: 0 and tag: canary. The first entry, sniplink-api-v1 at 100, is unchanged. All production traffic keeps going to v1 the whole time.

A tag gives a revision its own URL, regardless of its traffic percentage. The format is the tag, three dashes, then the service’s normal host:

https://sniplink-api-123456789012.europe-west1.run.app # main URL
https://canary---sniplink-api-123456789012.europe-west1.run.app # canary tag

Every service also keeps an older hash-based host, such as sniplink-api-abc123xyz-ew.a.run.app, and a tag works with it too: canary---sniplink-api-abc123xyz-ew.a.run.app. The URL that Cloud Run reports for a tag (in TrafficStatuses, and so in canaryUrl, and in gcloud) uses that hash-based form, so do not be surprised when it does not match the main URL above. You do not need to construct either by hand:

Terminal window
# From the stack (recommended)
pulumi stack output canaryUrl
# From Cloud Run directly (read-only): every traffic target with its URL
gcloud run services describe sniplink-api --region $REGION \
--format="table(status.traffic[].tag,status.traffic[].revisionName,status.traffic[].percent,status.traffic[].url)"

Requests to the tagged URL always go to sniplink-api-v2. They do not count towards or against the percentage split, so you can test v2 in production, with the production runtime SA, the production secret and the production network path, while no user sees it.

Terminal window
CANARY_URL=$(pulumi stack output canaryUrl)
curl -s $CANARY_URL/_lab/info | jq '{service, revision, version}'
# { "service": "sniplink-api", "revision": "sniplink-api-v2", "version": "2.0.0" }
# The new field, opt-in
curl -s -X POST $CANARY_URL/api/links \
-H 'content-type: application/json' \
-d '{"url":"https://example.com","expiresAt":"2030-01-01T00:00:00Z"}'
# An old-style request must still get an old-style response
curl -s -X POST $CANARY_URL/api/links \
-H 'content-type: application/json' \
-d '{"url":"https://example.com"}'
# { "slug": "…", "url": "https://example.com" } no expiresAt key

That last request is the most important test of the release. If you have a contract test suite for v1 clients, this is where you point it at the canary URL.

When the tagged revision looks right, give it a slice of real traffic:

Terminal window
pulumi config set sniplink:canaryPercent 10
pulumi up

The preview shows exactly one change: traffics[0].percent 100 → 90 and traffics[1].percent 0 → 10. No new revision, no image, no build. The update takes as long as Pulumi’s own round trip; the routing change itself applies in seconds.

Now watch it. In the read-only console, the service’s Metrics tab can break request count and latency down by revision; Cloud Logging can filter on resource.labels.revision_name="sniplink-api-v2". Compare 5xx rate and latency of the canary against the stable revision over a period that covers your normal traffic pattern. For Sniplink in a lab that is a few minutes; for a real service I want at least one peak hour.

Terminal window
# Errors from the canary only, last 30 minutes
gcloud logging read \
'resource.type="cloud_run_revision"
resource.labels.service_name="sniplink-api"
resource.labels.revision_name="sniplink-api-v2"
severity>=ERROR' \
--freshness=30m --limit=20

If it holds, promote. Promotion means v2 becomes the stable revision and the canary entry goes away:

Terminal window
pulumi config set sniplink:stableRevision sniplink-api-v2
pulumi config rm sniplink:canaryRevision
pulumi config set sniplink:canaryPercent 0
pulumi up

Removing the canary entry also removes the canary tag, and the tagged URL stops resolving. The next release starts at stage b again with release v3.

Rollback is the reverse of whichever step you last took, and it never touches the template.

  • During the canary (v2 at 10 %): pulumi config set sniplink:canaryPercent 0 and pulumi up. v1 is back to 100 %, and the tag stays, so you can keep debugging v2 on its own URL with zero users on it.
  • After promotion (v2 at 100 %): pulumi config set sniplink:stableRevision sniplink-api-v1 and pulumi up. The sniplink-api-v1 revision still exists with its old image and its old APP_VERSION, so it serves again immediately.

Compare that to the incident: 12 minutes of rebuilding a known-good image versus one config line and one pulumi up. The template still says v2 after a rollback, which is correct. “What was last deployed” and “what serves traffic” are different questions, and Cloud Run now answers them separately.

Cloud Run gives you the mechanism: immutable revisions, percentage splits and tags. It does not give you a policy. Specifically, it will not:

  • watch error rate or latency of the canary and roll back on its own;
  • move from 10 % to 25 % to 50 % on a schedule;
  • stop a promotion because a verification test failed.

All of that is still you, looking at a dashboard and running pulumi up. For one service and a handful of releases a month, that is honestly fine, and it is what most teams I have worked with actually do.

When you need the automation, Google’s first-party tool is Cloud Deploy. It supports canary strategies for Cloud Run targets: phased percentages, verification jobs between phases, approvals and promotion between environments. It has its own model (delivery pipelines, targets, releases, rollouts) and its own way of owning the traffic split, which would conflict with Pulumi managing Traffics directly. That design work is out of scope for this course; the point here is to know the tool exists, and that metric-driven rollback is something you add, not something Cloud Run does by default.

Goal: leave the service with v1 serving all regular traffic and v2 reachable on the canary tag, both managed by Pulumi config.

  1. Name the current release sniplink-api-v1 with APP_VERSION starting with 1. (for example 1.0.0) and pin 100 % of traffic to it (stage a).
  2. Implement the additive expiresAt change, build and push a v2 image.
  3. Deploy it as sniplink-api-v2 with APP_VERSION starting with 2., at 0 % and tag canary (stage b).
  4. Optionally run stage c: 10 %, then roll back to 0 %. Rehearsing the rollback once, while nothing is on fire, is worth the five minutes.
  5. Make sure canaryPercent is 0 when you click Verify (see below).
  6. Paste pulumi stack output url as the service URL and pulumi stack output canaryUrl as the canary URL.

Check your lab

Loading your lab…

Sign in to verify this lab and save your progress. Everything above works without an account.

  • Your Cloud Run service URL (pulumi stack output url)
  • Your canary tag URL (pulumi stack output canaryUrl)

5 checks run against the service you deployed.

Some checks are self-reported: the checker trusts what your service says about itself and verifies the rest from the outside.

What the checks verify:

  • Baseline, twice. POST /_lab/verify on both URLs returns a valid Google ID token for your project. This proves both URLs are real Cloud Run endpoints in the same project.
  • Canary tag. The canary URL’s host starts with canary---, so it is a tag, not a second service; the service (K_SERVICE) and project are identical on both URLs, and the revision (K_REVISION) differs.
  • Main is v1. GET /_lab/info on the main URL reports a version starting with 1..
  • Canary is v2. The same endpoint on the canary URL reports a version starting with 2..

The version checks are self-reported: the value is whatever you put in APP_VERSION. Service and revision names come from env vars Cloud Run sets itself, which is stronger, but LabKit still reports them in the response body rather than in the signed token. A strict check would read the service’s traffic configuration through the Cloud Run Admin API, which is what the future Verified tier does.

Why 0 % at check time: at 10 %, roughly one in ten requests to the main URL lands on v2, so “main is v1” would fail at random. The checker could sample many requests and accept a ratio, but then a pass or fail would depend on luck and on how many instances happen to be warm. A check that can flip on a re-run is worse than no check, so the lab only asserts things that are true with certainty: the main URL at 0 % canary always serves v1, and the tag always serves v2.

Keep the stack. Module 6 builds the CI/CD pipeline on top of this exact service. The pipeline keeps this model (stable revision, canary percentage and canary tag in stack config) and replaces only the hand-typed release with a revision suffix that every build gets from its commit.

If you are pausing the course for a while, you can tidy up without losing the setup:

Terminal window
# Optional: drop the canary tag so nothing points at v2
pulumi config rm sniplink:canaryRevision
pulumi config set sniplink:canaryPercent 0
pulumi up

Idle revisions with 0 % traffic and no minimum instances cost nothing to keep. Only run pulumi destroy if you are stopping the course entirely; module 6 would then need modules 1 to 5 re-applied first.

  • A revision is an immutable snapshot of image plus config; traffic is a separate list that decides which revisions get requests.
  • TRAFFIC_TARGET_ALLOCATION_TYPE_LATEST makes every deploy a release; pinning targets by revision name separates the two.
  • Explicit revision names, a version env var and traffic driven by stack config turn each release stage, including rollback, into a reviewable one-line change.
  • A tag gives a revision a stable canary--- URL so you can test it in production at 0 % traffic.
  • A canary only reduces risk if both versions can serve the same clients: additive changes, new routes for breaking ones, expand-and-contract for renames.