Pulling a Long-Lived Key Out of the Client: Governing an Inherited Upload Chain

I took over a mini program close to its delivery deadline, with several modules still needing rework. The question at that stage was not how to design an ideal module, but how to decide under delivery pressure: which features to restore first, which risks to keep on the books, and which old paths eventually to remove.

Uploads were one of them. Files went straight from the client to object storage without passing through the business server, and the client computed its own signature — so the long-lived key lived in the frontend.

Phase one: stopping the bleeding is not governance

The problem did not surface through a security audit. It surfaced through a 400 error.

Uploads started failing one day. Following the chain led to an empty key in the frontend configuration. The smallest, fastest fix was to put the existing real key back in. The submission and onboarding upload flows passed live tests immediately afterwards.

From a business perspective, uploads worked again. From an engineering perspective, the risk had only just become concrete: the fix answered “can it upload,” not “who can upload.” The key went from empty to usable, so the exposure moved from unusable to usable. Nothing had changed about the client holding a long-lived key.

So I recorded the long-term governance as a separate item of technical debt at the same moment I applied the stopgap. This was not an architecture I had designed, but having chosen to keep it temporarily, I could not let a temporary arrangement quietly become permanent. Stopping the bleeding and governing the risk are two different jobs, and they cannot be booked against each other.

Hiding the key better is not the answer

The first instinct is to hide the key more thoroughly: obfuscation, encryption, unpacking tricks, strings assembled at runtime. These help, but they solve a different problem.

If signing happens on the client, the key has to be on the client. A package can be unpacked, memory can be dumped, strings can be reconstructed. Obfuscation raises the cost of extraction; it does not remove the possibility. Worse, this key was long-lived: taken today, still valid tomorrow; taken by one person, available to everyone.

What worried me was not the storage platform being breached, but what a leaked key would hand over: bulk uploads, using our storage as a personal drive, consuming traffic we pay for — and, if the permission scope were broad enough, possibly affecting existing data.

So the real question was not “is it hidden well enough” but “does this capability belong on the client at all.” It does not. What the client needs is not a long-lived key, but one bounded upload permission.

Keep direct upload, take back signing

There were really two separate choices here.

The first is where files travel. Our lightweight servers had limited bandwidth and traffic allowances. Ordinary requests and responses cost little; images and videos would multiply that cost quickly. Direct upload had to stay — I was not going to pull every file back through the server just to reclaim control.

The second is what entitles the client to upload directly. The real fix was to move signing to the backend: after login, the client requests a business endpoint before uploading; the server computes the policy and HMAC signature with its own key and returns them; the client still uploads directly with that signature. The long-lived key no longer ships with the installation package.

We did not introduce a full STS temporary-credential system. What actually shipped was a backend policy-plus-signature endpoint. What the client receives is not a key that can keep minting permissions, but a permission slip for a target file.

That slip is bounded by four constraints: a fixed storage location, a restricted path prefix, a file-size cap, and a content-type constraint. The type rule proved unstable in testing, so we kept a documented fallback to remove it — a rule that cannot be verified reliably only creates false rejections. On top of those, each issuance targets one file and is isolated per user, and each user may request at most twenty signatures per minute.

These constraints govern write behavior. They should not be oversold as absolute replay prevention, and the content-type constraint is not content moderation. The bucket’s public-read semantics did not change either: this work addressed long-lived credential exposure and write constraints, not turning public content private.

Fig. 1: The boundary is not about where the key physically sits, but about which party holds which capability. The client gets a permission slip, not a key to the whole bucket; the server does not relay files; object storage does not hold business identity.

Every constraint needs a negative test

The easiest mistake in security work is verifying only that the normal path works. We tested the other side:

• No login credential must return 401.
• A forged credential must return 401.
• A real credential must return 200, with all six response fields present.
• Each of the four policy constraints was violated on purpose and rejected in turn.
• Twenty-one consecutive signature requests: the first twenty succeed, the twenty-first must return 429.
• A real upload must return 204, and the returned URL must resolve.

In that matrix, 200 is one row; everything else is “what must not pass must not pass.” Beyond my own checks, an independent review re-ran the code, the documentation, and the shared-environment tests. It sent back one configuration issue, which was fixed before merge.

A second gate: the content itself

Key governance was not the only lesson. WeChat’s review process had already taught me another one.

During remediation I inventoried the user-content entry points — avatars, profile text, submissions, and free text — and connected them in the backend to WeChat’s text and image content-safety interfaces. When the interface is healthy, content must pass before it enters the publishing flow; violations are rejected. It is a front gate, not a replacement for the existing human review.

The interesting part is the failure mode. If the safety interface itself fails, the system degrades to letting content through while recording it in a backend ledger for human re-checking, instead of blocking every upload. A broken third-party check should not take the whole business down with it — but the bypass has to be visible and auditable, not silent.

In hindsight, the safer default would be to decouple submission from public display: when checking is unavailable, allow saving and submission but hold the content in a pending state until the check recovers or a person approves it. That has its own cost in review backlog and delayed display, but it pulls the risk back from the public surface into a review queue. This is retrospective analysis, not what was implemented at the time. What we actually chose was business continuity first, with the ledger and human follow-up compensating for the short-term content risk.

Deleting code matters more than adding it

A new endpoint does not make the old risk disappear. The frontend had its own local signing logic, plus a legacy implementation nobody called anymore that still contained a second plaintext copy of the key. Both were deleted in the same pass: seven files, cleared to zero.

Adding a new path without deleting the old one produces a predictable result: new users go through the signing endpoint while the old code — and the key — stay in the package. The risk surface does not shrink; it just gains an entrance that looks safer. That is why “the old code is completely gone” was weighted as heavily as “the new endpoint works” in the acceptance criteria.

Fig. 2: The work has two phases. Phase one only kept the business alive; phase two answered "who can upload." Completion is marked as recorded: the client key is gone and the old path is deleted, while key rotation still has no execution receipt and least privilege is unconfirmed.

Release order is part of the security

This is the part most easily overlooked, and the part most likely to cause an incident. The order cannot be reversed. Older client versions were still uploading directly with the old key; rotating the key in the console first would have broken uploads for every user who had not upgraded. A functional outage would have arrived before any security benefit.

The order we set was: the backend goes first, with the new signing endpoint live while the old direct-upload path stays alive. Then the new client version is submitted and released. Once coverage rises, most uploads take the new path. Then the key is rotated in the console, with old and new keys coexisting so there is no switchover gap. Finally, the local key file is physically deleted.

Every step can be rolled back, and no step requires every user to upgrade at one particular moment. Security is not a switch; it is an ordered migration.

On the night of the rollout, the backend window took twenty-seven seconds of downtime with zero rollback. The new client version passed review and was released the next day, and the signing endpoint entered an observation period. Observation did not mean “are there any errors.” It watched two signals: in normal operation, 200 responses on the new endpoint rising with version coverage; in abnormal operation, clusters of 500s, or the same user repeatedly hitting 401 or 429.

What is not finished

This account cannot be written as “solved.”

Key rotation is still an open item in my records. Once coverage of the new version is high enough, it is to be performed in the console with old and new keys coexisting, followed by deleting the local file. As of the last record I can check, it is still waiting for a window and there is no execution receipt — so I cannot say the old key is dead. The permission scope of the old key was also not fully confirmed at the time, so I cannot claim that least privilege has been achieved.

A few loose ends remain: test files left in object storage need cleaning up, and the related operations documentation needs updating to the new procedure. Small things, but unfinished is unfinished.

The scale should be stated plainly too: this was a small-team mini program, not an enterprise system under heavy traffic. The value of the change is not in concurrency. It is in taking something that “always worked, so nobody looked at it” and breaking it into steps that can be verified and rolled back.

What I learned

The hardest part of security work is not finding a smarter design. It is admitting that phase one was only a stopgap — a stopgap can turn an exposure from unusable into usable, so stopping there looks like a fix on the balance sheet while making the risk more concrete in reality.

Real governance has to answer three questions: does this capability belong on the client; have the old paths actually been deleted; and could the switchover order hurt users first. None of those three questions lives in the code. They live in judgment.