← Engineering Notes

AWS & Backend Architecture

S3 Presigned Uploads

Accepting an upload and carrying the file are different jobs. On MarketMate the API decides whether an upload is allowed, then the browser sends the bytes straight to S3.

The problem

Profile photos and listing images are not small JSON bodies. If SpringMate accepted the file on the API, every byte would enter the application process, sit in memory or on a temporary buffer, and leave again toward object storage. The service would pay for bandwidth and request time on work that is mostly copying.

That cost scales with file size and with how many people upload at once. The API still has to know who is uploading, what kind of object it is, and which user or listing it belongs to. It does not have to be the pipe for the file.

The architecture

MarketMate uses a presigned PUT. The browser asks SpringMate for permission. SpringMate checks that the caller is authenticated, that the batch is non-empty, and that each file has a name, a content type, and a directory it understands: profile images or listing images. It then creates an object key it owns — a new id plus the original extension, under that directory — and returns a URL signed for a PUT of that key.

The signature is not a general pass to the bucket. It is permission for one PUT, for a configured window. The default in this service is fifteen minutes, and the signed request includes the content type the client must send. When that window ends, the URL should be rejected. A successful signature also does not mean the object is a valid image, or that the application has recorded it yet.

There is no completion callback from S3. After the PUT succeeds, the client makes a normal API call. For a profile photo that call saves the object key on the user. For a new listing it sends the keys with the listing itself, and the API accepts a listing image key only when it sits under the listings prefix. If saving those listing images fails, the service deletes the keys it had already accepted for that request.

Replacing a profile image also deletes the previous object. The database is what says which object is current. S3 is where the bytes live.

Why the file no longer goes through the API

On the common path the application server never reads the file. It authorizes the operation, names the object, and later stores that name. Bandwidth for the upload is between the browser and S3.

I am not treating that as a measured speedup. The repository does not contain timings for this path. The gain is structural: request threads are no longer occupied copying image bytes, and the API process is no longer in the middle of that transfer.

The backend still owns the decisions that make the object mean something. It chooses the key, limits how long the URL works, and only then attaches the key to a user or a listing. The presign request requires a content type, and the browser has to repeat that content type on the PUT, because the signature was made for it. The direct path does not apply the multipart size limit used when a file is posted to the API. A presigned URL is not a substitute for deciding what you are willing to store.

Retry, expiry, and fallback

A failed PUT is not always a reason to give up, and it is not always a reason to send the file through the API. Those are different recoveries.

The browser retries the same PUT up to three more times, with a short increasing delay and a little jitter. It reuses the URL it already has. That matters because the object key was chosen when the URL was signed. Writing that same key again replaces one object. Asking for a new signature would mint a new key and could leave the first object behind if anything had already been stored.

Those retries are for a transfer that might have blipped. They do not refresh an expired signature. If S3 rejects the URL because the window has passed, repeating it will keep failing. The client does not request a new presigned URL in that case.

Profile images have a second path, and it runs only after those retries fail. The client asks whether the object is already in the bucket. If it is, the PUT likely succeeded and the response was lost, so the app keeps that key and saves it on the user. If it is not, the browser posts the file to the API and SpringMate uploads it. The server generates the key on that path, stores it on the profile, and the client reloads the user instead of saving the original key. If the existence check itself fails, the client still attempts that API upload rather than assuming the object is missing or present.

When the file does go through the API, that server-side put is retried. If it still fails, the profile update fails. There is no second bucket.

Listing publish uses the same direct PUT and the same client retries. If that fails, the form reports the error and does not create the listing. The API has a listing-image fallback, but the publish form does not call it. I do not want the note to imply that every failed upload in MarketMate falls back through the backend. Today that fallback is the profile flow.

Retry the same PUT

Same URL, same object key. Fits a dropped connection or a brief S3 error while the signature is still valid. Does not help once the URL has expired.

API fallback

Used for profile images after the direct upload has failed and the object is not already stored. The API carries the bytes. Listing publish does not take this path.

Trade-offs

Upload through the API

  • One request can authorize, store, and record the object
  • Size limits and failure handling sit in one place
  • No expired-URL case for the client to reason about

The cost is that the application copies every byte, and upload traffic competes with the rest of the API.

Presigned PUT, then record the key

  • File bytes skip the application process
  • The API still names the object and attaches it
  • A lost response can be told apart from a missing object, on the profile path
  • The same URL can be retried without creating a second key

The cost is more steps, a signature that expires, and a gap between “S3 accepted the bytes” and “the database points at them.” A listing image that never gets recorded is cleaned up when image processing fails. An object uploaded and then abandoned before that call is not automatically removed.

Direct upload is the better default here because images are the bulky part and the API already has a separate write for the user or the listing. It is a worse default if you need the file and the database row to succeed or fail as one action, or if you are not prepared to handle an expired URL and a half-finished upload.

What I took away

The useful split was not “put files in S3.” It was keeping authorization, object identity, and the database record on the API, and letting the file transfer happen somewhere else. Retries belong on that transfer, against the key you already signed. Falling back through the API is a different decision, and in this system it is a profile recovery path, not the behavior of every upload.