A single request to a US east coast bucket costs about 200 milliseconds of round trip from Sydney before any work happens. That's tolerable for one uncached download and unacceptable for anything doing several.
The pattern that hurts is application code making sequential storage calls, check the object exists, fetch metadata, generate a presigned URL, then fetch. Four round trips to Virginia is most of a second of pure waiting, and it turns a page that should feel instant into one that feels broken.
Three fixes, in order of effectiveness. Put a CDN in front of anything publicly readable, which removes the origin from the hot path entirely for repeat requests. Restructure code to avoid sequential calls, fetch metadata alongside the object, generate presigned URLs without a round trip, batch where the API allows. And if the data genuinely needs to be read from your application server on every request, put the bucket in the same region as the server.
Also measure from Perth, not just Sydney. Perth is further from Sydney than Sydney is from Auckland, and applications tuned on east-coast numbers feel noticeably slower there. If you have Western Australian users, they are the ones who will complain first.