Repository navigation
feat: Amazon S3 and Cloud Storage buckets - #24
Merged
Merged
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
- An aws-chunked chunk size near the integer limit no longer panics the handler; sizes are parsed as bare hex. - Bodies over the limit fail with EntityTooLarge for PutObject and MaxMessageLengthExceeded otherwise, and keys over 1024 bytes with KeyTooLongError. - ListBuckets with any of its parameters pages, 10,000 at a time by default, and reports regions, as the API reference says, and rejects a bucket-region other than the request's. - A denied HeadBucket carries x-amz-bucket-region. - A version-id-marker of null still has a position after that version is deleted. - The SDK tests share their error code helpers through sdktest/internal/sdkerr.
…with NUL - An upload larger than the limit fails with 413 instead of being cut short while its resource was named. - Listing generations pages correctly when one name extends another with a NUL byte. - buckets.patch clears labels, versioning and encryption set to null. - Conflict messages are the ones the JSON API documents.
…gs with control characters DeleteObjects with an empty key made a delete marker no path could reach, so the bucket could not be deleted; ObjectIdentifier keys have a minimum length of 1. A key over 1024 bytes now fails on its own with KeyTooLongError. Tag text follows the documented pattern of letters, separators and numbers, so tabs and newlines fail with InvalidTag. Forgetting a key also forgets its null version's position.
… ACME challenge names cors that is not a list of objects was stored and served back to the client, which then failed to decode the bucket. A body with data after its JSON value was accepted. Object names under .well-known/acme-challenge/ are prohibited. Paths ServeMux would redirect now answer 404 in the API's shape, and a bucket insert peeks at no more than a bucket body.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Bucket stand-ins for both clouds, so code that creates buckets, finds them again by their tags or labels, empties them and deletes them can be tested against partial failure.
Amazon S3 (
aws/s3):CreateBucket,HeadBucket,ListBuckets(prefix,bucket-region,max-bucketspages with each bucket's region) andDeleteBucket.PutBucketTagging,GetBucketTaggingandDeleteBucketTagging, with S3's tag rules andNoSuchTagSetfor an untagged bucket.PutBucketVersioningandGetBucketVersioning;PutBucketPolicyandPutBucketCorsare checked and recorded.PutObject,DeleteObject,DeleteObjectsandListObjectVersions, with "null" versions for unversioned and suspended buckets and delete markers for versioned ones. Its markers are positions, so the next page still follows one whose versions were just deleted.BucketNotEmpty,NoSuchBucket(NotFoundonHeadBucket),BucketAlreadyExists,BucketAlreadyOwnedByYou(and the us-east-1 create that succeeds instead),AccessDenied(ForbiddenonHeadBucket),PermanentRedirectwithx-amz-bucket-regionfor a bucket in another region,IllegalLocationConstraintException,InvalidLocationConstraint,InvalidBucketNameandInvalidTag.PutForeignBucketholds a name for another account. Bodies sent with aws-chunked encoding are decoded.Cloud Storage (
gcp/storage):buckets.insert,get,list(prefix, pages),patch(labels, versioning, CORS, storage class, encryption) anddelete, with metageneration preconditions.objects.insertby multipart or media upload (checksums verified, generation preconditions),objects.list(prefix, offsets,versions, pages) andobjects.delete(live or by generation). With versioning on, old generations become noncurrent and keep the bucket from being deleted.conflictfor a name the caller has (which the client's retry of a lost create gets) and for one someone else has, 404notFound(the client'sErrBucketNotExistandErrObjectNotExist), 403forbiddenfor a foreign bucket, 409 for a bucket that is not empty, and 400 for invalid names and labels.option.WithEndpoint(srv.URL + "/storage/v1/")orSTORAGE_EMULATOR_HOST.Both follow the existing packages:
WithFaults,WithClock,Call.Resourcenaming the bucket or object (the one being created, for a create), accessors (Buckets,LookupBucket,PutBucket,PutObject,ObjectVersions/Objects,PutForeignBucket), and 404 for what they do not implement. Each package doc lists where it differs from the real API, and the README has a summary.A lost response to a HEAD now breaks off inside the status line. A HEAD response is complete at its header, so sending one would have read as a successful answer.
Testing
make checkpasses. New tests insdktestdrive the AWS SDK for Go v2 (service/s3) and the Cloud Storage Go client (cloud.google.com/go/storage) against the stand-ins. They cover create, tag and label reads, listings and pages, emptying versioned and unversioned buckets page by page, deletes and every error code above. They also cover aws-chunked uploads over TLS, emulator-host setup, lost responses, failures after a call took effect, targeted rejections, and seeded random faults that a retried sweep converges through.make bench(3 runs each). The only change to existing request handling is the HEAD case in the shared stub, and allocations are unchanged. Times were within noise on a loaded machine.Checklist
make checkpasses