operation error S3: GetObject, failed to sign request: failed to retrieve credentials: no EC2 IMDS role found, operation error ec2imds: GetMetadata, http response error StatusCode: 404, request to EC2 IMDS failed
. Can anyone help me? thanks
s3manager.Uploader
to stream the body of the incoming request (which is an io.Reader
) to S3 and to download (both single and multipart flavor) we used the primitive io.Copy
to write the response from the ´s3.GetObjectOutput.Body´ (which is an io.ReadCloser
).s3.UploadPartInput
needs a aws.ReadSeekCloser
and to pass the body of the incoming request you need to buffer it in some place (for example, in memory). Is there a way to stream an incoming request of an UploadPart
with no need to store the entire body in memory?
Hi everyone!
With my team we are building a reverse proxy to intercept all outgoing requests to S3 in order to audit and control the access from the different apps.
We've successfully implemented almost all operations by streaming the content of the files. For instance, to upload with single operation we used thes3manager.Uploader
to stream the body of the incoming request (which is anio.Reader
) to S3 and to download (both single and multipart flavor) we used the primitiveio.Copy
to write the response from the ´s3.GetObjectOutput.Body´ (which is anio.ReadCloser
).
The only operation we still couldn't implement via streaming was the upload-part (in the context of a multipart uploading). The problem is thats3.UploadPartInput
needs aaws.ReadSeekCloser
and to pass the body of the incoming request you need to buffer it in some place (for example, in memory). Is there a way to stream an incoming request of anUploadPart
with no need to store the entire body in memory?
just in case, here I asked this question: https://stackoverflow.com/q/67624774/3392786
did anyone try to parse the pricing API data in into a native Go object? I'm having a hard time with this
{"product":{"productFamily":"Storage","attributes":{"storageMedia":"SSD-backed","maxThroughputvolume":"1000 MiB/s","volumeType":"Provisioned IOPS","maxIopsvolume":"64000","servicecode":"AmazonEC2","usagetype":"EBS:VolumeUsage.io2","locationType":"AWS Region","volumeApiName":"io2","location":"US East (N. Virginia)","servicename":"Amazon Elastic Compute Cloud","maxVolumeSize":"16 TiB","operation":""},"sku":"6V576P37PS7K7KYU"},"serviceCode":"AmazonEC2","terms":{"OnDemand":{"6V576P37PS7K7KYU.JRTCKXETXF":{"priceDimensions":{"6V576P37PS7K7KYU.JRTCKXETXF.6YS6EN2CT7":{"unit":"GB-month","endRange":"Inf","description":"$0.125 per GB-month of Provisioned IOPS SSD (io2) provisioned storage - US East (Northern Virginia)","appliesTo":[],"rateCode":"6V576P37PS7K7KYU.JRTCKXETXF.6YS6EN2CT7","beginRange":"0","pricePerUnit":{"USD":"0.1250000000"}}},"sku":"6V576P37PS7K7KYU","effectiveDate":"2021-07-01T00:00:00Z","offerTermCode":"JRTCKXETXF","termAttributes":{}}}},"version":"20210709001917","publicationDate":"2021-07-09T00:19:17Z"}
and json-to-go gives this output
type AutoGenerated struct {
Product struct {
ProductFamily string `json:"productFamily"`
Attributes struct {
StorageMedia string `json:"storageMedia"`
MaxThroughputvolume string `json:"maxThroughputvolume"`
VolumeType string `json:"volumeType"`
MaxIopsvolume string `json:"maxIopsvolume"`
Servicecode string `json:"servicecode"`
Usagetype string `json:"usagetype"`
LocationType string `json:"locationType"`
VolumeAPIName string `json:"volumeApiName"`
Location string `json:"location"`
Servicename string `json:"servicename"`
MaxVolumeSize string `json:"maxVolumeSize"`
Operation string `json:"operation"`
} `json:"attributes"`
Sku string `json:"sku"`
} `json:"product"`
ServiceCode string `json:"serviceCode"`
Terms struct {
OnDemand struct {
SixV576P37PS7K7KYUJRTCKXETXF struct {
PriceDimensions struct {
SixV576P37PS7K7KYUJRTCKXETXF6YS6EN2CT7 struct {
Unit string `json:"unit"`
EndRange string `json:"endRange"`
Description string `json:"description"`
AppliesTo []interface{} `json:"appliesTo"`
RateCode string `json:"rateCode"`
BeginRange string `json:"beginRange"`
PricePerUnit struct {
Usd string `json:"USD"`
} `json:"pricePerUnit"`
} `json:"6V576P37PS7K7KYU.JRTCKXETXF.6YS6EN2CT7"`
} `json:"priceDimensions"`
Sku string `json:"sku"`
EffectiveDate time.Time `json:"effectiveDate"`
OfferTermCode string `json:"offerTermCode"`
TermAttributes struct {
} `json:"termAttributes"`
} `json:"6V576P37PS7K7KYU.JRTCKXETXF"`
} `json:"OnDemand"`
} `json:"terms"`
Version string `json:"version"`
PublicationDate time.Time `json:"publicationDate"`
}
any clues how to handle the OnDemand
and PriceDimensions
structs?
Hi,
I am trying to use AWS SDK for Go to implement S3 apis to connect with third party storage provider (Dell OneFS) who support S3 API. First in place, is AWS SDK is tightly coupled with AWS or it can be used to connect with any storage provider who support S3 API? If so, can someone help me to configuring the target host and other details?
Thanks
JK
NewSession()
call from: github.com/aws/aws-sdk-go/aws/session
NewConfig()
with endpoint, credentials, region, and all other parameters pointing to your storage provider