by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Jan 31 2019 14:07
    jugalde-r7 commented #2352
  • Jan 31 2019 09:36
    takeyourhatoff commented #2427
  • Jan 31 2019 09:35
    takeyourhatoff commented #2427
  • Jan 30 2019 23:10
    diehlaws commented #2427
  • Jan 30 2019 23:09
    diehlaws labeled #2427
  • Jan 30 2019 23:09
    diehlaws labeled #2427
  • Jan 30 2019 21:47
    diehlaws labeled #2352
  • Jan 30 2019 21:47
    diehlaws commented #2352
  • Jan 30 2019 21:12
    diehlaws commented #2342
  • Jan 30 2019 20:58
    diehlaws assigned #2427
  • Jan 30 2019 20:54
    diehlaws unlabeled #81
  • Jan 30 2019 20:54
    diehlaws unlabeled #142
  • Jan 30 2019 20:54
    diehlaws unlabeled #618
  • Jan 30 2019 20:54
    diehlaws unlabeled #81
  • Jan 30 2019 20:54
    diehlaws unlabeled #142
  • Jan 30 2019 20:54
    diehlaws unlabeled #619
  • Jan 30 2019 20:54
    diehlaws unlabeled #628
  • Jan 30 2019 20:54
    diehlaws unlabeled #568
  • Jan 30 2019 20:54
    diehlaws unlabeled #521
  • Jan 30 2019 20:54
    diehlaws unlabeled #487
Simon Woldemichael
@swoldemi
@walters_paris_twitter If the need for replicating across regions is just an operational requirement, then this might be helpful: https://github.com/aws-samples/amazon-ecr-cross-region-replication
If it's an application requirement then using the Docker client SDK for Go would make it easier (assuming the host that your service runs on also runs docker): https://docs.docker.com/engine/api/sdk/examples/
You would just need to authenticate with ECR by calling GetAuthorizationToken
Paris “Pear 🐻” Walters
@walters_paris_twitter
@swoldemi The aws-samples, I started with. I try to create a lambda just with the bash script,however lambda also needed docker package to be added as a layer. I'm not sure how to package up docker as a layer. For uploading images via API, what and where can I get the these values - layerPartBlob": blob, "partFirstByte": number, "partLastByte": number,
Paris “Pear 🐻” Walters
@walters_paris_twitter
Needs some help, I have the image manifest. I am getting an error of LayersNotFoundException: Layers with digests '[sha256:7912335d3aebe27fcfa61affb4ce3e08b8435739e1072906f99f3aa8c3cfae7d]' required .
Any ideas??
Paris “Pear 🐻” Walters
@walters_paris_twitter
how you used Uploadlayerpart? @swoldemi
Simon Woldemichael
@swoldemi
@walters_paris_twitter I'm not sure, I've never done this without using some kind of container engine. I'll DM you though
Mark Richman
@mrichman
How do I create an &dynamodb.AttributeValue{B: ... ? What conversion type do I need for B ? I'm basically trying to infer the "schema" of a table in order to generate some random test input. I see that B maps to []byte but I don't see the equivalent of aws.String() for []byte.
Mark Richman
@mrichman
Do I just use the []byte value directly? e.g. item[pkName] = &dynamodb.AttributeValue{B: getRandomBinary(256)}
Simon Woldemichael
@swoldemi
So maybe item[pkName] = dyanmodbattribute.Marshal(getRandomBinary(256))
Mark Richman
@mrichman
I have not, but in my case I don't think I can use it because there is no compile-time struct — I'm doing this all at runtime based on whatever type the PK/SK are
Simon Woldemichael
@swoldemi
assuming return type of getRandomBinary(256) is []byte
Mark Richman
@mrichman
yep
func getRandomBinary(length int64) []byte {
    token := make([]byte, length)
    rand.Read(token)
    return token
}
func createPutRequest() (*dynamodb.PutRequest, error) {
    var item = make(map[string]*dynamodb.AttributeValue)
    switch pkType {
    case "S":
        item[pkName] = &dynamodb.AttributeValue{S: aws.String(getRandomString(20))}
    case "N":
        item[pkName] = &dynamodb.AttributeValue{N: aws.String(getRandomNumber(1, 100))}
    case "B":
        item[pkName] = &dynamodb.AttributeValue{B: getRandomBinary(256)}
    }

    if skName != "" {
        switch skType {
        case "S":
            item[skName] = &dynamodb.AttributeValue{S: aws.String(getRandomString(20))}
        case "N":
            item[skName] = &dynamodb.AttributeValue{N: aws.String(getRandomNumber(1, 100))}
        case "B":
            item[skName] = &dynamodb.AttributeValue{B: getRandomBinary(256)}
        }
    }

    pr := &dynamodb.PutRequest{Item: item}
    return pr, nil
}
i've only tested on S type so far
I can't think of a good use case for why anyone would want a PK or SK with binary data, but I figured I'd try to support it
Simon Woldemichael
@swoldemi
@mrichman How are you reading the types of the keys from the table?
Mark Richman
@mrichman
func discoverSchema(tableName string) error {
    svc = dynamodb.New(getSession())
    response, err := svc.DescribeTable(&dynamodb.DescribeTableInput{
        TableName: aws.String(tableName),
    })

    if err != nil {
        return err
    }

    // get pk and sk attrib names
    for attrib := range response.Table.KeySchema {
        if *response.Table.KeySchema[attrib].KeyType == "HASH" {
            pkName = *response.Table.KeySchema[attrib].AttributeName
        }
        if *response.Table.KeySchema[attrib].KeyType == "RANGE" {
            skName = *response.Table.KeySchema[attrib].AttributeName
        }
    }

    // get pk and sk attrib types:
    // S - the attribute is of type String
    // N - the attribute is of type Number
    // B - the attribute is of type Binary
    for attrib := range response.Table.AttributeDefinitions {
        if *response.Table.AttributeDefinitions[attrib].AttributeName == pkName {
            pkType = *response.Table.AttributeDefinitions[attrib].AttributeType
        }
        if *response.Table.AttributeDefinitions[attrib].AttributeName == skName {
            skType = *response.Table.AttributeDefinitions[attrib].AttributeType
        }
    }
    return nil
}
Simon Woldemichael
@swoldemi
@mrichman Passing a []byte should be fine, since the SDK will encode it for you to probably format the PutItem request. I get what you mean on not being able to easily use Marshal. What you currently have should work
Mark Richman
@mrichman
Thanks @swoldemi I'll report back if I have any issues with that. In the meantime, I do have one other issue. After my load test app runs for a few minutes, I hit this error: RequestError: send request failed\ncaused by: Post https://dynamodb.us-east-1.amazonaws.com/: dial tcp: lookup dynamodb.us-east-1.amazonaws.com on 172.31.0.2:53: no such host. I'm thinking this is a linux kernel problem (ulimit?) and not the SDK itself. I'm running on a t3.medium instance, if that helps.
Mark Richman
@mrichman
LOL the illustrious Dave Cheney chimed in https://github.com/golang/go/issues/18588#issuecomment-661807513
Simon Woldemichael
@swoldemi
@mrichman Lol nice. Too many open files does look like a ulimit error. Also worth checking if your EC2 instance allows access to the outside internet. Are you using a dynamodb VPC endpoint?
Tharaka Kumburuthenne
@danakum
hi i would like to know how use username password in aws
hi i would like to know how use username password in aws
as authentication in SES
Simon Woldemichael
@swoldemi
@danakum Can you elaborate on what you mean? Are you wanting to use SES to send emails with an SMTP username and password?
Tharaka Kumburuthenne
@danakum
yes @swoldemi
Simon Woldemichael
@swoldemi
@danakum You should be able to use the native smtp package: https://golang.org/pkg/net/smtp/#SendMail
There are a few examples in other languages here: https://docs.aws.amazon.com/ses/latest/DeveloperGuide/send-using-smtp-programmatically.html
For the second parameter of SendMail, you should be able to just use a PlainAuth and pass your username and password there: https://golang.org/pkg/net/smtp/#PlainAuth
Nicolas Sitbon
@nsitbon
Hi I'm using the Upload method (https://docs.aws.amazon.com/sdk-for-go/api/service/s3/s3manager/#Uploader.Upload) to push a file in a S3 bucket. The method returns a https://docs.aws.amazon.com/sdk-for-go/api/service/s3/s3manager/#UploadOutput containing an UploadId. How can I pass it back to the same Upload method to resume the upload of big files when an error occurred? Best regards
Simon Woldemichael
@swoldemi
@nsitbon Upload won't return an UploadId if there's an error uploading a file that is too big for 1 part. You'll just get an error that says something like "upload multipart failed". Best way might be to run the multipart upload yourself and resume the upload when an error occurs on a specific part.
Simon Woldemichael
@swoldemi
@nsitbon Actually I'm wrong, the MultiUploadError returns has an UploadId() too, not just the message. If you set, LeavePartsOnError to true on your Uploader, you can resume from where it failed. I think there are 2 S3 APIs that will be helpful, ListMultiPartUploads and ListParts.
To see where it failed you can call s3.ListMultiPartUploads to get the process, just need to include the UploadId: https://docs.aws.amazon.com/sdk-for-go/api/service/s3/#ListMultipartUploadsInput
It might be better to call s3.ListParts instead to see where to resume from, since the list of Parts that are returned give you the part number and the byte where that part ended, but I'm not sure since I haven't done this before: https://docs.aws.amazon.com/sdk-for-go/api/service/s3/#S3.ListParts
From that info you should be able to figure out where to resume, with a bit of math. It doesn't look like there's a way to do this with just s3manager.Uploader.
Aditya Kumar Gupta
@adityakrgupta25
I have been trying to run WaitUntilTasksStopped after triggering a task on the ECS, but when the tasks fail with non zero exit code, it does not return an error. I have proded around a bit and tried to look at the returned response from aws when the tasks returns. Turns out the failiure field is empty, but i can see is exit code is non zero. What can be done if non-zero exit codes are not being captured as error by WaitUntilTasksStopped by the SDK?
Simon Woldemichael
@swoldemi
@adityakrgupta25 What behavior are you trying to produce other than waiting for the task to stop? Maybe calling DescribeTasks then reading the exit code from there?
puvvadav
@puvvadav
Is there any way to specify client timeout ? For example if COS is not responding client seems to be stuck forever while downloading an object.
Simon Woldemichael
@swoldemi
@puvvadav You can set a timeout on the context you pass to GetObjectWithContext: https://docs.aws.amazon.com/sdk-for-go/api/service/s3/#S3.GetObjectWithContext
You can also provide a custom HTTP client when creating your session, but I don't think that will timeout while reading the response body so you'll need to provide a context: https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/custom-http.html
Simon Woldemichael
@swoldemi
Likewise on DownloadWithContext if you're using s3manager's downloader: https://docs.aws.amazon.com/sdk-for-go/api/service/s3/s3manager/#Downloader.DownloadWithContext
VictorHSB
@victorhsb
hello, good evening! does anyone here know if i can request ona specific taks inside a service on ECS? i know that with ECS's service discovery i can call by "service.cluster" but can i do something like this but specifying which instance i want to call? i'm trying to make multiple instances talk to each other. (talking specifically about the sdk-go, i know i can list the instances of a service and get the instance id's with it)
1 reply
Josh
@jnprogrammer
Hello everyone
Promaethius
@Promaethius
This may be a silly question but I don't have access to an IC C2S environment to test it out. Does the aws-go-sdk automatically resolve services to FIPS compliant endpoints? This issue says no
aws/aws-sdk-go#2219
but I see FIPS resolution for us-gov--1 in endpoints.json
https://github.com/aws/aws-sdk-go/blob/master/models/endpoints/endpoints.json
Without any modification or custom resolver functions, will aws-go-sdk work with a given us-gov-
-1 region out of the box and resolve to available FIPS endpoints?
1 reply
Miles Conn
@MilesConn
Hi I'm fetching personal information about users from cognito. I have a list of usernames but I'm trying to come up with a fast way to query cognito. Seems like ListUsers only allows a single string to filter. Any ideas? Right now getting personal information is super slow. I'm not quite ready to move personal information to my own DB
1 reply
Michael Bonner
@bonner
Hello, anyone if/how to write data to s3 in parquet? possible out of the box with the go sdk?
4 replies
Charlie
@zhouchunli
session name is defined by ourself?
2 replies
Aditya Kumar Gupta
@adityakrgupta25
How to check if auto-scaling is enabled for an ECS service?
4 replies
Aditya Kumar Gupta
@adityakrgupta25
ecs'sWaitUntilServicesStable() takes as input a list of services and returns just the error. If there's an error, how can I know which Services were the ones that did not become stable?
1 reply
Eric Bressler
@ebressler
has anyone seen xmlutil.XMLToStruct taking a lot of memory? It seems to be holding on to a lot and thus putting my process to 1gb or so of used memory. Seems to be using at least half of the allocated amount.
7 replies
this is mostly from running ListObjectsV2 calls
Eric Bressler
@ebressler
hmm, I guess nobody has any great guidance of what to look for here. it seems like I should be closing something out that I am not. I am going to have to narrow down what is holding references I guess.
Michael Lindes
@mlindes
anyone know? - using the aws-sdk-go, is it possible to do an sts-assume-role with environment variables only? ie - not using the ~/.aws/credentials file?
2 replies
Kostas Papageorgiou
@kostaspap

I have a Lambda application that assumes a role in a different AWS Account using STS temporary credentials and reads some data from S3.
The credentials have expiration time of 1 hr.
I am using the creds provider returned from stscreds.NewCredentials . I am setting the ExpiryWindow of the provider to 1 minute.

From time to time (once every day) I will see an "Access Denied" exception when trying to get an object from S3. The call will succeed when my Lambda retries (after few minutes)

Anyone has seen that? Is it possible that this is due to some AWS glitch and I just have retry such an error programmatically? Or is possible that the 1minute ExpiryWindow that I have configured is somehow very low?

Parshuram Patil
@parshuram-patil
what is the import statement for batch service in golang?
2 replies