Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Code Ninja
    @nikolai-git-house
    Hello @ianwow I am new to MIE. I am a frontend developer and want to implement MIE service to my React web app via Rest API. The thing is I am not familiar with AWS authentication so I can't find way how to set authentication for using /create api which upload video to MIE?
    3 replies
    Ian Downard
    @ianwow
    Luther Hill
    @lutherkhill
    Hi there. Checking to see if anyone is working on a video, photo, and audio asset management tooling using the framework.
    3 replies
    Ian Downard
    @ianwow
    The Media Insights (MI) demo app has been updated to work with MIE v2.0.3. Here is the release: https://github.com/awslabs/aws-media-insights/releases/tag/v2.0.3
    • Includes a new feature for controlling video playback speed
    • Also includes more robust support for deploying multiple MI instances per region
    Luther Hill
    @lutherkhill
    Does the app have the ability to get / display / search by exif meta data from the pictures or videos. I am working on adding an operator for it.
    1 reply
    Luther Hill
    @lutherkhill
    Is anyone working on something similar for pictures
    1 reply
    Ian Downard
    @ianwow
    Ian Downard
    @ianwow
    The Media Insights (MI) demo app was also updated today so it uses MIE v2.0.4: https://github.com/awslabs/aws-media-insights/releases/tag/2.0.4
    Chinmay337
    @Chinmay337
    Hello AWS Folks! We are working on a customization implementation on top of the Media Insights Repo (https://github.com/awslabs/aws-media-insights)
    We ran into an issue when trying to upload .MXF format video files. The upload is Successful when we 'Start Workflow' but the uploaded video file does not show up under 'Collections'. We tested the upload functionality for .MP4 and .MOV files and it worked fine and showed up under 'Collections'.
    We had some questions- is there a bucket policy or a Lambda policy that prevents .MXF files from proceeding in the flow? Any help would be highly appreciated, thank you!
    2 replies
    Chinmay337
    @Chinmay337
    Good Afternoon! We had some questions regarding the implementation and an issue we were facing. After troubleshooting through Developer tools when trying to upload files greater than 1 GB, we were seeing a CORS 'No-Access-Control-Origin-Allow' header error in the console. We checked the API Gateway function for 'workflowAPI' and figured that the API gateway resource did not have CORS permissions to invoke the parallel upload feature for larger files.
    After enabling CORS for the resources , we were seeing a 304 error but this potentially caused some issues that we were hoping we could get some assistance on! After enabling CORS, we were seeing Errors for files uploaded in the collection part of the solution. All of the uploaded files had Status of Error and after checking the Step Function logs it looks like the flow was failing at Thumbnail generation. I am assuming this was happening due to headers being overwritten when we tried to enable CORS for the API Gateway resources, and I was trying to find a way to revert the CORS changes or if there was a solution to this problem. Any help is greatly appreciated, thank you so much!
    2 replies
    Ian Downard
    @ianwow
    MIE 2.0.5 was released today. This is the first version of MIE to be released as an official AWS solution. Yay! https://github.com/awslabs/aws-media-insights-engine/releases/tag/v2.0.5
    alokmandloi
    @alokmandloi
    i was able to create the stack but on the README i do not see any specific instructions as to how to find the UI as shown in the screenshot in README
    https://github.com/awslabs/aws-media-insights
    1 reply
    alokmandloi
    @alokmandloi
    @ianwow tagging you since you have commented on the releases for this
    alokmandloi
    @alokmandloi
    3 replies
    Zoltan Arvai
    @zoltanarvai_twitter

    Hi all! MIE looks like an absolutely awesome piece of product / solution. We are considering it to be a central part in our application. We are currently evaluating it and an interesting question came up:

    • Can we implement custom operators outside of the MIE codebase with another language like node?

    I'm digging through the code base / the CFN templates and while it seems kinda doable feels very hacky. I'm wondering what is your position on that?

    5 replies
    Russell Claxton
    @russellc92

    Hi,
    Some of cloudformation templates from the MI (not MIE) repo contain git change/merge syntax with in them, for example:

    ...
    ...
    Mappings:
    <<<<<<< HEAD:cloudformation/aws-content-analysis-use-existing-mie-stack.yaml
    =======
      MediaInsightsEngine:
        Release:
          Version: "v2.0.3"
    >>>>>>> ef8e22eb9e28d8057d207ce692c76279ca15d5b2:cloudformation/aws-content-analysis-create-new-mie-stack.yaml
      ContentAnalysisApp:
        SourceCode:
    ...
    ...

    in this template:
    https://github.com/awslabs/aws-media-insights/blob/2.0.4/cloudformation/aws-content-analysis-use-existing-mie-stack.yaml,

    This exists in the releases as well (2.0.4). Is this intended ?

    7 replies
    Jack Liddiard
    @deadcore
    Hey chaps, first up fantastic product :+1:
    I have a quick question - is there a way to start an execution and pass through some arbitrary (in our case K/V) metadata at the start of the execution.
    An example use case of this would be passing through the human readable 'Name' of the media object, so that the Kineses consumer it is able to resolve that value.
    I hope that makes sense?
    5 replies
    Ian Downard
    @ianwow
    The Media Insights (MI) demo app was updated today, featuring misc bug fixes, doc improvements, and a return of dropzone.js for drag-and-drop multi-file uploads. https://github.com/awslabs/aws-media-insights/releases/tag/v2.0.5
    Hiren
    @happyhirendra_twitter
    Hi,
    I am trying to setup aws-media-insights and getting following error for CompleteVideoWorkflow and CompleteImageWorkflow steps:
    Exception during processing: 'An error occurred (AccessDeniedException) when calling the CreateStateMachine operation: User: arn:aws:sts::<accountid>:assumed-role/WorkflowCustomResourceRole/M3-MediaInsightsWorkflowApi-WorkflowCustomResource-JQ8PMRqATfB0 is not authorized to perform: iam:PassRole on resource: arn:aws:iam::<accountid>:role/CustomerManaged/M3-StepFunctionRole-QC7QPYKFEZLE
    Any idea, what could be the issue?
    35 replies
    Hiren
    @happyhirendra_twitter
    One question - in which version of aws-media-insights .. custom dictionary feature is available ? I have installed v 2.0.3 however not seeing option to define custom dictionary
    3 replies
    Ian Downard
    @ianwow
    MIE 3.0.0 was released today, which features the ability to trigger workflows from S3 events, so, for example, workflows can automatically process files when they are copied to an S3 bucket. https://github.com/awslabs/aws-media-insights-engine/releases/tag/v3.0.0
    Jack Liddiard
    @deadcore

    Hey chaps, another quick question for you.

    Have you ever run into an error where you exceed the max size of the Step Functions IAM Role's policy? I've deployed MIE 2.0.5 with 3 custom operators who's policies are pretty much identical looking something like this:

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Action": "lambda:InvokeFunction",
                "Resource": [
                    "arn:aws:lambda:...:...:function:jackliddiardMediaInsightsEngine-..."
                ]
            }
        ]
    }

    When Cloud Formation comes to adding the custom operator it returns this error

    Exception An error occurred (LimitExceeded) when calling the PutRolePolicy operation: Maximum policy size of 10240 bytes exceeded for role jackliddiardMediaInsightsEngine-StepFunctionRole-PG...

    I did see a brief mention on the topic here: awslabs/aws-media-insights-engine#291
    I was wondering if you have ever worked around it in the past?

    Jack Liddiard
    @deadcore
    Screenshot 2021-06-28 at 11.23.34.png
    I did also come across another peculiar thing which is built in operators are being added to the inline policies even tho the code hint's they should not be:
    # TODO: Once IAM supports the ability to use tag-based policies for
    #  InvokeFunction, put that in the StepFunctionRole definition in
    #  media-insights-stack.yaml and remove the following code block. Inline
    #  policies have length limitations which will prevent users from adding
    #  more than about 35 new operators via the MIE workflow api. Tag based
    #  policies will not have any such limitation.
    
    # Skip the inline policy creation for operators packaged with MIE.
    # The inline policy is not needed for those operators because the
    # StepFunctionRole has already been defined with permission to invoke
    # those Lambdas (see media-insigts-stack.yaml).
    if not ("OperatorLibrary" in operation["StartLambdaArn"] or "start-wait-operation" in operation["StartLambdaArn"]):
        # If true then this is a new user-defined operator which needs to be added
        # to the StepFunctionRole.
        #
        # Create an IAM policy to allow InvokeFunction on the StartLambdaArn
    aburkleaux-amazon
    @aburkleaux-amazon
    @deadcore - this looks like a problem in the code you referenced above. In the case where the MIE stack is deployed from an application stack, the operator library lambdas names do not contain the string "OperatorLibrary" so that code block is being executed unintentionally. Are you deploying the MIE stack separately or as a nested stack from another stack?
    aburkleaux-amazon
    @aburkleaux-amazon
    I'll add an issue to track this problem. I think you can work around it by deploying your application and MIE stacks separately. There are instructions on how to do this with the content analysis application here: https://github.com/awslabs/aws-content-analysis/tree/Release/v3.0.0#option-1-install-demo-app-over-an-existing-mie-stack. Basically, you deploy the MIE back end, then deploy the application and pass in the stack name for the MIE back end the application should use.
    Jack Liddiard
    @deadcore
    @aburkleaux-amazon - Yep we're deploying it as part of a nested stack :)
    aburkleaux-amazon
    @aburkleaux-amazon
    Thanks for confirming. One more question, you mentioned that you are adding your operators via the custom resource. Did you add them to the OperatorLibrary stack?
    Jack Liddiard
    @deadcore
    We're adding them through custom resources in CDK and the provided custom resource Lambda in MIE. Something in the spirit of this:
    const resource = new cdk.CustomResource(this, `${operatorName}Operation`, {
      serviceToken,
      properties: {
        ResourceType: "Operation",
        Name: operatorName,
        Type: type,
        Configuration: configuration,
        StartLambdaArn: this.startLambda.handler.functionArn,
      },
    });
    
    const cfnResource = resource.node.defaultChild as cdk.CfnResource;
    cfnResource.overrideLogicalId(`${operatorName}Operation`);
    Jack Liddiard
    @deadcore
    Screenshot 2021-06-29 at 08.40.38.png
    So during some further digging yesterday we discovered yesterday that the names are being truncated by what I assume is CDK - and removing the key word OperatorLibrary
    Screenshot 2021-06-29 at 08.40.16.png

    Shorting the prefix to something like j1 was one work around but this does go against the spirit of our project - each developer has their own stack instead of sharing the same MIE insights.

    Are you aware of any other possible work arounds/fixes/hacks? Or are we looking at bitting the bullet and sharing a MIE stack?

    p.s. I really do appreciate your support @aburkleaux-amazon :)

    aburkleaux-amazon
    @aburkleaux-amazon
    One workaround would be to disable that block of code that dynamically adds lambda policies and put in a more permissive pattern-based policy for the role that is being built up. There is already a pattern-based policy for the OperatorLibrary lambdas, but it is going to hit a variation of this same bug because we're relying on that string always being embedded in the name of the lambda function. Here's where you would need to add the policy: https://github.com/awslabs/aws-media-insights-engine/blob/62f2fb78aa24e68740ef6cda47cf65e7070e62bf/deployment/media-insights-stack.yaml#L490
    BTW - I entered a new issue for this. Feel free to update it if you discover more info. awslabs/aws-media-insights-engine#522
    Ian Downard
    @ianwow
    MIE 3.0.1 was released on Github today. It includes minor documentation and security improvements. Most notably, the Implementation Guide now has up-to-date API docs. https://github.com/awslabs/aws-media-insights-engine/releases/tag/v3.0.1
    Ian Downard
    @ianwow
    We’re looking for someone outside the core MIE development team to validate the manual build instructions documented in the MIE readme file. Is anyone here able to help us with that?
    7 replies
    Ian Downard
    @ianwow
    MIE 3.0.2 was released on Github today. It includes minor bug fixes and improved test coverage for Amazon Translate operations. https://github.com/awslabs/aws-media-insights-engine/releases/tag/v3.0.2
    Zoltan Arvai
    @zoltanarvai_twitter
    Hi @ianwow! We have developed a pretty cool solution on top of Media Insights for a big client and we'll be processing over 100k videos from their media library. We are planning a spike to see how many videos we can process concurrently. We are aware of some AWS default quotas / limitations but we are wondering if there is anything specific you can think of that could be a bottleneck or some non-obvious AWS quotas that could block us. (let's say kicking off 10k jobs). (yes we are aware it'll be very costly :D)
    9 replies
    Ian Downard
    @ianwow

    I've released version 2.0.0 of Content Analysis on Github: https://github.com/awslabs/aws-content-analysis/releases/tag/v2.0.0

    This release includes big back-end changes but very few UX/UI changes. Most notably, MIE has been decoupled from Content Analysis, thus greatly improving code maintainability.

    Ian Downard
    @ianwow
    Version 2.0.0 of the AWS Content Analysis solution has been published on the AWS Solutions catalog. Release notes here: https://github.com/awslabs/aws-content-analysis/releases/tag/v2.0.0 . Notable user interface changes include:
    • Allow users to control playback speed in the video player.
    • New in-app dialog window shows users how to run search queries via the command line.
    Ian Downard
    @ianwow
    MIE 3.0.3 was released on Github today. It includes improvements in control plane scalability, build automation, and documentation. https://github.com/awslabs/aws-media-insights-engine/releases/tag/v3.0.3
    6 replies
    Ian Downard
    @ianwow
    MIE 3.0.4 was released today. This release includes relatively minor improvements to CloudFormation organization and IAM policy scope. See: https://docs.aws.amazon.com/solutions/latest/aws-media-insights-engine and https://github.com/aws-solutions/aws-media-insights-engine/releases/tag/v3.0.4
    Jason Tadros
    @winston1000
    @ianwow would you mind sharing the command to quickly sync the build to an existing web? It seems way more complicated than it should be. At the moment I am creating new webstacks each time but only keeping the mie. Is that right?
    1 reply
    Really just trying to shorten the cycle for making changes.. I tried sync but it failed.
    It sync'd but then I couldn't log in
    Jason Tadros
    @winston1000
    @ianwow ok I figured it out I think, I'm having trouble sending metadata as a dict though. Do you have an example post payload for metadata?
    8 replies
    João Guimarães
    @jccguimaraes
    Hi! I have been trying to get the step function arn for the workflow on MIE but no success. Can any one tell me if that's possible and how? Thanks
    1 reply