Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Gina Trapani
    @ginatrapani
    Hello world!
    Paul Ford
    @ftrain
    Hello, Gina.
    Adam Pash
    @adampash
    😄🎉
    Roberio Moreira
    @RoberioMoreira4_twitter
    Hi Gina!
    Harshdeep Singh Hura
    @kinngh
    Hey
    Anyone had luck deploying mercury?
    On Firebase funcitons* ^
    harmavenso
    @harmavenso
    It would be nice to have instructions to run it on a non Amazon server too, or is that a too digital vegan thing to want?
    Adam Pash
    @adampash
    @kinngh I haven't tried it on a Firebase function, but it should work the same as anywhere else. Have you had trouble with that?
    @harmavenso What did you have in mind? Our API repo runs on serverless on AWS, but deploying to an AWS server would look a lot the same — you'd just need to have a working server and deploy pipeline https://github.com/postlight/mercury-parser-api/
    Harshdeep Singh Hura
    @kinngh
    @adampash I am stuck with things. I added the package as a dependency in packages.json , and because I lack the JS knowledge, I am stuck with actually building the index file
    harmavenso
    @harmavenso
    @adampash yeah, i found that out after posting my question. Thanks !
    Adam Pash
    @adampash
    @harmavenso great!
    @kinngh have you deployed anything with dependencies to a firebase functions before?
    Babak Fakhamzadeh
    @MastaBaba
    Pardon the ignorance. I don't grasp the distinction between the mercury parser and the mercury parser API. In my projects, I use CURL calls to something like https://mercury.postlight.com/parser?url=".$site["link"]. What do I need to do to keep this working after the Web Parser API is shut down?
    Babak Fakhamzadeh
    @MastaBaba
    @harmavenso I also would like to install on a non-Amazon server.
    Adam Pash
    @adampash
    @MastaBaba Mercury Parser is the library that powers the Mercury Parser API. You could run Mercury Parser in any JavaScript code directly without ever calling an API. In fact, if you're only using curl now, you could actually use the mercury-parser cli tool as described in the README here https://github.com/postlight/mercury-parser/#usage
    if, on the other hand, you'd still prefer to be able to call an API to get your parsed response, that's what the Mercury Parser API repo is for: https://github.com/postlight/mercury-parser-api/
    If you follow the instructions to deploy that to AWS, you can continue calling the API as you've been doing — just with a new base url.
    Babak Fakhamzadeh
    @MastaBaba
    Thanks @adampash. When I look at a command like "import Mercury from '@postlight/mercury-parser';", as mentioned in the README for the parser, I do not know where I am doing this import. If, for the API, the 'installation' as listed in that README is for deploying to AWS, I'm really lost; I'd think that the first step of deploying to AWS would be something like setting up some kind of environment on AWS, no?
    Haoji Xu
    @HaojiXu
    Hello everyone! Is there any way I can use Mercury parser in browser? I looked around and did not find any documentation about such usage but docs implied I can do this.
    Harshdeep Singh Hura
    @kinngh
    @adampash I don't have any experience running JS / firebase functions tbh
    Adam Pash
    @adampash

    @MastaBaba The import would happen in your javascript code. If you're not writing javascript/node, that's not something you'd need/want.

    Re the API: No, the only step to deploying to AWS using that repo is making sure your AWS credentials are set up as described in the README. Other than that, it uses the Serverless framework to provision and deploy your code to AWS. It doesn't run in a server — it runs on AWS Lambda, with API Gateway providing you with a URL to make requests.

    @HaojiXu Yes, you can run Mercury in the browser. Usage is the same, though one caveat is: If you're trying to parse URLs outside of your domain, you could run into CORS issues, depending on the policy of the site you're trying to parse.
    @kinngh I think your best bet, then, would be to look into deploying the Mercury Parser API https://github.com/postlight/mercury-parser-api/
    Babak Fakhamzadeh
    @MastaBaba
    Thanks again, @adampash. I'm still not completely clear on the import; I run my javascript in scripts included in PHP pages. I don't use yarn or npm.
    On the serverless framework; In a nutshell, if I understand this correctly, I first have to use npm to install serverless framework, then use git to clone the repo, yarn to install, serve and deploy it. While at some point I have to put my AWS credentials in the code. Yeah?
    Adam Pash
    @adampash
    @MastaBaba Mercury isn't hosted on a CDN, and more likely than not, you want to run Mercury in a Node environment, not a browser environment. you don't have to put your AWS credentials in the code. all you need to do is set up your aws credentials using the aws cli https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html#cli-quick-configuration
    Bryan Hackett
    @BryanHackett_twitter
    Do we need to run nodejs8.10 locally for the API? I encountered this error attempting to deploy using v11.9.0.
     internal/modules/cjs/loader.js:611
        throw err;
        ^
    
    Error: Cannot find module '..'
    Bryan Hackett
    @BryanHackett_twitter
    @adampash Could you post an example IAM json policy to use with this? Thanks!
    Bryan Hackett
    @BryanHackett_twitter

    I got this to finally work:

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "VisualEditor0",
                "Effect": "Allow",
                "Action": [
                    "iam:GetPolicyVersion",
                    "logs:*",
                    "sns:Unsubscribe",
                    "dynamodb:*",
                    "iam:CreateRole",
                    "cloudformation:DescribeStackResource",
                    "xray:PutTraceSegments",
                    "iot:DescribeEndpoint",
                    "kinesis:ListStreams",
                    "cognito-sync:SetCognitoEvents",
                    "cloudformation:DescribeStackEvents",
                    "iam:ListAttachedRolePolicies",
                    "cloudformation:UpdateStack",
                    "sns:Subscribe",
                    "cloudformation:DescribeChangeSet",
                    "iam:ListRolePolicies",
                    "events:*",
                    "cloudformation:ListStackResources",
                    "iam:GetRole",
                    "sns:ListSubscriptionsByTopic",
                    "iam:GetPolicy",
                    "iot:GetTopicRule",
                    "sqs:SendMessage",
                    "cloudformation:DescribeStacks",
                    "iot:CreateTopicRule",
                    "kinesis:PutRecord",
                    "cloudwatch:*",
                    "iot:ListPolicies",
                    "ec2:DescribeSubnets",
                    "iot:ListThings",
                    "iam:GetRolePolicy",
                    "cloudformation:ValidateTemplate",
                    "iot:ReplaceTopicRule",
                    "tag:GetResources",
                    "xray:PutTelemetryRecords",
                    "iot:AttachThingPrincipal",
                    "cognito-identity:ListIdentityPools",
                    "sns:ListTopics",
                    "iot:CreatePolicy",
                    "iam:CreateUser",
                    "iam:PassRole",
                    "sns:Publish",
                    "cognito-sync:GetCognitoEvents",
                    "iot:CreateKeysAndCertificate",
                    "cloudformation:ListStacks",
                    "sqs:ListQueues",
                    "iot:ListTopicRules",
                    "iot:CreateThing",
                    "s3:*",
                    "iot:AttachPrincipalPolicy",
                    "iam:ListRoles",
                    "kinesis:DescribeStream",
                    "sns:ListSubscriptions",
                    "ec2:DescribeSecurityGroups",
                    "cloudformation:CreateStack",
                    "ec2:DescribeVpcs",
                    "kms:ListAliases",
                    "lambda:*"
                ],
                "Resource": "*"
            },
            {
                "Sid": "VisualEditor1",
                "Effect": "Allow",
                "Action": "apigateway:*",
                "Resource": "arn:aws:apigateway:*::*"
            }
        ]
    }

    But it feels like there are probably more permissions than needed.

    Adam Pash
    @adampash
    :point_up: February 13, 2019 8:47 PM I'm not completely sure about this, but could you please try with 8.10 and if you still experience this error, let me know?
    @BryanHackett_twitter it's true that one drawback to the serverless framework is that it requires a lot of access (a lot of ppl end up just using admin privileges, which isn't exactly ideal). I think this is a decent post on the ins and outs of IAM requirements for serverless
    Bryan Hackett
    @BryanHackett_twitter
    @adampash Thanks. Everything worked when I manually set the mercury parser dependency to 2.0.0.
    Adam Pash
    @adampash
    :thumbsup:
    atoxx
    @atoxx
    Hi all, i have a Problem, i am a Mac/iOS Developer that relies on the Mercury Parse API for one of my Apps, now i am looking for a way to install the Code on my own Server, but its not a Root or AWS, its just a hosting PHP Server and i have no clue how to do this.
    Is there any tutorial or way to install the API without NodeJS or YARN? (just by uploading all Files and call a Script with parameters like http://www.myserver.com?parse=whater url)
    Adam Pash
    @adampash
    @atoxx Unfortunately if you want to use our out-of-the-box solution (mercury-parser-api), you'll need to use AWS for that. if that's not an option, you'll need to either use yarn/npm to install the cli or have some sort of javascript code to run the parser.
    atoxx
    @atoxx
    Thx for clarifications, sorry it seems i am out of luck on this, maybe there might be a mirrror APi endpoint or an alternative Install Tutorial one day 11
    Harshdeep Singh Hura
    @kinngh
    So figured out how to deploy the mercury-parser-api on Lambda, and got it working, but here's the thing: I have function invocations, about ~300 of them while I have made only 2-3 requests? Is it because of the pinging every 5 minutes to avoid cold start?
    Adam Pash
    @adampash

    @kinngh That's correct. If you don't want to keep your function running, you can remove the scheduled events by commenting out these lines:

    https://github.com/postlight/mercury-parser-api/blob/master/serverless.yml#L43-L45

    And these:

    https://github.com/postlight/mercury-parser-api/blob/master/serverless.yml#L57-L59

    Tom Lai
    @tomlai19852004
    Hi. Thank you for this open source project. It seems that the import statement in code sample doesn't work for typescript. It worked after I changed the import statement to the following. Thanks!
    import * as Mercury from '@postlight/mercury-parser';
    wwmc
    @underTheFunSun_twitter
    Hi, I'm a bit of a newbie developer and I have a question. I was previously using the Mercury API inside a Chrome Extension. Now that it's gone would my only option be to include Mercury in extension? Meaning I would have a folder, mercuryFiles, with all the files from github and just import it inside the main file with import Mercury from "./mercuryFiles/src/mercury.js" and then I can do something likeMercury.parse(url).then(result => console.log(result))?
    Adam Pash
    @adampash
    @underTheFunSun_twitter You have two options:
    1. You could include Mercury directly in your extension. I don't know what your extensions build steps look like, but if you can use npm modules, you can install it and import/require it into your projects the same way you would any other dependency.
    2. Alternately, you could deploy your own Mercury Parser API drop-in replacement. This process is detailed here: https://github.com/postlight/mercury-parser-api
    Stephen Bradley
    @std_ptr_null_twitter

    Hi there!

    I found the information on the GitHub page to be a little incomplete. Although I'm a full time IT professional and part-time developer, it was not obvious to me how to proceed with the Mercury installation on Amazon Lambda.

    Making the installation work requires some software that is not installed on any OS by default.

    So, I made a note with the instructions that I used to create a working API on Amazon. Feel free to share and to get in touch if you have questions or need assistance.

    There are two links, so you can choose your poison.
    Evernote

    https://www.evernote.com/l/AAPoJR49OThHu5lBZLt8b1fyKomly8gRz6Q
    Notion

    https://www.notion.so/Installing-the-Mercury-Reader-API-on-AWS-Lambda-9b994989be1d49959a894b65def1095d

    wwmc
    @underTheFunSun_twitter
    Just a small update: I manage to get it working by doing npm install @postlight/mercury-parser. Then inside one of the folders in @postlight/mercury-parser is a Mercury.web.js file. Add a line at the end: export default Mercury. Afterwards, I could just import it regularly into my extension and use it without any issues.
    wwmc
    @underTheFunSun_twitter
    Though, one question: What's the best way to get around Cross-Origin Read Blocking with Mercury Web Parser on the browser?
    David Hirmes
    @hirmes

    I'm using the mercury-parser in node, and it's working beautifully:

    const Mercury = require('@postlight/mercury-parser');
    const url = 'https://en.wikipedia.org/wiki/John_von_Neumann';
    Mercury.parse(url).then(result => { console.log(result); } );

    but it is unclear to me how to utilize the custom extractors. There is one for wikipedia here but I don't see instructions for how to incorporate that into the above code?

    Toufic Mouallem
    @toufic-m
    Hi @hirmes , while Mercury Parser can extract content from almost any website, some websites might require special handling in order to find the content more quickly and more accurately than it might otherwise do, and that's where custom extractors come into play. This Custom Parsers README file explains how custom extractors work and details the process of creating one.
    Essentially, they're internal to the Mercury Parser package, and can't be directly referenced from apps using the package.