A place to discuss using Mercury Web Parser, an open source utility that extracts content from the chaos of the web. https://github.com/postlight/mercury-parser https://github.com/postlight/mercury-parser-api
@kinngh have you deployed anything with dependencies to a firebase functions before?
Pardon the ignorance. I don't grasp the distinction between the mercury parser and the mercury parser API. In my projects, I use CURL calls to something like https://mercury.postlight.com/parser?url=".$site["link"]. What do I need to do to keep this working after the Web Parser API is shut down?
@MastaBaba Mercury Parser is the library that powers the Mercury Parser API. You could run Mercury Parser in any JavaScript code directly without ever calling an API. In fact, if you're only using curl now, you could actually use the
mercury-parser cli tool as described in the README here https://github.com/postlight/mercury-parser/#usage
if, on the other hand, you'd still prefer to be able to call an API to get your parsed response, that's what the Mercury Parser API repo is for: https://github.com/postlight/mercury-parser-api/
If you follow the instructions to deploy that to AWS, you can continue calling the API as you've been doing — just with a new base url.
Thanks @adampash. When I look at a command like "import Mercury from '@postlight/mercury-parser';", as mentioned in the README for the parser, I do not know where I am doing this import. If, for the API, the 'installation' as listed in that README is for deploying to AWS, I'm really lost; I'd think that the first step of deploying to AWS would be something like setting up some kind of environment on AWS, no?
@MastaBaba The import would happen in your javascript code. If you're not writing javascript/node, that's not something you'd need/want.
Re the API: No, the only step to deploying to AWS using that repo is making sure your AWS credentials are set up as described in the README. Other than that, it uses the Serverless framework to provision and deploy your code to AWS. It doesn't run in a server — it runs on AWS Lambda, with API Gateway providing you with a URL to make requests.
@HaojiXu Yes, you can run Mercury in the browser. Usage is the same, though one caveat is: If you're trying to parse URLs outside of your domain, you could run into CORS issues, depending on the policy of the site you're trying to parse.
@kinngh I think your best bet, then, would be to look into deploying the Mercury Parser API https://github.com/postlight/mercury-parser-api/
Thanks again, @adampash. I'm still not completely clear on the import; I run my javascript in scripts included in PHP pages. I don't use yarn or npm.
On the serverless framework; In a nutshell, if I understand this correctly, I first have to use npm to install serverless framework, then use git to clone the repo, yarn to install, serve and deploy it. While at some point I have to put my AWS credentials in the code. Yeah?
On the serverless framework; In a nutshell, if I understand this correctly, I first have to use npm to install serverless framework, then use git to clone the repo, yarn to install, serve and deploy it. While at some point I have to put my AWS credentials in the code. Yeah?
@MastaBaba Mercury isn't hosted on a CDN, and more likely than not, you want to run Mercury in a Node environment, not a browser environment. you don't have to put your AWS credentials in the code. all you need to do is set up your aws credentials using the aws cli https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html#cli-quick-configuration
I got this to finally work:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"iam:GetPolicyVersion",
"logs:*",
"sns:Unsubscribe",
"dynamodb:*",
"iam:CreateRole",
"cloudformation:DescribeStackResource",
"xray:PutTraceSegments",
"iot:DescribeEndpoint",
"kinesis:ListStreams",
"cognito-sync:SetCognitoEvents",
"cloudformation:DescribeStackEvents",
"iam:ListAttachedRolePolicies",
"cloudformation:UpdateStack",
"sns:Subscribe",
"cloudformation:DescribeChangeSet",
"iam:ListRolePolicies",
"events:*",
"cloudformation:ListStackResources",
"iam:GetRole",
"sns:ListSubscriptionsByTopic",
"iam:GetPolicy",
"iot:GetTopicRule",
"sqs:SendMessage",
"cloudformation:DescribeStacks",
"iot:CreateTopicRule",
"kinesis:PutRecord",
"cloudwatch:*",
"iot:ListPolicies",
"ec2:DescribeSubnets",
"iot:ListThings",
"iam:GetRolePolicy",
"cloudformation:ValidateTemplate",
"iot:ReplaceTopicRule",
"tag:GetResources",
"xray:PutTelemetryRecords",
"iot:AttachThingPrincipal",
"cognito-identity:ListIdentityPools",
"sns:ListTopics",
"iot:CreatePolicy",
"iam:CreateUser",
"iam:PassRole",
"sns:Publish",
"cognito-sync:GetCognitoEvents",
"iot:CreateKeysAndCertificate",
"cloudformation:ListStacks",
"sqs:ListQueues",
"iot:ListTopicRules",
"iot:CreateThing",
"s3:*",
"iot:AttachPrincipalPolicy",
"iam:ListRoles",
"kinesis:DescribeStream",
"sns:ListSubscriptions",
"ec2:DescribeSecurityGroups",
"cloudformation:CreateStack",
"ec2:DescribeVpcs",
"kms:ListAliases",
"lambda:*"
],
"Resource": "*"
},
{
"Sid": "VisualEditor1",
"Effect": "Allow",
"Action": "apigateway:*",
"Resource": "arn:aws:apigateway:*::*"
}
]
}But it feels like there are probably more permissions than needed.
:point_up: February 13, 2019 8:47 PM I'm not completely sure about this, but could you please try with 8.10 and if you still experience this error, let me know?
@BryanHackett_twitter it's true that one drawback to the serverless framework is that it requires a lot of access (a lot of ppl end up just using admin privileges, which isn't exactly ideal). I think this is a decent post on the ins and outs of IAM requirements for serverless
Hi all, i have a Problem, i am a Mac/iOS Developer that relies on the Mercury Parse API for one of my Apps, now i am looking for a way to install the Code on my own Server, but its not a Root or AWS, its just a hosting PHP Server and i have no clue how to do this.
Is there any tutorial or way to install the API without NodeJS or YARN? (just by uploading all Files and call a Script with parameters like http://www.myserver.com?parse=whater url)
Is there any tutorial or way to install the API without NodeJS or YARN? (just by uploading all Files and call a Script with parameters like http://www.myserver.com?parse=whater url)
@atoxx Unfortunately if you want to use our out-of-the-box solution (mercury-parser-api), you'll need to use AWS for that. if that's not an option, you'll need to either use yarn/npm to install the cli or have some sort of javascript code to run the parser.
@kinngh That's correct. If you don't want to keep your function running, you can remove the scheduled events by commenting out these lines:
https://github.com/postlight/mercury-parser-api/blob/master/serverless.yml#L43-L45
And these:
https://github.com/postlight/mercury-parser-api/blob/master/serverless.yml#L57-L59
Hi, I'm a bit of a newbie developer and I have a question. I was previously using the Mercury API inside a Chrome Extension. Now that it's gone would my only option be to include Mercury in extension? Meaning I would have a folder, mercuryFiles, with all the files from github and just import it inside the main file with
import Mercury from "./mercuryFiles/src/mercury.js" and then I can do something likeMercury.parse(url).then(result => console.log(result))?
@underTheFunSun_twitter You have two options:
- You could include Mercury directly in your extension. I don't know what your extensions build steps look like, but if you can use npm modules, you can install it and import/require it into your projects the same way you would any other dependency.
- Alternately, you could deploy your own Mercury Parser API drop-in replacement. This process is detailed here: https://github.com/postlight/mercury-parser-api
Hi there!
I found the information on the GitHub page to be a little incomplete. Although I'm a full time IT professional and part-time developer, it was not obvious to me how to proceed with the Mercury installation on Amazon Lambda.
Making the installation work requires some software that is not installed on any OS by default.
So, I made a note with the instructions that I used to create a working API on Amazon. Feel free to share and to get in touch if you have questions or need assistance.
There are two links, so you can choose your poison.
Evernote
https://www.evernote.com/l/AAPoJR49OThHu5lBZLt8b1fyKomly8gRz6Q
Notion
Just a small update: I manage to get it working by doing
npm install @postlight/mercury-parser. Then inside one of the folders in @postlight/mercury-parser is a Mercury.web.js file. Add a line at the end: export default Mercury. Afterwards, I could just import it regularly into my extension and use it without any issues.
I'm using the mercury-parser in node, and it's working beautifully:
const Mercury = require('@postlight/mercury-parser');
const url = 'https://en.wikipedia.org/wiki/John_von_Neumann';
Mercury.parse(url).then(result => { console.log(result); } );but it is unclear to me how to utilize the custom extractors. There is one for wikipedia here but I don't see instructions for how to incorporate that into the above code?
Hi @hirmes , while Mercury Parser can extract content from almost any website, some websites might require special handling in order to find the content more quickly and more accurately than it might otherwise do, and that's where custom extractors come into play. This Custom Parsers README file explains how custom extractors work and details the process of creating one.
Essentially, they're internal to the Mercury Parser package, and can't be directly referenced from apps using the package.
Essentially, they're internal to the Mercury Parser package, and can't be directly referenced from apps using the package.
@toufic-m Thanks for your response. Are you saying that the custom parsers listed here are automatically run when the domains they are associated with are encountered in the url given to
Mercury.parse()? If so, that doesn't seem to be the case for me. In the example I posted above, the wikipedia.org parser doesn't seem to be executed, just the default Mercury parser. Is there some way to ensure that a custom parser is run that I'm missing?
@hirmes Yes, a custom extractor is automatically used in order to extract content if its domain name matches that of the current request. Could you please submit an issue describing what you're experiencing, while including the current result and what is incorrect about it?
I'm trying to install the mercury parser, but am not yet succeeding. I've got npm, nvm and node installed and "npm install @postlight/mercury-parser" completed successfully. But, then what? Where do I place "import Mercury from '@postlight/mercury-parser';" and how do I execute it? Note, I'm not familiar at all with npm and nvm.
Greetings, folks!
I have been using the Mercury parser on a near-continual basis, going back seven years to the days of Readability. I have set up multiple AppleScripts on my Mac to extensively process the resulting HTML to my personal specifications, such that it has been a joy and a blessing to be able to read webpages off-line on my iPhone.
I have been watching this page since the original announcement, to see if anyone has been willing or able to provide a means of running the Mercury parser on a local Mac, by simply processing the HTML taken from a browser window.
Now that there are only two weeks remaining before Postlight shuts down the hosted Mercury Web Parser API, I am starting to believe that it will not be possible for end-users to be able to take advantage of the open source code.
Any words of advice would be greatly appreciated!
Blessings, and thank you!
I have been using the Mercury parser on a near-continual basis, going back seven years to the days of Readability. I have set up multiple AppleScripts on my Mac to extensively process the resulting HTML to my personal specifications, such that it has been a joy and a blessing to be able to read webpages off-line on my iPhone.
I have been watching this page since the original announcement, to see if anyone has been willing or able to provide a means of running the Mercury parser on a local Mac, by simply processing the HTML taken from a browser window.
Now that there are only two weeks remaining before Postlight shuts down the hosted Mercury Web Parser API, I am starting to believe that it will not be possible for end-users to be able to take advantage of the open source code.
Any words of advice would be greatly appreciated!
Blessings, and thank you!
Hello, I can't seem to pass the headers option through to the request successfully. I have it implemented as in the examples:
Mercury.parse(url, {
headers: {
dnt: '1',
cookie: '__cfduid=vqwo24522env9832hgo23gh3g23gkewe',
'accept-language': 'en-US,en;q=0.9,tr;q=0.8',
'accept-encoding': 'gzip, deflate, br',
accept: 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3',
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36',
'upgrade-insecure-requests': '1',
'cache-control': 'max-age=0,no-cache',
authority: 'medium.com'
},
contentType: 'markdown',
}).then(result => console.log(result));Am I missing something? I have made an edit to the source to fix for me so long.
function parse(_x) {
if (arguments[1].headers)
REQUEST_HEADERS = arguments[1].headers;
return _parse.apply(this, arguments);
}– mercury.js:6537
If this is an actual issue I will submit a PR
@RichardFairbanks the mercury-parser-api provides a drop-in replacement for the existing hosted API; deploying your own instance of this would make your transition seamless, and the only change you need to make to your API requests in your AppleScripts is the base URL.
Additionally, Mercury supports parsing pre-fetched HTML as detailed here.
Does this address your concerns?
Additionally, Mercury supports parsing pre-fetched HTML as detailed here.
Does this address your concerns?
