Upload, Store and Download small and huge files to/from file system (FS) via DDP and HTTP
if (http.request.headers.range) {
const vRef = fileRef.versions[version];
let range = _.clone(http.request.headers.range);
const array = range.split(/bytes=([0-9]*)-([0-9]*)/);
const start = parseInt(array[1]);
let end = parseInt(array[2]);
if (isNaN(end)) {
// Request data from AWS:S3 by small chunks
end = (start + this.chunkSize) - 1;
if (end >= vRef.size) {
end = vRef.size - 1;
}
}
opts.Range = `bytes=${start}-${end}`;
http.request.headers.range = `bytes=${start}-${end}`;
}const s3Stream = s3.getObject(opts).createReadStream();
// Listen for errors returned by the service
s3Stream.on('error', function(err) {
// NoSuchKey: The specified key does not exist
console.error(err);
});
s3Stream.pipe(http.response).on('error', function(err) {
// capture any errors that occur when writing data to the file
console.error('File Stream:', err);
}).on('close', function() {
console.log('Done.');
});@dr-dimitru Ok. I think the better option would probably be to check if the header is set and if not then check the size of the file being requested, if it's a large one then only return the first chunk? I'm not sure if once started then the browser will catch on and keep requesting the rest, worth testing though. The problem with the current method is that it just streams directly from start to finish, so whereas with your solution I can "skip" to the middle of the video and start playback, with this solution I cannot (if that makes sense?)
Once I get a nice solution I'll definitely make the PR :)
s3Stream.pipe(http.response) for downloading whole file and let HTTP protocol manage progressive download by itselfand use
Range header for playback (e.g. inline)
protected hook — https://github.com/VeliovGroup/Meteor-Files/blob/master/docs/constructor.md
Hello @kakadais ,
Thank you for your proposal, we are looking forward to release major release for Meteor-Files package, with idea of pluggable adapters for 3rd party storage, we will take your suggestions into consideration.
To be honest upload re-tries should be managed by adapter/3rd party storage implementation
what I need to do is:
- Create a temporary file;
- Write lots of data to it - This includes looping through a collection and exporting data, so I need to be able to "append" data to the temp file;
- return the newly generated temp file to the client - It will be immediately downloaded there;
- After the download ends, the file has to be removed/destroyed.
From all I read about this component I think I would be able to do all that with it - Is that correct??
Can anyone give me any instructions on how I should proceed in order to do that?
-= Thanks in advance =-
webapp module
.addFile, then fileURL to get downloadable URL
But I do not know exactly how to get the link after uploading and put it in the editor.
Do I have to subscribe to get the link to that file after upload?
And if so. Is there a way to subscribe to that file right away?
.link() method documentation, hope now this method usage will be more straightforward. Also answered on your veliovgroup/Meteor-Files#769
FilesCollection#unlink(fileObj, 'version')
Hello guys-
Meteor-Files is great to integrate file server as a single one, so server scales are much easier than before, but using multi-core would be good to go from the scratch.
Have you any idea for clustering for Meteor?
I'm expecting none Node or System level something like pm2/passenger , but something higher level integration such as kadira:clustering, but seems like not maintained.
Any idea?
@dr-dimitru Oh I think I made a misunderstanding, that's why I said None System Level something cause I have a knowledge about the field and used it.
I've seen the below video recently for cluster package and realized Microservices for Meteor were already concerned from 2015,
and thought that this model is super great.
https://www.youtube.com/watch?v=oudsAQZkvzQ&feature=youtu.be&t=15m27s
I think the reason why people are not implemented there system as a microservice is 'Annoying'. If we handle whole system level to salce up/out then the configuration and flexibility will hold us even a single changes, so some people uses NetflixOSS stuff to build their DevOps efficiently.
But after I've looked up the NetflixOSS things, it is still complex enough because of too much options and almost of it is not required at the small/middle system so we couldn't start on the same line.
The metreorhacks/cluster package is simple enough and especially Multi-Core Support strategy by node worker is so cool and works good still, I got a curiosity 'Isn't there any movement about this' things.
I should dig some more for this movement in Meteor or Node- ;)
So Need and Interesting part-