{o:outportname} placeholder in the command, of course given that the executable can take the destination filename as a parameter.
So, imagining that pdfimage has a flag, -o for output filename (and -i for input files), that would be:
proc := wf.NewProc("proc", "pdfimage -i {i:inport-name} -o {o:outport-name}")etc.
components.NewFileSplitter(wf, "file_splitter", 1)func main() {
path := "./data/*/attachments/*.pdf"
wf := sp.NewWorkflow("pdf2ocr", 20)
globpdf := comp.NewFileGlobber(wf, "findpdf", path)
ximages := wf.NewProc("pdfimages", "pdfimages -tiff -p \"{i:in}\" \"{i:in}\" ; ls \"{i:in}\"*.tif > {o:out} ") // {o:out}
ximages.In("in").From(globpdf.Out())
// < test xargs -L 1 -I{} tesseract {} {} --psm 1 --oem 1 -l eng
ocr := wf.NewProc("tesseract", "< \"{i:in}\" xargs -L 1 -I{} tesseract {} {}.ocr --psm 1 --oem 1 -l eng; ls \"{i:in}\"*.txt > {o:out} ") // {o:out}
ocr.In("in").From(ximages.Out("out"))
wf.Run()
}And, is there some documentation about the pdfimages command online?
I was trying something like this ... but running into errors: https://gist.github.com/samuell/56ac30625417b881427dfea7d2675d43
The errors might be with the tesseract command ... I'm not too familiar with it yet, and might have changed too many things.
@windhooked Will check!
The temporary path is created as a subfolder under the main Go-script's folder.
This is why, if you want to refer to existing files in the main directory, you need to prepend ../ to the paths. Output paths though, are supposed to be written relative to the temporary directory, from where they will be moved into their final paths as part of scipipe's code for finalizing a task (to make file writes atomic).
Ok, one problem with the approach of my code suggestion:
sp.ExecCmd() will not be executed inside the temporary task's folder.
Thus, since input paths are prepended with ../ by SciPipe, to support the normal operation (executing inside the temp folder), one will need to remove that ../ part manually ...
for ... line in your code:tifFile = strings.Replace(tifFile, "../", "", 1)
I've just started using scipipe for some of my workflows, and I'm still figuring things out.
Sorry for a possibly naïve question: is there a way to automatically re-run the workflow when some files are updated (i.e. something Make-like)?
Thanks for the quick response!
Indeed, I was mainly wondering about the changes in the upstream files and quickly picking up the workflow from the respective part of the pipeline, maybe having something an argument to track timestamps (or/and hashes as well that might be stored in audit files) of files and only skip the step in the Make-like fashion when the downstream file is younger than the upstream one...
On that note, also forcing the workflow to re-run would be nice, I think manually deleting files for large workflows might be too error-prone. Works fine for small workflows, you're right!
Thanks again, looking forward to scipipe's progress!
Hi @biotom! Welcome to the chat, and thank you for your question! Re. contributors: Right now, I think any bug fixes is the most important way of contributing at the moment.
There would also be great with more support for various HPC systems etc, but I think a bit of architecture for easily plugging in remote runners is needed there, which might not be an easy first task (the architectural bits).