These are chat archives for frictionlessdata/chat

14th
Jan 2019
Michael Joseph Rosenthal
@micimize
Jan 14 17:02

@akariv I'm running into a really confusing issue with dataflows.

Basically, I made a general flow called partition, and I want to wrap it in order to partition based on a goodtables report:

def goodtables_partition(source_name, target_name):
    def func(package: PackageWrapper):
        source = next((r for r in package if r.res.name == target), None)
        assert source is not None, (
            "cannot validate nonexistent resource %s" % source_name
        )
        report = validator.validate(source)
        partitioner = build_error_partitioner(report)
        partition_flow = partition(source_name, target_name, partitioner)
        return partition_flow(package)
    return func
the issue is that the assertion always fails, because the contents of the package wrapper is always empty when I try to extract the source, and I have no idea why
Michael Joseph Rosenthal
@micimize
Jan 14 17:10
if I try and print out the packages as a stand-alone it works, and partition seems to work on it's own. Is it maybe some quality of lazy iterators, or an implementation detail that makes the contents of the package empty before yield package.pkg or something?
Michael Joseph Rosenthal
@micimize
Jan 14 17:50
Idk what the issue was, but I just added a get_partitioner argument to avoid building the flow-within-flow
Adam Kariv
@akariv
Jan 14 18:29
@micimize
The contract for a package processor is that you need to yield the package (package.pkg) first, and only then the resources are populated.
then you should yield each resource iterator / ResourceWrapper that’s in the package
Michael Joseph Rosenthal
@micimize
Jan 14 18:39
@akariv ok, cool that makes sense. I think I've almost got all the kinks ironed out, still struggling with the goodtables integration though
Michael Joseph Rosenthal
@micimize
Jan 14 18:53
looks like I'm writing a goodtables+dataflows streaming bridge :sweat_smile:
Adam Kariv
@akariv
Jan 14 20:46
@micimize this is awesome!