Python library to extend functional operations in the spirit of functional programming in scala. Provides support for missing operations and chain style functional calls.
good idea for tests though
class LazyFile(object):
# pylint: disable=too-few-public-methods,too-many-instance-attributes
def __init__(self, path, delimiter=None, mode='r', buffering=-1, encoding=None,
errors=None, newline=None):
# pylint: disable=too-many-arguments
self.path = path
self.delimiter = delimiter
self.mode = mode
self.buffering = buffering
self.encoding = encoding
self.errors = errors
self.newline = newline
self.file = None
def __iter__(self):
if self.file is not None:
self.file.close()
self.file = builtins.open(self.path, mode=self.mode, buffering=self.buffering,
encoding=self.encoding, errors=self.errors, newline=self.newline)
return self.file
however, the file stays open...
class LazyFile(object):
# pylint: disable=too-few-public-methods,too-many-instance-attributes
def __init__(self, path, delimiter=None, mode='r', buffering=-1, encoding=None,
errors=None, newline=None):
# pylint: disable=too-many-arguments
self.path = path
self.delimiter = delimiter
self.mode = mode
self.buffering = buffering
self.encoding = encoding
self.errors = errors
self.newline = newline
self.file = None
def __iter__(self):
with builtins.open(self.path,
mode=self.mode,
buffering=self.buffering,
encoding=self.encoding,
errors=self.errors,
newline=self.newline) as file_content:
for line in file_content:
yield line
I don't really have to say anything about the docs.
Well, after a while of wrestling with the release, its done. In releasing
0.4.0 I found an issue where because the wheel distribution depends on the build environment (aka, running on python 2 including enum34 breaking python3 OR running on python3 not including enum34 and breaking python2) I needed to make a hotfix release in 0.4.1
Solution was to completely remove enum34 since I don't use it all that much so its not worth the monolithic hastle (and I dont trust
enum-compat either)
But, here are the release notes: https://github.com/EntilZha/ScalaFunctional/releases/tag/v0.4.1
Small suggestions:
- make
__str__show only part of the sequence, as in[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, ...]. Would be even greater if this only evaluated this part, as this would make it usable with infinite generators.* - make
__repr__show what it really is; maybe<Sequence: [1, 2, 3, 4...]>or not evaluate at all and show<Sequence wrapping list>,<Sequence wrapping generator>,<Sequence wrapping File> - add
seq.countas a wrapper foritertools.count? I'm not sure how far down the "replace seq(thing()) with seq.thing()" path you want to go, that may be an overkill.
*(seq(itertools.count()) just killed my computer :P)
the main reason for
__repr__ to do the same thing is for ipython notebook
and interactive python terminal sessions
double checking right now, but I believe these use repr
Ya, it uses repr
hm, maybe
__str__ and __repr__ could behave differently depending on type of underlying sequence?
right now, there are a list of execution strategies
right now there is only one
but this would be the mechanism to change this I think
Thought I would mention I am working on a new project (starting as a final course project in distributed systems) here https://github.com/EntilZha/GrappaRDD. It uses a C++ MPI library called Grappa as an execution engine to implement RDDs from spark. The interesting part is if it goes well, I might think about making python hooks to it via ScalaFunctional
Getting an odd failure in only Python 3.4 (3.3 and 3.5) work, if anyone has any thoughts that would be useful https://travis-ci.org/EntilZha/PyFunctional/jobs/128396745. It fails on the parallel tests for some reason
Planning on releasing 0.7.0 next week, primary features that will be shipping is the parallel execution engine and file compression support. The main website and docs are now at www.pyfunctional.org and docs.pyfunctional.org respectively (readthedocs can't change scalafunctional.readthedocs.org to pyfunctional.readthedocs.org so this was an easy solution).
The release following this one will most likely be 1.0, so open to thoughts on what should go in. So far, would like to revamp the readme docs, the webpage at www.pyfunctional.org (currently a clone of the readme), the actual docs, clarify package description, revisit linq, revisit underscore, revisit supporting more than just sqlite, and/or look into adding an option/support to stream things (rather than force open iterables). All this is on top of my head so there could be more
eg: functools /itertools
@mingsterism The primary goal is to mimick the Scala collections and Spark RDD API. functools/itertools can accomplish the same goal, but it will just look different. The other difference is that there are a number of nice features out of the box like reading json/jsonl/text/csv and in the next release parallelism
0.7.0 is released now, thanks @ChuyuHsu and versae for the great work https://github.com/EntilZha/PyFunctional/releases/tag/v0.7.0. Also posted on python reddit https://www.reddit.com/r/Python/
Hey! Just got directed to this project by @carlthome from https://github.com/neverendingqs/pyiterable/issues/71#issuecomment-226752854. I'm off to bed, but just wanted to say hi and am interested to see if there's opportunities to consolidate or even jump on board here.
Looks like docs aren't public for some reason http://pyiterable.readthedocs.io/en/latest/
Would be great to hear about any ideas you have in mind
Np. Not sure exactly what I want to do next. A SQL to lineage compiler would be interesting. Would be along the lines of letting you query python objects with SQL in a similar way you can write Spark SQL queries against a data frame. I might also be interested in venturing out of data pipelines to a better
_ operator or pattern matching (that would be really hard to do I think, been looking at existing libraries). Open to suggestions
