TFIO_DATAPATH=/home/pi/.cache/bazel/_bazel_pi/be6ac8eba0db45fb771509e53438aa5b/execroot/org_tensorflow_io/bazel-out/k8-fastbuild/bin/ python3 -m pytest -s -v tests/test_pcap_eager.py
export TFIO_DATAPATH=$(bazel info bazel-bin )
or export TFIO_DATAPATH=$(bazel info bazel-bin -c opt )
E NotImplementedError: unable to open file: /home/pi/tf/io/tensorflow_io/core/python/ops/__init__.py, from paths: ['/home/pi/tf/io/tensorflow_io/core/python/ops/libtensorflow_io.so', 'bazel-bin/tensorflow_io/core/python/ops/libtensorflow_io.so']
E caused by: ['/home/pi/tf/io/tensorflow_io/core/python/ops/libtensorflow_io.so: cannot open shared object file: No such file or directory', 'bazel-bin/tensorflow_io/core/python/ops/libtensorflow_io.so: undefined symbol: _ZN4absl14lts_2020_02_2520raw_logging_internal21internal_log_functionB5cxx11E']
bazel-bin/tensorflow_io/core/python/ops/libtensorflow_io.so: undefined symbol: _ZN4absl14lts_2020_02_2520raw_logging_internal21internal_log_functionB5cxx11E
(p3) pi@pig:~/tf/io$ TFIO_DATAPATH=bazel-bin python3 -m pytest -s -v tests/test_pcap.py
================================================================== test session starts ===================================================================
platform linux -- Python 3.8.5, pytest-6.2.2, py-1.10.0, pluggy-0.13.1 -- /home/pi/p3/bin/python3
cachedir: .pytest_cache
rootdir: /home/pi/tf/io
collecting ... 2021-03-22 12:34:48.901457: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
collected 1 item
tests/test_pcap.py::test_pcap_input Testing PcapDataset
FAILED
======================================================================== FAILURES ========================================================================
____________________________________________________________________ test_pcap_input _____________________________________________________________________
def test_pcap_input():
"""test_pcap_input
"""
print("Testing PcapDataset")
pcap_filename = os.path.join(
os.path.dirname(os.path.abspath(__file__)), "test_pcap", "http.pcap"
)
file_url = "file://" + pcap_filename
> dataset = tfio.IODataset.from_pcap(file_url, capacity=5).batch(1)
tests/test_pcap.py:34:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tensorflow_io/core/python/ops/io_dataset.py:309: in from_pcap
return pcap_dataset_ops.PcapIODataset(filename, internal=True, **kwargs)
tensorflow_io/core/python/ops/pcap_dataset_ops.py:36: in __init__
resource = core_ops.io_pcap_readable_init(
tensorflow_io/core/python/ops/__init__.py:88: in __getattr__
return getattr(self._load(), attrb)
tensorflow_io/core/python/ops/__init__.py:84: in _load
self._mod = _load_library(self._library)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
filename = '/home/pi/tf/io/tensorflow_io/core/python/ops/__init__.py', lib = 'op'
def _load_library(filename, lib="op"):
"""_load_library"""
f = inspect.getfile(sys._getframe(1)) # pylint: disable=protected-access
# Construct filename
f = os.path.join(os.path.dirname(f), filename)
filenames = [f]
# Add datapath to load if en var is set, used for running tests where shared
# libraries are built in a different path
datapath = os.environ.get("TFIO_DATAPATH")
if datapath is not None:
# Build filename from:
# `datapath` + `tensorflow_io` + `package_name` + `relpath_to_library`
rootpath = os.path.dirname(sys.modules["tensorflow_io"].__file__)
filename = sys.modules[__name__].__file__
f = os.path.join(
datapath,
"tensorflow_io",
os.path.relpath(os.path.dirname(filename), rootpath),
os.path.relpath(f, os.path.dirname(filename)),
)
filenames.append(f)
# Function to load the library, return True if file system library is loaded
if lib == "op":
load_fn = tf.load_op_library
elif lib == "dependency":
load_fn = lambda f: ctypes.CDLL(f, mode=ctypes.RTLD_GLOBAL)
elif lib == "fs":
load_fn = lambda f: tf.experimental.register_filesystem_plugin(f) is None
else:
load_fn = lambda f: tf.compat.v1.load_file_system_library(f) is None
# Try to load all paths for file, fail if none succeed
errs = []
for f in filenames:
try:
l = load_fn(f)
if l is not None:
return l
except (tf.errors.NotFoundError, OSError) as e:
errs.append(str(e))
> raise NotImplementedError(
"unable to open file: "
+ "{}, from paths: {}\ncaused by: {}".format(filename, filename
Hi, I am searching for a way to build a tf.data.dataset or tfio.IODataset from Apache Parquet files residing on S3. However, I cannot access the data from S3, e.g. tf.data.Dataset.list_files(s3uri + "/*", shuffle=True)
gives the error InvalidArgumentError: Expected 'tf.Tensor(False, shape=(), dtype=bool)' to be true. Summarized data: b'No files matched pattern:...
.
On Sagemaker Studio this worked out of the box but I assume they mount s3?
Is there a good way to achieve this?
s3://bucket/object
without mounting. If it is not working, it could be related to configuration as s3 file system needs information about AWS region and permissions , either config file or environmental variable (see https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html)
@diggerk thanks for the suggestion. I got the statcktrace as below, but it does not seem to be very informative.
Current thread 0x00007fe3a832d740 (most recent call first):
File "/home/victor.xie/.cache/pypoetry/virtualenvs/image-inference-pipeline-2_RwuX6D-py3.8/lib/python3.8/site-packages/tensorflow/python/eager/execute.py", line 59 in quick_execute
File "/home/victor.xie/.cache/pypoetry/virtualenvs/image-inference-pipeline-2_RwuX6D-py3.8/lib/python3.8/site-packages/tensorflow/python/ops/gen_io_ops.py", line 596 in read_file_eager_fallback
File "/home/victor.xie/.cache/pypoetry/virtualenvs/image-inference-pipeline-2_RwuX6D-py3.8/lib/python3.8/site-packages/tensorflow/python/ops/gen_io_ops.py", line 558 in read_file
File "test/test_tf_s3_support.py", line 30 in <module>