Hi, I am searching for a way to build a tf.data.dataset or tfio.IODataset from Apache Parquet files residing on S3. However, I cannot access the data from S3, e.g. tf.data.Dataset.list_files(s3uri + "/*", shuffle=True) gives the error InvalidArgumentError: Expected 'tf.Tensor(False, shape=(), dtype=bool)' to be true. Summarized data: b'No files matched pattern:....
On Sagemaker Studio this worked out of the box but I assume they mount s3?
Is there a good way to achieve this?
1 reply
@SimonCW Yes s3 file system in tensorflow provides the support so that it is possible to access s3 files through
s3://bucket/object without mounting. If it is not working, it could be related to configuration as s3 file system needs information about AWS region and permissions , either config file or environmental variable (see https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html)
@SimonCW s3 file system in tensorflow use AWS C++ SDK so the same configuration for AWS CLI will work for s3 file system as well.
Hi, I was wondering whether someone can kindly give me some pointers to solve a strange problem I met with Tensorflow accessing s3 file from EC2. Basically, the test code is as simple as tf.io.read_file('s3://my_private_bucket/some/file'). The line can run successfully from my local machine, but somehow get stuck (i.e. never return) when I ran on a EC2. Notice the EC2 has been provisioned with appropriate AWS IAM role to access the s3 url (verified by aws cli on the box). I tried both TF v2.5 and v2.4. Both have the same problem. Notice TF 2.6 does not have built in S3 support, so I can't try with TF 2.6.
@diggerk thanks for the suggestion. I got the statcktrace as below, but it does not seem to be very informative.
Current thread 0x00007fe3a832d740 (most recent call first):
File "/home/victor.xie/.cache/pypoetry/virtualenvs/image-inference-pipeline-2_RwuX6D-py3.8/lib/python3.8/site-packages/tensorflow/python/eager/execute.py", line 59 in quick_execute
File "/home/victor.xie/.cache/pypoetry/virtualenvs/image-inference-pipeline-2_RwuX6D-py3.8/lib/python3.8/site-packages/tensorflow/python/ops/gen_io_ops.py", line 596 in read_file_eager_fallback
File "/home/victor.xie/.cache/pypoetry/virtualenvs/image-inference-pipeline-2_RwuX6D-py3.8/lib/python3.8/site-packages/tensorflow/python/ops/gen_io_ops.py", line 558 in read_file
File "test/test_tf_s3_support.py", line 30 in <module>
All right. I find a workaround by reading some comments in tensorflow/tensorflow#38054. Basically, I had to set these two env vars to make it working - AWS_REGION=<your_bucket_region> and S3_VERIFY_SSL=0.
I was attempting to use https://www.tensorflow.org/io/api_docs/python/tfio/experimental/serialization/decode_json to take a string tensor containing JSON and convert it to a series of feature tensors. However I noticed that it doesn't appear to support nested json. Can someone comment, is this something that shouldn't be attempted in tensorflow preprocessing?
@diggerk We are trying to release 0.22 as soon as possible. Currently we are trying to fix the issue in tensorflow/io#1546 . Once the issue is resolved we will release 0.22.0 (compatible with TF 2.7).
2021-11-12 19:19:30.562116: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cudart64_110.dll
Traceback (most recent call last):
File "Tensorflow\models\research\object_detection\builders\model_builder_tf2_test.py", line 25, in <module>
from object_detection.builders import model_builder
File "C:\Users\vaibh\ANPR\anpr\lib\site-packages\object_detection-0.1-py3.8.egg\object_detection\builders\model_builder.py", line 37, in <module>
from object_detection.meta_architectures import deepmac_meta_arch
File "C:\Users\vaibh\ANPR\anpr\lib\site-packages\object_detection-0.1-py3.8.egg\object_detection\meta_architectures\deepmac_meta_arch.py", line 28, in <module>
import tensorflow_io as tfio # pylint:disable=g-import-not-at-top
File "C:\Users\vaibh\ANPR\anpr\lib\site-packages\tensorflow_io-0.22.0-py3.8-win-amd64.egg\tensorflow_io__init.py", line 17, in <module>
from tensorflow_io.python.api import * # pylint: disable=wildcard-import
File "C:\Users\vaibh\ANPR\anpr\lib\site-packages\tensorflow_io-0.22.0-py3.8-win-amd64.egg\tensorflow_io\python\api\init.py", line 19, in <module>
from tensorflow_io.python.ops.io_dataset import IODataset
File "C:\Users\vaibh\ANPR\anpr\lib\site-packages\tensorflow_io-0.22.0-py3.8-win-amd64.egg\tensorflow_io\python\ops\init.py", line 96, in <module>
plugin_ops = _load_library("libtensorflow_io_plugins.so", "fs")
File "C:\Users\vaibh\ANPR\anpr\lib\site-packages\tensorflow_io-0.22.0-py3.8-win-amd64.egg\tensorflow_io\python\ops\init.py", line 64, in _load_library
l = load_fn(f)
File "C:\Users\vaibh\ANPR\anpr\lib\site-packages\tensorflow_io-0.22.0-py3.8-win-amd64.egg\tensorflow_io\python\ops\init__.py", line 56, in <lambda>
load_fn = lambda f: tf.experimental.register_filesystem_plugin(f) is None
File "C:\Users\vaibh\ANPR\anpr\lib\site-packages\tensorflow\python\framework\load_library.py", line 218, in register_filesystem_plugin
py_tf.TF_RegisterFilesystemPlugin(plugin_location)
tensorflow.python.framework.errors_impl.AlreadyExistsError: File system for s3 already registered
Traceback (most recent call last):
File "Tensorflow\models\research\object_detection\builders\model_builder_tf2_test.py", line 25, in <module>
from object_detection.builders import model_builder
File "C:\Users\vaibh\ANPR\anpr\lib\site-packages\object_detection-0.1-py3.8.egg\object_detection\builders\model_builder.py", line 37, in <module>
from object_detection.meta_architectures import deepmac_meta_arch
File "C:\Users\vaibh\ANPR\anpr\lib\site-packages\object_detection-0.1-py3.8.egg\object_detection\meta_architectures\deepmac_meta_arch.py", line 28, in <module>
import tensorflow_io as tfio # pylint:disable=g-import-not-at-top
File "C:\Users\vaibh\ANPR\anpr\lib\site-packages\tensorflow_io-0.22.0-py3.8-win-amd64.egg\tensorflow_io__init.py", line 17, in <module>
from tensorflow_io.python.api import * # pylint: disable=wildcard-import
File "C:\Users\vaibh\ANPR\anpr\lib\site-packages\tensorflow_io-0.22.0-py3.8-win-amd64.egg\tensorflow_io\python\api\init.py", line 19, in <module>
from tensorflow_io.python.ops.io_dataset import IODataset
File "C:\Users\vaibh\ANPR\anpr\lib\site-packages\tensorflow_io-0.22.0-py3.8-win-amd64.egg\tensorflow_io\python\ops\init.py", line 96, in <module>
plugin_ops = _load_library("libtensorflow_io_plugins.so", "fs")
File "C:\Users\vaibh\ANPR\anpr\lib\site-packages\tensorflow_io-0.22.0-py3.8-win-amd64.egg\tensorflow_io\python\ops\init.py", line 64, in _load_library
l = load_fn(f)
File "C:\Users\vaibh\ANPR\anpr\lib\site-packages\tensorflow_io-0.22.0-py3.8-win-amd64.egg\tensorflow_io\python\ops\init__.py", line 56, in <lambda>
load_fn = lambda f: tf.experimental.register_filesystem_plugin(f) is None
File "C:\Users\vaibh\ANPR\anpr\lib\site-packages\tensorflow\python\framework\load_library.py", line 218, in register_filesystem_plugin
py_tf.TF_RegisterFilesystemPlugin(plugin_location)
tensorflow.python.framework.errors_impl.AlreadyExistsError: File system for s3 already registered
I got this error when i trained my SDD mobilenet model.
Hi everyone,
I have an Apple M1 chip. For a project I need an environment with Python 3.8.3, Tensorflow 2.4.1 and Tensorflow-io 0.17.1. Python and Tensorflow already work. I installed Python with Rosetta 2 and pyenv and compiled Tensorflow from source. So far I haven't found a suitable solution for tensorflow-io.
'python3 setup.py -q bdist_wheel' didn't worked.
Does anyone have an idea how I can install tensorflow-io?
I have an Apple M1 chip. For a project I need an environment with Python 3.8.3, Tensorflow 2.4.1 and Tensorflow-io 0.17.1. Python and Tensorflow already work. I installed Python with Rosetta 2 and pyenv and compiled Tensorflow from source. So far I haven't found a suitable solution for tensorflow-io.
'python3 setup.py -q bdist_wheel' didn't worked.
Does anyone have an idea how I can install tensorflow-io?
Hi all, does tensorflow 2.8.0 support fsspec based file systems? I am trying to write tensorboard logs using keras callback ``` tf.keras.callbacks.TensorBoard("myfsspec:path") ? I believe network IO operations get routed to tensorflow io module. I am not sure of tensorflow io is capable of supporting new fsspec based implementations. Any pointers will be much appreciated. thanks !
Hi All, i am able to install tf OD api, but when i try to run, am getting the below error. Anybody has faced this issue and any workarounds for the same. Thanks! File "C:\Users\Veeru\anaconda3\envs\tf41_py38\lib\site-packages\tensorflow\python\framework\load_library.py", line 178, in register_filesystem_plugin
py_tf.TF_RegisterFilesystemPlugin(plugin_location)
tensorflow.python.framework.errors_impl.AlreadyExistsError: File system for s3 already registered
py_tf.TF_RegisterFilesystemPlugin(plugin_location)
tensorflow.python.framework.errors_impl.AlreadyExistsError: File system for s3 already registered
Oh, sorry. Looks like @vaibhavthapli posted the same error, did you find any solution for the same. Thankse!
when i try tf 2.4.4, tf-io 0.17.1 and tf-models-official=2.2.0 solved the above issue.
Hi folks, I'm looking to get tensorflow and tensorflow_io working on an M1 mac in an arm64 conda environment. I'm able to get tensorflow working just fine, but when I install tensorflow_io, there seems to be some sort of compiled file linking issue (
unable to open file: libtensorflow_io.so). Has anyone successfully gotten this to build on an M1 mac? I'm using python 3.9 and tensorflow is working just fine (it's able to see my Mac's GPU etc) .
Hi all, I am facing the below issue while reading tfrecords from aws s3 bucket: tensorflow.python.framework.errors_impl.InvalidArgumentError: Expected 'tf.Tensor(False, shape=(), dtype=bool)' to be true. Summarized data: b'No files matched pattern: s3://distraction-model-retrain-dataset/20200408_motion_trigger_200530/train*.tfrecord.gz'
Any inputs to solve this issue.
