Where communities thrive

  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
    Yong Tang
    @SimonCW can you check to see if the permissions and region are configured correctly?
    @SimonCW s3 file system in tensorflow use AWS C++ SDK so the same configuration for AWS CLI will work for s3 file system as well.
    Simon WeiƟ
    Mh, yes, everything is configured and I would it expect to ask me for MFA. Usually, I work with boto3 and assume_role to get a session that I pass to methods but it seems there is no way to pass this session.
    Victor Xie
    Hi, I was wondering whether someone can kindly give me some pointers to solve a strange problem I met with Tensorflow accessing s3 file from EC2. Basically, the test code is as simple as tf.io.read_file('s3://my_private_bucket/some/file'). The line can run successfully from my local machine, but somehow get stuck (i.e. never return) when I ran on a EC2. Notice the EC2 has been provisioned with appropriate AWS IAM role to access the s3 url (verified by aws cli on the box). I tried both TF v2.5 and v2.4. Both have the same problem. Notice TF 2.6 does not have built in S3 support, so I can't try with TF 2.6.
    Andrey Klochkov
    @xwk , I'd suggest doing something like faulthandler.register(signal.SIGUSR1) and then sending SIGUSR1 to the process to see the stacktrace.
    Victor Xie

    @diggerk thanks for the suggestion. I got the statcktrace as below, but it does not seem to be very informative.

    Current thread 0x00007fe3a832d740 (most recent call first):
    File "/home/victor.xie/.cache/pypoetry/virtualenvs/image-inference-pipeline-2_RwuX6D-py3.8/lib/python3.8/site-packages/tensorflow/python/eager/execute.py", line 59 in quick_execute
    File "/home/victor.xie/.cache/pypoetry/virtualenvs/image-inference-pipeline-2_RwuX6D-py3.8/lib/python3.8/site-packages/tensorflow/python/ops/gen_io_ops.py", line 596 in read_file_eager_fallback
    File "/home/victor.xie/.cache/pypoetry/virtualenvs/image-inference-pipeline-2_RwuX6D-py3.8/lib/python3.8/site-packages/tensorflow/python/ops/gen_io_ops.py", line 558 in read_file
    File "test/test_tf_s3_support.py", line 30 in <module>

    Victor Xie
    All right. I find a workaround by reading some comments in tensorflow/tensorflow#38054. Basically, I had to set these two env vars to make it working - AWS_REGION=<your_bucket_region> and S3_VERIFY_SSL=0.
    Kyle Prifogle
    I was attempting to use https://www.tensorflow.org/io/api_docs/python/tfio/experimental/serialization/decode_json to take a string tensor containing JSON and convert it to a series of feature tensors. However I noticed that it doesn't appear to support nested json. Can someone comment, is this something that shouldn't be attempted in tensorflow preprocessing?
    Kyle Prifogle
    Nevermind I can just call decode_json iteratively
    Andrey Klochkov
    @yongtang , how soon would a release compatible with TF 2.7.x be released? Thanks!
    Yong Tang
    @diggerk We are trying to release 0.22 as soon as possible. Currently we are trying to fix the issue in tensorflow/io#1546 . Once the issue is resolved we will release 0.22.0 (compatible with TF 2.7).
    Vaibhav Singh Thapli
    2021-11-12 19:19:30.562116: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cudart64_110.dll
    Traceback (most recent call last):
    File "Tensorflow\models\research\object_detection\builders\model_builder_tf2_test.py", line 25, in <module>
    from object_detection.builders import model_builder
    File "C:\Users\vaibh\ANPR\anpr\lib\site-packages\object_detection-0.1-py3.8.egg\object_detection\builders\model_builder.py", line 37, in <module>
    from object_detection.meta_architectures import deepmac_meta_arch
    File "C:\Users\vaibh\ANPR\anpr\lib\site-packages\object_detection-0.1-py3.8.egg\object_detection\meta_architectures\deepmac_meta_arch.py", line 28, in <module>
    import tensorflow_io as tfio # pylint:disable=g-import-not-at-top
    File "C:\Users\vaibh\ANPR\anpr\lib\site-packages\tensorflow_io-0.22.0-py3.8-win-amd64.egg\tensorflow_io__init.py", line 17, in <module>
    from tensorflow_io.python.api import * # pylint: disable=wildcard-import
    File "C:\Users\vaibh\ANPR\anpr\lib\site-packages\tensorflow_io-0.22.0-py3.8-win-amd64.egg\tensorflow_io\python\api\
    init.py", line 19, in <module>
    from tensorflow_io.python.ops.io_dataset import IODataset
    File "C:\Users\vaibh\ANPR\anpr\lib\site-packages\tensorflow_io-0.22.0-py3.8-win-amd64.egg\tensorflow_io\python\ops\
    init.py", line 96, in <module>
    plugin_ops = _load_library("libtensorflow_io_plugins.so", "fs")
    File "C:\Users\vaibh\ANPR\anpr\lib\site-packages\tensorflow_io-0.22.0-py3.8-win-amd64.egg\tensorflow_io\python\ops\
    init.py", line 64, in _load_library
    l = load_fn(f)
    File "C:\Users\vaibh\ANPR\anpr\lib\site-packages\tensorflow_io-0.22.0-py3.8-win-amd64.egg\tensorflow_io\python\ops\
    init__.py", line 56, in <lambda>
    load_fn = lambda f: tf.experimental.register_filesystem_plugin(f) is None
    File "C:\Users\vaibh\ANPR\anpr\lib\site-packages\tensorflow\python\framework\load_library.py", line 218, in register_filesystem_plugin
    tensorflow.python.framework.errors_impl.AlreadyExistsError: File system for s3 already registered
    I got this error when i trained my SDD mobilenet model.
    Hi everyone,
    I have an Apple M1 chip. For a project I need an environment with Python 3.8.3, Tensorflow 2.4.1 and Tensorflow-io 0.17.1. Python and Tensorflow already work. I installed Python with Rosetta 2 and pyenv and compiled Tensorflow from source. So far I haven't found a suitable solution for tensorflow-io.
    'python3 setup.py -q bdist_wheel' didn't worked.
    Does anyone have an idea how I can install tensorflow-io?
    ranjeet gupta
    Hi all, does tensorflow 2.8.0 support fsspec based file systems? I am trying to write tensorboard logs using keras callback ``` tf.keras.callbacks.TensorBoard("myfsspec:path") ? I believe network IO operations get routed to tensorflow io module. I am not sure of tensorflow io is capable of supporting new fsspec based implementations. Any pointers will be much appreciated. thanks !
    Junfan Zhang
    Hi anyone could help me check this PR? tensorflow/io#1656
    Hi all, Arrow has released version 7.0 with a number of bug fixes and significant performance improvements. How about upgrading arrow to version 7.0?
    Austin Anderson
    @yongtang FYI, the GCP credentials job is broken due to an infrastructure problem; I'm working on fixing it now
    Yong Tang
    Thanks @angerson !
    Veeranjaneyulu Toka
    Hi All, i am able to install tf OD api, but when i try to run, am getting the below error. Anybody has faced this issue and any workarounds for the same. Thanks! File "C:\Users\Veeru\anaconda3\envs\tf41_py38\lib\site-packages\tensorflow\python\framework\load_library.py", line 178, in register_filesystem_plugin
    tensorflow.python.framework.errors_impl.AlreadyExistsError: File system for s3 already registered
    Oh, sorry. Looks like @vaibhavthapli posted the same error, did you find any solution for the same. Thankse!
    Veeranjaneyulu Toka
    Looks like tensorflow, tf-models-official and tensorflow-io versions should be compatible
    when i try tf 2.4.4, tf-io 0.17.1 and tf-models-official=2.2.0 solved the above issue.
    Ankit Gupta
    Hi folks, I'm looking to get tensorflow and tensorflow_io working on an M1 mac in an arm64 conda environment. I'm able to get tensorflow working just fine, but when I install tensorflow_io, there seems to be some sort of compiled file linking issue (unable to open file: libtensorflow_io.so). Has anyone successfully gotten this to build on an M1 mac? I'm using python 3.9 and tensorflow is working just fine (it's able to see my Mac's GPU etc) .
    Shabbeer Basha
    Hi all, I am facing the below issue while reading tfrecords from aws s3 bucket: tensorflow.python.framework.errors_impl.InvalidArgumentError: Expected 'tf.Tensor(False, shape=(), dtype=bool)' to be true. Summarized data: b'No files matched pattern: s3://distraction-model-retrain-dataset/20200408_motion_trigger_200530/train*.tfrecord.gz'
    Any inputs to solve this issue.
    Tom Birch
    Is it expected behavior that tf.keras.models.Model.save does not save optimizer weights/variables when using the default "TensorFlow SavedModel" format, but does save them when using the "h5" format?
    AttributeError: module 'tensorflow._api.v2.experimental' has no attribute 'image'