Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    android
    @henrytansetiawan
    ERROR: /root/.cache/bazel/_bazel_root/a7ecd8237744645c5d189c197108d6d2/external/libwebp/BUILD.bazel:8:1: C++ compilation of rule '@libwebp//:libwebp' failed (Exit 1): gcc failed: error executing command
    Yong Tang
    @yongtang
    It works fine in Travis CI. We were using Ubuntu 14.04 and using Bazel 0.24.1. Newer versions of Bazel or new Ubuntu might have issues, though.
    android
    @henrytansetiawan
    Verified using the same Bazel v0.24.1 and Ubuntu 14.04.5 LTS
    Yong Tang
    @yongtang
    @henrytansetiawan At the minimum the following has to be installed:
    apt-get -y install git gcc g++ make patch unzip curl
    @henrytansetiawan Also apt-get -y install python
    android
    @henrytansetiawan
    Ok, redone
    seemed to fail somewhere else :)
    ERROR: /root/.cache/bazel/_bazel_root/a7ecd8237744645c5d189c197108d6d2/external/giflib/BUILD.bazel:8:1: C++ compilation of rule '@giflib//:giflib' failed (Exit 1): gcc failed: error executing command
    orthogonal to this issue, I am wondering if you have any experience of porting tf_py_test(...) into tensorflow_io ?
    @yongtang I also noticed that there is tensorflow_io/bigquery and also cloud/bigquery (one I ported), is this the same bigquery bits?
    Yong Tang
    @yongtang
    @henrytansetiawan I think by default, most likely python test could work already, if you place it in tests directory, and type TFIO_DATAPATH=bazel python -m pyrest -s -v tests/test_xxx.py
    @henrytansetiawan The tensorflow_io/bigquery was added by vlasenkoalexey. I think there might be overlap but I am not sure.
    android
    @henrytansetiawan

    @yongtang ok thanks for the pointers. I have gone ahead opening 2 (test) issues (below) to complete porting contrib.cloud/*, just so it can be tracked and make it more explicit what is missing from the original contrib.cloud. Outside of porting these tf_py_test targets, the rest however are already ported.

    tensorflow/io#396
    tensorflow/io#395

    Yong Tang
    @yongtang
    @henrytansetiawan Thanks!

    @henrytansetiawan Oh by the way, to invoke pytest it should be

    TFIO_DATAPATH=bazel-bin python -m pyrest -s -v tests/test_xxx.py

    The earlier message missed the TFIO_DATAPATH=bazel => TFIO_DATAPATH=bazel-bin part

    android
    @henrytansetiawan
    Thanks!
    Fabian-Robert Stöter
    @faroit
    Hi, just a quick question since the readme is not clear on this: the tf io master branch works with tf 1.x and 2.x or just 2.x beta?
    Yong Tang
    @yongtang

    @faroit the code base of master branch could work with 1.x and 2.0 if you build on your own.

    If you are looking for pre-built binaries, at the moment:

    pip install tensorflow-io-nightly

    works with TF 1.14

    pip install tensorflow-io-2.0-preview

    works with TF 2.0 (beta 1)

    Fabian-Robert Stöter
    @faroit
    thanks, this exactly what I needed ;-)
    I would suggest you add this bit to the readme
    Yong Tang
    @yongtang
    @faroit documentation is probably something we badly missing, will definitely try to improve.
    AsimHomaid
    @AsimHomaid
    hi, can someone explain why if using tensorflow_io.ffmpeg on videos. shapes are still undefined?
    dataset = tf_io.VideoDataset(filename=['file1.mp3', 'file2.mp3'], batch=10)
    still shows `dataset.output_shapes
    dataset.output_shapes
    TensorShape([Dimension(None), Dimension(None), Dimension(None), Dimension(3)])
    Yong Tang
    @yongtang
    @AsimHomaid Can you open the issue in GitHub? I can take a look.
    AsimHomaid
    @AsimHomaid
    hi, how do I display the images if i used io.ffmpeg as the following dataset = tensorflow_io.ffmpeg.VideoDataset(video_path, batch=60)?
    AsimHomaid
    @AsimHomaid
    for context, I am running a video for prediction on a pre-trained model. and want to know what are the frames the neural network is seeing before it runs the model.
    congrats Bryan!
    Yong Tang
    @yongtang
    Great work Bryan! And thanks @ewilderj for the help! :tada: :fireworks: :+1:
    awl
    @awl
    is this the right place to ask questions related to problems i'm having with tensorflow-serving? i have a bert model that i'm trying to serve with it and everything works from the command line, but when i package it up into a docker container and try to make the same prediction via the http interface, it gives me an error that "label_ids is both fed and fetched" which i'm led to believe means it isn't allowed in both the input and the output. if this is the case, how do i go about fixing it, and more importantly why does it work from the CLI?
    thanks for any help you can provide or for pointing me to the right place to ask
    Kai Waehner
    @kaiwaehner

    @yongtang After summer break, we finally have some time now to implement our sophisticated "Streaming Kafka ML" demo in the next few weeks and want to leverage TensorFlow IO.

    We have some test data from car sensors here: https://github.com/kaiwaehner/hivemq-mqtt-tensorflow-kafka-realtime-iot-machine-learning-training-inference/blob/master/testdata/car-sensor-data.csv

    While this is CSV right now (and easy to process similar to our earlier example (https://github.com/kaiwaehner/hivemq-mqtt-tensorflow-kafka-realtime-iot-machine-learning-training-inference/blob/master/python-scripts/autoencoder-anomaly-detection/Sensor-Kafka-Consumer-and-TensorFlow-Model-Training.py), we will use Avro format for the whole pipeline. I.e. the car sensors will already produce Avro messages.

    Is it possible to easily also use a consumer for TensorFlow I/O which deserializes Avro data? We will probably use KafkaAvroSerializer (https://docs.confluent.io/current/schema-registry/serializer-formatter.html). I think this should be no problem as TensorFlow I/O Kafka Plugin probably does not require a specific deserializer?

    Can I come back to you when we have all the details ready?
    (in the MVP, we will just implement the pipeline from car via mqtt and kafka broker to consumer, but in V2 in a few weeks, we want to add TensorFlow I/O for model training...)

    Yong Tang
    @yongtang

    Thanks @kaiwaehner for the update.

    I agree CSV is not a good format for IoT message.

    In TensorFlow I/O we have Avro support for file format. It is not the same as Avro deserializer, though it could be straightforward to add one.

    If you have some sample message I can play with, I probably could add Avro message support easily. (The TenorFlow I/O itself is generic so it does not necessarily tied to Avro. However, it would be really interesting to see tensorflow-io genreating deserialized Avro message out of the box.)

    Let me know if there is any update, and in the meantime, I will take a look at Avro deserializer and see if I could get started early.

    Kai Waehner
    @kaiwaehner
    Sounds great! I will come back to you soon...
    Alexis BRENON
    @AlexisBRENON
    Hi. Is there any benchmark of Parquet vs TFRecord reading time? I am trying to load a Parquet dataset with TF2.0 (to do this kind of benchmark) but cannot achieve it. Why ParquetDataset requires to load only 1 column? How to use it properly with the tf.data.Dataset API ?
    Yong Tang
    @yongtang

    @AlexisBRENON The tf.data is best suited when you already have preprocessed data stored in file (parquet/tfrecord/etc) and is ready to be fed into tf.keras. In that case multiple columns may not help a lot. In other situations, tf.data gives you an iterable and further processing might be limited.

    Though you may have use cases for multiple columns for generic data engineering. In such a case you could still zip multiple columns with parquet. However, I would expect subpar performance as parquet is naturally RowGroup based.

    @AlexisBRENON Can you open an issue on GitHub? The issue could be easily addressed with some simple C++ maneuver I think.

    Alexis BRENON
    @AlexisBRENON
    @yongtang My data are already preprocessed but I have one "features" column and one "label" column in my data. Do you advise me to concat both in a single column (just using indexing to fetch my label) ? I will open the issue but if it's not the recommended way maybe it would be better to not easily allow it ^^.
    Yong Tang
    @yongtang

    @AlexisBRENON (feature, label) could use zip method to form a tuple dataset.

    Under the current implementation there might be some performance penalties with the above mentioned method. But that could be resolved easily as well, just need several PRs. One PR is on tensorflow's core repo:
    tensorflow/tensorflow#31801

    I haven't had enough time to complete this PR recently but I plan to get this one updated and get merged in the next week or so.

    Yong Tang
    @yongtang

    Hi All,

    It's time for our next monthly meeting agin tomorrow. There are several important items that really would like community to help:

    1) Documentation for SIG I/O and linkage to tensorflow.org (sent with previous email)
    2) StructTensor RFC (https://github.com/tensorflow/community/pull/151). The StructTensor is very much related to our work with columnar data formats. (also sent with previous email)

    Would really appreciate community to join and discuss in the upcoming meeting.

    The SIG IO monthly meeting is scheduled for tomorrow 09/12 Thursday, 11:00 AM -12:00 Pacific Time.

    Below is the link to the meeting docs we could build up:

    https://docs.google.com/document/d/1CB51yJxns5WA4Ylv89D-a5qReiGTC0GYum6DU-9nKGo/edit#heading=h.7ck4k2782ggg

    Denis Zubo
    @dzubo
    I'm very interested in using tf.io for audio files with TF 2.0.
    I wonder what is the estimated date for v 0.9 release?
    Yong Tang
    @yongtang
    @dzubo TFIO v0.9 will be released when TF 2.0.0 release version is available. I don't have any direct information, though I assume it is very likely TF 2.0.0 will be released before the TF World next month.
    Denis Zubo
    @dzubo
    @yongtang Thank you!
    Ivelin Ivanov
    @ivelin
    @yongtang I saw the docs PR #498 was accepted. That’s nice. Is there a sandbox to see what the result would look like on the tf docs site? Respectively, how would subpackage docs show up and where we would add additional content. For example I can rework my blog post into a docs format, but I need to get a bit of guidance on location, linking and formatting. Who can we ping for that? Thanks!
    Yong Tang
    @yongtang
    Thanks @ivelin, I think Mark (@MarkDaoust) from Tensorflow doc team might be able to help. Pinged him on GitHub in the opening PR.
    Ivelin Ivanov
    @ivelin
    Yes, I saw the thread. Thank you!
    Billy Lamberta
    @lamberta
    Yong Tang
    @yongtang
    Thanks @lamberta , Mark, and docs team, greatly appreciated 👍🎉🎊🎈!