Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Summary for Single image
    Now the confusion was , Tensorflow is able to log multiple images in single step : https://raw.githubusercontent.com/tensorflow/tensorboard/master/docs/images/images_multiple.png : Something similar to this
    jeffin sam
    @jeffin143
    Going through the source code , I noticed that we store the byte string in an tensor along with its dimension as a tensor : https://github.com/tensorflow/tensorboard/blob/8277e9f8d50758bb4d2b7a0fce1cab7e736b3dd2/tensorboard/plugins/image/summary_v2.py#L18
    Now my question is how does tensor board interpret this as image and not normal text , since tensor is tensor of string and if you log that it could consider it as a text
    Now my assumption is if you are logging a tensor then you definitely need to use plugin, in metadata right, I am assuming this is false if you are using scaler histo or audio or something since these are defined
    jeffin sam
    @jeffin143
    So I am asumming , if we have to display multiple images then we have to log a tensor with plugin name as https://github.com/tensorflow/tensorboard/blob/8277e9f8d50758bb4d2b7a0fce1cab7e736b3dd2/tensorboard/plugins/image/metadata.py#L27
    I know I have written a lot and it may be confusing a lot but let me know if I have to clear something Out
    I tried displaying multiple images using plugin as images but that doesn't work out , My question is Now my question is how does tensor board interpret this as image and not normal text , since tensor is tensor of string and if you log that it could consider it as a text
    jeffin sam
    @jeffin143
    Also I wanted to know if Tensor of string is stored into summary as mutiple value or summary of Single value tensor ? Although more how is tensor stored in protobuf as I can see the tensor.proto states it is repeated values , I mean how do you set repeated values for tensor proto do you call the set-value() function multiple times but I guess that would just override previous value and it doesn't make sense
    BTW Tensorboard is awesome :)
    jeffin sam
    @jeffin143
    You are a hero
    @yongtang
    I guess , It worked just by taking your name :)
    jeffin sam
    @jeffin143
    int TensorBoardLogger::add_texts(const string &tag, int step, vector<std::string> text) {
        auto *plugin_data = new SummaryMetadata::PluginData();
        plugin_data->set_plugin_name("images");
    
        auto *meta = new SummaryMetadata();
        meta->set_allocated_plugin_data(plugin_data);
    
        auto *tensor = new TensorProto();
        tensor->set_dtype(tensorflow::DataType::DT_STRING);
        for (size_t i = 0; i < text.size(); ++i) {
            tensor->add_string_val(text[i]);
        }
    
    
        auto *summary = new Summary();
        auto *v = summary->add_value();
        v->set_tag(tag);
        v->set_allocated_tensor(tensor);
        v->set_allocated_metadata(meta);
    
    
        return add_event(step, summary);
    }
    Yong Tang
    @yongtang
    @jeffin143 This is very much tied to tensorboard specifically. Have you opened an issue in tensorboard's GitHub?
    jeffin sam
    @jeffin143
    @yongtang since this was a question about internals is it ok to add ??
    Is it ok to Open a issue ?
    Yong Tang
    @yongtang
    It is open source so I don't see an issue to ask. I personally don't have a lot of experience with tensorboard, so not very familiar with how Metadata you mentioned interact with tensorboard's plugin. Think on tensorboard's GitHub you might be able to get more help.
    jeffin sam
    @jeffin143
    Thanks
    @yongtang
    Lucky Srivastava
    @lucky096
    Hi @all can anyone point me on how to read/write from hdfs (specifically mp4 files)?
    Yong Tang
    @yongtang
    @lucky096 Can you open an issue in https://github.com/tensorflow/io and provide more details? Specifically, can you provide a simple example about the data you are trying to read/write to hdf5? (you mentioned mp4 files but I am not sure mp4 is relevant to hdf5).
    Lucky Srivastava
    @lucky096
    @yongtang I don't have to convert mp4 files to hdf5 rather I want to process videos (mp4 files) which are stored in hdfs. Currently my process involves downloading videos to local directory and then process them, I want to avoid this. I can see there is TF_IO for audio, image, parquet, arrow files etc Is there any such similar provision for video files as well? One of the method is to convert video to tfrecord files and then perform read/write on tfrecords directly from hdfs, but this is not a proper solution for what I am looking.
    Yong Tang
    @yongtang

    @lucky096 Ah you mean hdfs. I was incorrectly mistaken by hdf5.

    For HDFS, tensorflow provides a HDFS file system though I am not sure if it is part of the default tensorflow pip wheel package.

    If it is not part of the default pip wheel package, then you will need to rebuilt tensorflow from source.

    @lucky096 We are also in the process of moving most of the file system implementations (HDF5, GCS, S3) from tensorflow to tensorflow-io. This will help as it is easier to maintain in separate repos in tensorflow-io, than a tensorflow repo with a huge code base. Building tensorflow from source takes from several hours to dozens of hours depending on your machine power, so modularization is the way to go.
    @lucky096 The file system modularization will need to wait until modular file system C API in place, which I think likely will be released in 2.3.0.
    Vo Van Nghia
    @vnvo2409
    @lucky096 hdfs is a part of the default pip now
    Ruhua Jiang
    @RuhuaJiang
    hi, is there some instruction on running lint on tfio build? i am getting this lint error: https://github.com/tensorflow/io/pull/995/checks?check_run_id=765938756
    Yong Tang
    @yongtang

    @RuhuaJiang You can use the following to automatically reformat:

    bazel run //tools/lint:lint -- black pyupgrade -- tensorflow_io/bigquery/python/ops/bigquery_api.py

    Once the reformat is complete, you can use git diff to check for difference.

    Ruhua Jiang
    @RuhuaJiang
    nice, thanks @yongtang
    Ruhua Jiang
    @RuhuaJiang
    i am getting following error when running lint
    ERROR: /io/tools/lint/BUILD:18:1: //tools/lint:pyupgrade_py depends on @lint_dependencies_pypi__tokenize_rt_4_0_0//:pkg in repository @lint_dependencies_pypi__tokenize_rt_4_0_0 which failed to fetch. no such package '@lint_dependencies_pypi__tokenize_rt_4_0_0//': whl_library failed: (src/main/tools/process-wrapper-legacy.cc:58: "execvp(python3, ...)": No such file or directory ) ERROR: Analysis of target '//tools/lint:check' failed; build aborted: no such package '@lint_dependencies_pypi__tokenize_rt_4_0_0//': whl_library failed: (src/main/tools/process-wrapper-legacy.cc:58: "execvp(python3, ...)": No such file or directory )
    tried on v1.13.0 instead of master branch, same error
    even with simple bazel run //tools/lint:check
    Yong Tang
    @yongtang
    @RuhuaJiang Can you specify your OS version and python version? You can open an issue in https://github.com/tensorflow/io/issues so that community might also benefit.
    Ruhua Jiang
    @RuhuaJiang

    `FROM centos:7

    ENV TF_VERSION 2.2.0
    ENV BZL_VERSION 3.0.0
    ENV BZL_URL https://github.com/bazelbuild/bazel/releases/download
    ENV BZL_SCRIPT bazel-${BZL_VERSION}-installer-linux-x86_64.sh

    RUN yum install -y centos-release-scl
    RUN yum install -y curl devtoolset-9 git rh-python36

    RUN scl enable rh-python36 devtoolset-9 'python3 -m pip install -U pip'
    RUN scl enable rh-python36 devtoolset-9 'python3 -m pip install -U pytest'
    RUN scl enable rh-python36 devtoolset-9 'python3 -m pip install -U tensorflow==${TF_VERSION}'

    RUN curl -sSOL ${BZL_URL}/${BZL_VERSION}/${BZL_SCRIPT}
    RUN bash -x -e ${BZL_SCRIPT}

    RUN git clone https://github.com/RuhuaJiang/io.git
    WORKDIR /io

    COPY .sh /io/
    RUN chmod +x
    .sh`

    this is the docker file i used. basically centos7
    will file an issue
    Ruhua Jiang
    @RuhuaJiang
    issue created, thanks @yongtang tensorflow/io#999
    Alexander Abraham
    @RealAAbraham
    Hi everybody! :) I'm a Pythonista who's interested in AI since yesterday, I was specifically looking for some resources on how to create my own dataset of say dating chats, and tensorflow terminology and maybe good explanations of these terms.
    If I am on the wrong channel here, Id appreciate it if you could point me to right one :)
    Yong Tang
    @yongtang
    @RealAAbraham A starter would be https://www.tensorflow.org/guide/data . Once you can build a couple of simple examples and familiar with basic operations such as map, filter, etc. you can try build from more complicated or additional data sources. For example, depending on your data source framework (e.g, Kafka? BigQuery?), you might be able to pull your dating chats into TF graph in a continuous streaming fashion.
    Alexander Abraham
    @RealAAbraham
    @yongtang Thanks! I'll read that for sure. But I was also wondering if anyone of you can point me to resources that will explain everything to do with Machine Learning specifically for Tensorflow for a rookie in this field, like myself. :)
    Yong Tang
    @yongtang
    @RealAAbraham Not sure if anyone can help here but I think some online tutorials or stackoverflow might have some additional resources.
    Alexander Abraham
    @RealAAbraham
    @yongtang Thank you!🙏 I'll see what I can find.
    Adam Szałucha
    @aszalucha
    Hi, TF 1.x was supporting Bigtable through tf.contrib. Is there any support for it in TF 2.x?