Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Anirudh
    @anirudhacharya
    I have made some source changes to the TF codebase and added some functionality from boost::filesystem. How can I add compile flags to bazel so that g++ picks them up while comipling the c++ code

    code

    Build TF with C++ 17 features.

    build:c++17 --cxxopt=-std=c++1z
    build:c++17 --cxxopt=-stdlib=libc++
    build:c++17 --cxxopt=-lboost_system
    build:c++17 --cxxopt=-lboost_filesystem
    build:c++1z --config=c++17

    I have added the above flags in '.bazelrc', but I still end up getting the below errors while trying to compile the codebase
    code
    ERROR: /home/ubuntu/aws-tensorflow/tensorflow/python/BUILD:7990:1: Linking of rule '//tensorflow/python:gen_decode_proto_ops_py_wrappers_cc' failed (Exit 1)
    bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Clibtensorflow_Uframework_Uimport_Ulib_Utensorflow/libtensorflow_framework.so: undefined reference to `boost::system::system_category()'
    bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Clibtensorflow_Uframework_UimportUlib
    Utensorflow/libtensorflow_framework.so: undefined reference to boost::filesystem::detail::create_directory(boost::filesystem::path const&, boost::system::error_code*)' bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Clibtensorflow_Uframework_Uimport_Ulib___Utensorflow/libtensorflow_framework.so: undefined reference toboost::filesystem::detail::file_size(boost::filesystem::path const&, boost::system::error_code*)'
    bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Clibtensorflow_Uframework_Uimport_Ulib___Utensorflow/libtensorflow_framework.so: undefined reference to `boost::system::generic_category()'
    collect2: error: ld returned 1 exit status
    Target //tensorflow/tools/pip_package:build_pip_package failed to build
    INFO: Elapsed time: 1.708s, Critical Path: 0.77s
    INFO: 7 processes: 7 local.
    FAILED: Build did NOT complete successfully
    jeffin sam
    @jeffin143
    P
    @yongtang , Hi I am a gsoc student for Mlpack org ,
    Since this Channel is for io , I would like to clarify some internals about tf.summary.write()
    Do you have the bandwidth , I would be working on building a custom backend to log so that tensorboard could be used to visualise
    Yong Tang
    @yongtang
    @jeffin143 Sure. Let m know if I can be of any help.
    jeffin sam
    @jeffin143
    Hi @yongtang , since the code base is very vast and there are v1 and v2 version, it was difficult to get internals clear, Thanks for helping out
    So I know that tensorflow uses protobuf to write the logs
    So the image would use image protobuf and histogram would use histogram protobuf
    jeffin sam
    @jeffin143
    Now we were able to plot images as well using the same logic
    Just a heads up we were implementing all this in a c++ backend , So here is a snippet
    Summary for Single image
    Now the confusion was , Tensorflow is able to log multiple images in single step : https://raw.githubusercontent.com/tensorflow/tensorboard/master/docs/images/images_multiple.png : Something similar to this
    jeffin sam
    @jeffin143
    Going through the source code , I noticed that we store the byte string in an tensor along with its dimension as a tensor : https://github.com/tensorflow/tensorboard/blob/8277e9f8d50758bb4d2b7a0fce1cab7e736b3dd2/tensorboard/plugins/image/summary_v2.py#L18
    Now my question is how does tensor board interpret this as image and not normal text , since tensor is tensor of string and if you log that it could consider it as a text
    Now my assumption is if you are logging a tensor then you definitely need to use plugin, in metadata right, I am assuming this is false if you are using scaler histo or audio or something since these are defined
    jeffin sam
    @jeffin143
    So I am asumming , if we have to display multiple images then we have to log a tensor with plugin name as https://github.com/tensorflow/tensorboard/blob/8277e9f8d50758bb4d2b7a0fce1cab7e736b3dd2/tensorboard/plugins/image/metadata.py#L27
    I know I have written a lot and it may be confusing a lot but let me know if I have to clear something Out
    I tried displaying multiple images using plugin as images but that doesn't work out , My question is Now my question is how does tensor board interpret this as image and not normal text , since tensor is tensor of string and if you log that it could consider it as a text
    jeffin sam
    @jeffin143
    Also I wanted to know if Tensor of string is stored into summary as mutiple value or summary of Single value tensor ? Although more how is tensor stored in protobuf as I can see the tensor.proto states it is repeated values , I mean how do you set repeated values for tensor proto do you call the set-value() function multiple times but I guess that would just override previous value and it doesn't make sense
    BTW Tensorboard is awesome :)
    jeffin sam
    @jeffin143
    You are a hero
    @yongtang
    I guess , It worked just by taking your name :)
    jeffin sam
    @jeffin143
    int TensorBoardLogger::add_texts(const string &tag, int step, vector<std::string> text) {
        auto *plugin_data = new SummaryMetadata::PluginData();
        plugin_data->set_plugin_name("images");
    
        auto *meta = new SummaryMetadata();
        meta->set_allocated_plugin_data(plugin_data);
    
        auto *tensor = new TensorProto();
        tensor->set_dtype(tensorflow::DataType::DT_STRING);
        for (size_t i = 0; i < text.size(); ++i) {
            tensor->add_string_val(text[i]);
        }
    
    
        auto *summary = new Summary();
        auto *v = summary->add_value();
        v->set_tag(tag);
        v->set_allocated_tensor(tensor);
        v->set_allocated_metadata(meta);
    
    
        return add_event(step, summary);
    }
    Yong Tang
    @yongtang
    @jeffin143 This is very much tied to tensorboard specifically. Have you opened an issue in tensorboard's GitHub?
    jeffin sam
    @jeffin143
    @yongtang since this was a question about internals is it ok to add ??
    Is it ok to Open a issue ?
    Yong Tang
    @yongtang
    It is open source so I don't see an issue to ask. I personally don't have a lot of experience with tensorboard, so not very familiar with how Metadata you mentioned interact with tensorboard's plugin. Think on tensorboard's GitHub you might be able to get more help.
    jeffin sam
    @jeffin143
    Thanks
    @yongtang
    Lucky Srivastava
    @lucky096
    Hi @all can anyone point me on how to read/write from hdfs (specifically mp4 files)?
    Yong Tang
    @yongtang
    @lucky096 Can you open an issue in https://github.com/tensorflow/io and provide more details? Specifically, can you provide a simple example about the data you are trying to read/write to hdf5? (you mentioned mp4 files but I am not sure mp4 is relevant to hdf5).
    Lucky Srivastava
    @lucky096
    @yongtang I don't have to convert mp4 files to hdf5 rather I want to process videos (mp4 files) which are stored in hdfs. Currently my process involves downloading videos to local directory and then process them, I want to avoid this. I can see there is TF_IO for audio, image, parquet, arrow files etc Is there any such similar provision for video files as well? One of the method is to convert video to tfrecord files and then perform read/write on tfrecords directly from hdfs, but this is not a proper solution for what I am looking.
    Yong Tang
    @yongtang

    @lucky096 Ah you mean hdfs. I was incorrectly mistaken by hdf5.

    For HDFS, tensorflow provides a HDFS file system though I am not sure if it is part of the default tensorflow pip wheel package.

    If it is not part of the default pip wheel package, then you will need to rebuilt tensorflow from source.

    @lucky096 We are also in the process of moving most of the file system implementations (HDF5, GCS, S3) from tensorflow to tensorflow-io. This will help as it is easier to maintain in separate repos in tensorflow-io, than a tensorflow repo with a huge code base. Building tensorflow from source takes from several hours to dozens of hours depending on your machine power, so modularization is the way to go.
    @lucky096 The file system modularization will need to wait until modular file system C API in place, which I think likely will be released in 2.3.0.
    Vo Van Nghia
    @vnvo2409
    @lucky096 hdfs is a part of the default pip now
    Ruhua Jiang
    @RuhuaJiang
    hi, is there some instruction on running lint on tfio build? i am getting this lint error: https://github.com/tensorflow/io/pull/995/checks?check_run_id=765938756
    Yong Tang
    @yongtang

    @RuhuaJiang You can use the following to automatically reformat:

    bazel run //tools/lint:lint -- black pyupgrade -- tensorflow_io/bigquery/python/ops/bigquery_api.py

    Once the reformat is complete, you can use git diff to check for difference.

    Ruhua Jiang
    @RuhuaJiang
    nice, thanks @yongtang