Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Keqiu Hu
    @oliverhu
    @vnvo2409 is there any temp fix for the CICD issue ?
    Vo Van Nghia
    @vnvo2409
    I think libtensorflow_io_plugins.so is having some problems, could you delete it inside bazel-bin and try again ?
    Keqiu Hu
    @oliverhu
    it doesn't exist at all :(
    (p3) pi@pig:~/tf/io$ find . -name "libtensorflow_io*"
    (p3) pi@pig:~/tf/io$
    Vo Van Nghia
    @vnvo2409
    it is inside bazel-bin/tensorflow_io/core/python/ops/libtensorflow_io_plugins.so
    and your error message suggests that you are using the wrong bazel-bin I think :/
    Keqiu Hu
    @oliverhu
    I don't have that .so file at all in the bazel-bin folder
    (p3) pi@pig:~/tf/io$ ls
    bazel-bin  bazel-out       BUILD.bazel   CONTRIBUTING.md  LICENSE    RELEASE.md  setup.py        tensorflow_io  third_party  WORKSPACE
    bazel-io   bazel-testlogs  configure.sh  docs             README.md  R-package   STYLE_GUIDE.md  tests          tools
    oh.. nvm
    it is a simlink
    so find doesn't work
    again.. we should update the wiki page
    Vo Van Nghia
    @vnvo2409
    Yeah I see. What do you think @yongtang ?
    Keqiu Hu
    @oliverhu
    this works
    TFIO_DATAPATH=/home/pi/.cache/bazel/_bazel_pi/be6ac8eba0db45fb771509e53438aa5b/execroot/org_tensorflow_io/bazel-out/k8-fastbuild/bin/ python3 -m pytest -s -v tests/test_pcap_eager.py
    Vo Van Nghia
    @vnvo2409
    It is the real bazel-bin path. I am not sure why bazel-bin path does not work
    but usually, I call export TFIO_DATAPATH=$(bazel info bazel-bin ) or export TFIO_DATAPATH=$(bazel info bazel-bin -c opt )
    Keqiu Hu
    @oliverhu
    ^that is definitely better
    Keqiu Hu
    @oliverhu
    actually.. i misread the error message @vnvo2409
    E       NotImplementedError: unable to open file: /home/pi/tf/io/tensorflow_io/core/python/ops/__init__.py, from paths: ['/home/pi/tf/io/tensorflow_io/core/python/ops/libtensorflow_io.so', 'bazel-bin/tensorflow_io/core/python/ops/libtensorflow_io.so']
    E       caused by: ['/home/pi/tf/io/tensorflow_io/core/python/ops/libtensorflow_io.so: cannot open shared object file: No such file or directory', 'bazel-bin/tensorflow_io/core/python/ops/libtensorflow_io.so: undefined symbol: _ZN4absl14lts_2020_02_2520raw_logging_internal21internal_log_functionB5cxx11E']
    the caused by has 2 statements
    it does point to the right location of tensorflow-io bazel output
    but got bazel-bin/tensorflow_io/core/python/ops/libtensorflow_io.so: undefined symbol: _ZN4absl14lts_2020_02_2520raw_logging_internal21internal_log_functionB5cxx11E
    Vo Van Nghia
    @vnvo2409
    Did you recompile everything after install tf-nightly ?
    Keqiu Hu
    @oliverhu
    yeah i did
    Vo Van Nghia
    @vnvo2409
    how about bazel clean --expunge and retry one more time :/ ?
    Keqiu Hu
    @oliverhu
    perhaps it is becauuse i didn't rerun ./configure.sh
    rebuilding
    Keqiu Hu
    @oliverhu
    same error
    (p3) pi@pig:~/tf/io$ TFIO_DATAPATH=bazel-bin python3 -m pytest -s -v tests/test_pcap.py
    ================================================================== test session starts ===================================================================
    platform linux -- Python 3.8.5, pytest-6.2.2, py-1.10.0, pluggy-0.13.1 -- /home/pi/p3/bin/python3
    cachedir: .pytest_cache
    rootdir: /home/pi/tf/io
    collecting ... 2021-03-22 12:34:48.901457: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
    collected 1 item
    
    tests/test_pcap.py::test_pcap_input Testing PcapDataset
    FAILED
    
    ======================================================================== FAILURES ========================================================================
    ____________________________________________________________________ test_pcap_input _____________________________________________________________________
    
        def test_pcap_input():
            """test_pcap_input
          """
            print("Testing PcapDataset")
            pcap_filename = os.path.join(
                os.path.dirname(os.path.abspath(__file__)), "test_pcap", "http.pcap"
            )
            file_url = "file://" + pcap_filename
    
    >       dataset = tfio.IODataset.from_pcap(file_url, capacity=5).batch(1)
    
    tests/test_pcap.py:34:
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
    tensorflow_io/core/python/ops/io_dataset.py:309: in from_pcap
        return pcap_dataset_ops.PcapIODataset(filename, internal=True, **kwargs)
    tensorflow_io/core/python/ops/pcap_dataset_ops.py:36: in __init__
        resource = core_ops.io_pcap_readable_init(
    tensorflow_io/core/python/ops/__init__.py:88: in __getattr__
        return getattr(self._load(), attrb)
    tensorflow_io/core/python/ops/__init__.py:84: in _load
        self._mod = _load_library(self._library)
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
    
    filename = '/home/pi/tf/io/tensorflow_io/core/python/ops/__init__.py', lib = 'op'
    
        def _load_library(filename, lib="op"):
            """_load_library"""
            f = inspect.getfile(sys._getframe(1))  # pylint: disable=protected-access
    
            # Construct filename
            f = os.path.join(os.path.dirname(f), filename)
            filenames = [f]
    
            # Add datapath to load if en var is set, used for running tests where shared
            # libraries are built in a different path
            datapath = os.environ.get("TFIO_DATAPATH")
            if datapath is not None:
                # Build filename from:
                # `datapath` + `tensorflow_io` + `package_name` + `relpath_to_library`
                rootpath = os.path.dirname(sys.modules["tensorflow_io"].__file__)
                filename = sys.modules[__name__].__file__
                f = os.path.join(
                    datapath,
                    "tensorflow_io",
                    os.path.relpath(os.path.dirname(filename), rootpath),
                    os.path.relpath(f, os.path.dirname(filename)),
                )
                filenames.append(f)
            # Function to load the library, return True if file system library is loaded
            if lib == "op":
                load_fn = tf.load_op_library
            elif lib == "dependency":
                load_fn = lambda f: ctypes.CDLL(f, mode=ctypes.RTLD_GLOBAL)
            elif lib == "fs":
                load_fn = lambda f: tf.experimental.register_filesystem_plugin(f) is None
            else:
                load_fn = lambda f: tf.compat.v1.load_file_system_library(f) is None
    
            # Try to load all paths for file, fail if none succeed
            errs = []
            for f in filenames:
                try:
                    l = load_fn(f)
                    if l is not None:
                        return l
                except (tf.errors.NotFoundError, OSError) as e:
                    errs.append(str(e))
    >       raise NotImplementedError(
                "unable to open file: "
                + "{}, from paths: {}\ncaused by: {}".format(filename, filename
    let me do the --expunge nuclear version
    Jason Zaman
    @perfinion
    is that your machine or a CI worker? and do you have caching enabled? (either remote or --disk_cache=?)
    Keqiu Hu
    @oliverhu
    this is from my local machine
    no caching
    ci has a similar but different error
    Yong Tang
    @yongtang
    @oliverhu The missing symbol issue is likely caused by some API change on tf-nightly. I will take a look.
    Keqiu Hu
    @oliverhu
    thanks! @yongtang
    yeah, it still breaks with the same error after expunge
    Yong Tang
    @yongtang
    I am able to reproduce the issue on my local environment. I tends to believe the issue is that tensorflow/core/platform/cloud/gcs_file_system.cc was not picked up when tf-nightly was packaged. Though may need to take a further look to validate it.
    Yong Tang
    @yongtang
    @oliverhu @vnvo2409 Added a PR tensorflow/io#1336 for the build fix.
    Keqiu Hu
    @oliverhu
    thanks!
    gonna +1..
    Keqiu Hu
    @oliverhu
    @yongtang it works now. validated
    Keqiu Hu
    @oliverhu
    @yongtang thoughts on tensorflow/io#1334 ? i think it was a mistake to call it a columnar.py
    Yong Tang
    @yongtang
    @oliverhu columnar is to categorize avro into the same category as parquet/feather/csv, as they essentially are column data. The intention is to limit the number of top level python module as the current number is growing too big.
    Keqiu Hu
    @oliverhu
    @yongtang avro/csv are not columnar
    we can group them into a row based. and i don't think there are many row based storage format nowadays
    Keqiu Hu
    @oliverhu
    i'm having challenges building tf/io again.. if i use tf 2.4.1, I got this ./tensorflow_io/core/plugins/gs/expiring_lru_cache.h:27:10: fatal error: tensorflow/c/env.h: No such file or directory; if i use tf-nightly (2.6.0), it complains tf doesn't have sysconfig property
    ok, it seems we need tf 2.5.0rc
    Vignesh Kothapalli
    @kvignesh1420
    @oliverhu yes, you would need tf 2.5.0rc0 for building tfio. In case you encounter build issues, try doing a bazel clean --expunge, followed by ./configure.sh to remove old symbols and try building again.