Hello! I've run into some intermittent issues reading from a handful of ~18MB files I've combined repeatedly with tdb_cons_add(). Have anyone seen any behavior that resembles this:
==15444== Invalid read of size 8
==15444== at 0x4E3FD52: read_bits (tdb_bits.h:14)
==15444== by 0x4E3FD52: read_bits64 (tdb_bits.h:38)
==15444== by 0x4E3FD52: huff_decode_value (tdb_huffman.h:72)
==15444== by 0x4E3FD52: _tdb_cursor_next_batch (tdb_decode.c:282)
==15444== by 0x935C57: tdb_cursor_next (traildb.h:304)
==15444== by 0x935C57: _cgo_4805fbb2d53a_Cfunc_tdb_cursor_next (cgo-gcc-prolog:222)
==15444== by 0x46565F: runtime.asmcgocall (/usr/local/bin/go/src/runtime/asm_amd64.s:688)
==15444== by 0xC4200928FF: ???
==15444== by 0xB07CE87: ???
==15444== by 0x460D81: runtime.(*mcache).nextFree.func1 (/usr/local/bin/go/src/runtime/malloc.go:556)
==15444== by 0xC4201AABFF: ???
==15444== by 0x43BB8F: ??? (/usr/local/bin/go/src/runtime/proc.go:1092)
==15444== Address 0xe323ff9 is in a r-- mapped file /home/vagrant/app_files2/0157e8982def92b71fcc767d568e57883b86dba4298b66c2468127de0ef9c8cc segment
==15444==
fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0xe324000 pc=0x4e3fd52]
runtime stack:
runtime.throw(0xb18c4c, 0x2a)
/usr/local/bin/go/src/runtime/panic.go:616 +0x81
runtime.sigpanic()
/usr/local/bin/go/src/runtime/signal_unix.go:372 +0x28e
goroutine 12 [syscall]:
runtime.cgocall(0x935c00, 0xc42006ca10, ==15444== Use of uninitialised value of size 8
==15444== at 0x438673: runtime.printhex (/usr/local/bin/go/src/runtime/print.go:219)
==15444== by 0x45AA68: runtime.gentraceback (/usr/local/bin/go/src/runtime/traceback.go:406)
==15444== by 0x45C4F8: runtime.traceback1 (/usr/local/bin/go/src/runtime/traceback.go:684)
==15444== by 0x45C371: runtime.traceback (/usr/local/bin/go/src/runtime/traceback.go:645)
==15444== by 0x45CF56: runtime.tracebackothers (/usr/local/bin/go/src/runtime/traceback.go:816)
==15444== by 0x437B54: runtime.dopanic_m (/usr/local/bin/go/src/runtime/panic.go:736)
==15444== by 0x46271B: runtime.dopanic.func1 (/usr/local/bin/go/src/runtime/panic.go:598)
==15444== by 0x437479: runtime.dopanic (/usr/local/bin/go/src/runtime/panic.go:597)
==15444== by 0x437550: runtime.throw (/usr/local/bin/go/src/runtime/panic.go:616)
==15444== by 0x44CD7D: runtime.sigpanic (/usr/local/bin/go/src/runtime/signal_unix.go:372)
==15444== by 0x4E3FD51: read_bits (tdb_bits.h:13)
==15444== by 0x4E3FD51: read_bits64 (tdb_bits.h:38)
==15444== by 0x4E3FD51: huff_decode_value (tdb_huffman.h:72)
==15444== by 0x4E3FD51: _tdb_cursor_next_batch (tdb_decode.c:282)
==15444== by 0x935C57: tdb_cursor_next (traildb.h:304)
==15444== by 0x935C57: _cgo_4805fbb2d53a_Cfunc_tdb_cursor_next (cgo-gcc-prolog:222)
==15444==
==15444== Conditional jump or move depends on uninitialised value(s)
==15444== at 0x438685: runtime.printhex (/usr/local/bin/go/src/runtime/print.go:220)
==15444== by 0x45AA68: runtime.gentraceback (/usr/local/bin/go/src/runtime/traceback.go:406)
==15444== by 0x45C4F8: runtime.traceback1 (/usr/local/bin/go/src/runtime/traceback.go:684)
==15444== by 0x45C371: runtime.traceback (/usr/local/bin/go/src/runtime/traceback.go:645)
==15444== by 0x45CF56: runtime.tracebackothers (/usr/local/bin/go/src/runtime/traceback.go:816)
==15444== by 0x437B54: runtime.dopanic_m (/usr/local/bin/go/src/runtime/panic.go:736)
==15444== by 0x46271B: runtime.dopanic.func1 (/usr/local/bin/go/src/runtime/panic.go:598)
==15444== by 0x437479: runtime.dopanic (/usr/local/bin/go/src/runtime/panic.go:597)
==15444== by 0x437550: runtime.throw (/usr/local/bin/go/src/runtime/panic.go:616)
==15444== by 0x44CD7D: runtime.sigpanic (/usr/local/bin/go/src/runtime/signal_unix.go:372)
==15444== by 0x4E3FD51: read_bits (tdb_bits.h:13)
==15444== by 0x4E3FD51: read_bits64 (tdb_bits.h:38)
==15444== by 0x4E3FD51: huff_decode_value (tdb_huffman.h:72)
I'm using the traildb-go bindings.
hello! i saw --threads
on the CLI help and am wondering what is made parallel?
I know that tdb handles aren't thread safe but am thinking of ways to build something parallel and ordered on top of multiple tdb files and cursors within a single process. possibly a batching multi-multicursor? could that work, or is there a good chance that i'd run into other issues that i'm not thinking of? thanks!
tdb_multi_cursor_new
but calling a version of tdb_multi_cursor_reset
that threads the initial calls to tdb_cursor_peek
might actually do the trick (since many cursors will be exhausted right away). i'll have to see how much time is spent in tdb_multi_cursor_new
vs tdb_multi_cursor_next
tdb_multi_cursor_reset
when it's created
_tdb_cursor_next_batch
is called again
TDB_ERR_IO_OPEN