zip as in Python) then printing the two records.
open_func: Union[function[gzFile,str,str], function[File,str,str]] = open
open_func = open
open_mode = 'w'
# Define open function and open mode to the correct setting, depending on the fact
# that the output file is gzipped or not.
if fastq_filename.endswith('.gz'):
open_func = gzip.open
open_mode = 'wb6'
else:
open_func = open
open_mode = 'w'
with open_func(fastq_filename, open_mode) as fastq_fh:open_func behave like I want (gzip.open and open are different types)?
def process_file(fastq_fh):
...
if fastq_filename.endswith('.gz'):
with gzip.open(fastq_filename, 'wb6') as fastq_fh: process_file(fastq_fh)
else:
with open(fastq_filename, 'w') as fastq_fh: process_file(fastq_fh)process_file is a generic function -- I believe the APIs for normal files and for gzip'd files should be basically identical
pigz (parallel gzip). I was using the buildin gzip functionality (with gzip level 6) before, but then my script takes 35 minutes to run on that FASTQ file. After just writing to a file or stdout, it takes less than 8 minutes. When compressing that plain text output with gzip -6 it takes 15 minutes to compress (same time them when running the script and piping the output directly to gzip -6. When piping the plain text output (uncompressed modifed FASTQ file) of my script to pigz -t 4 -6, it takes less than8 minutes too. So buildin gzip compression seems to be quite slow. Being able to write directly to stdin of pigz (or other tools) would be great. In AWK output redirection to commands exists: https://www.gnu.org/software/gawk/manual/gawk.html#Redirection
- Process the file using blocks (https://docs.seq-lang.org/cookbook.html#parallel-fastq-processing)
- For each block, create an empty list the size of the block (e.g.
[s''] * N)
CC=clang CXX=clang++ ./deps.sh to run it. You can also pass it an argument to use >1 core (e.g. deps.sh 2 for 2 cores)
deps folder path to cmake: cd build && cmake .. -DSEQ_DEP=/path/to/deps
cmake .. -DSEQ_DEP=../deps -DLLVM_DIR=llvm-config --cmakedir -DHTS_LIB=htslib-1.9/libhts.a -DGC_LIB=gc-8.0.4/release/lib/libgc.a CPATH=gc-8.0.4/release/include:htslib-1.9 cmake --build .
-- Dependency directory: ../deps
-- Found LLVM 6.0.0
-- Using LLVMConfig.cmake in: /Users/yusufcakan/Dropbox/6_Project_Files/1_Personal_Projects/FX/fx_seq_orig/deps/lib/cmake/llvm
-- Found zlib: ../deps/lib/libz.a
-- Found bdwgc: ../deps/lib/libgc.a
-- Found OCaml: /Users/yusufcakan/Dropbox/6_Project_Files/1_Personal_Projects/FX/fx_seq_orig/deps/lib/ocaml
-- Found Menhir: ../deps/share/menhir
-- Found OpenMP: ../deps/lib/libomp.dylib
CMake Deprecation Warning at CMakeLists.txt:1 (cmake_minimum_required):
Compatibility with CMake < 2.8.12 will be removed from a future version of
CMake.
Update the VERSION argument <min> value or use a ...<max> suffix to tell
CMake that the project does not need compatibility with older versions.
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/yusufcakan/Dropbox/6_Project_Files/1_Personal_Projects/FX/fx_seq_orig/build/googletest-download
[ 11%] Performing update step for 'googletest'
[ 22%] No patch step for 'googletest'
[ 33%] No configure step for 'googletest'
[ 44%] No build step for 'googletest'
[ 55%] No install step for 'googletest'
[ 66%] No test step for 'googletest'
[ 77%] Completed 'googletest'
[100%] Built target googletest
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/yusufcakan/Dropbox/6_Project_Files/1_Personal_Projects/FX/fx_seq_orig/build
cmake --build build