by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    aranjan1002
    @aranjan1002
    Oh okay
    Gavin Halliday
    @ghalliday
    regress.sh only generates the c++ - it doesn't even compile it by default
    aranjan1002
    @aranjan1002
    Should I give it some argument to make sure it compiles?
    I will probably check the instructions
    Gavin Halliday
    @ghalliday
    Probably -e to speciffy the path of the eclcc, and you will need a copy of eclcc.ini in all current directories you run it from.
    I use it to process a single example (using the -q flag)
    aranjan1002
    @aranjan1002
    Oh okay, I will make sure of that
    Gavin Halliday
    @ghalliday
    (I have a script which calls regress.sh passing the default parameters e.g., -e for the path)
    Anyway, have a good weekend.
    aranjan1002
    @aranjan1002
    Thank you!
    And same to you
    aranjan1002
    @aranjan1002
    I sent an email with some questions
    aranjan1002
    @aranjan1002
    @ghalliday Did you get a chance to read the email I sent yesterday?
    Gavin Halliday
    @ghalliday
    did you see my reply? Sorry for the delay I was creating the example.
    aranjan1002
    @aranjan1002
    Yes, I just replied. Thank you.
    aranjan1002
    @aranjan1002
    Quick question: what is a resourced graph? Does it mean that its result is stored to be used by other graph nodes?
    aranjan1002
    @aranjan1002
    I am trying to rebuild the system and getting the following errors unfortunately
    -DCMAKE_BUILD_TYPE:STRING=Debug ../src
    Any ideas how to resolve this?

    -- Found APRUTIL: /usr/lib/x86_64-linux-gnu/libaprutil-1.so
    -- Could NOT find TBB (missing: TBB_LIBRARIES TBB_INCLUDE_DIR)
    CMake Error at cmake_modules/commonSetup.cmake:703 (message):
    TBB requested but package not found
    Call Stack (most recent call first):
    CMakeLists.txt:101 (include)

    -- Configuring incomplete, errors occurred!
    See also "/home/aranjan/HPCC/Build5/CMakeFiles/CMakeOutput.log".
    See also "/home/aranjan/HPCC/Build5/CMakeFiles/CMakeError.log".

    Gavin Halliday
    @ghalliday
    See email, but there are two ways
    use -DUSE_TBB=0
    or install the package libtbb-dev
    aranjan1002
    @aranjan1002
    Thank you, that worked well
    bchukiat
    @bchukiat
    Possible to provide the confirmation dialogue when click "Reset" button, I accidentally click reset and my pending to commit work gone.
    bchukiat
    @bchukiat
    This message was deleted
    @bchukiat Sorry wrong post
    Rajath Shashidhara
    @rajathshashidhara
    Hello.. I am trying to build HPCC-Platform on Arch Linux.
    make package produces a tar.gz file and a shell script
    running the script extracts the tar.gz in the current directory.
    I have also tried make install.
    Gordon Smith
    @GordonSmith
    What format does Arch Linux normally use deb or rpm or other?
    Nitin Choudhary
    @nitinkgp23
    Hey. I am completely new to open source. Have no idea where to start. Can anyone help?
    Rushikesh Ghatpande
    @rushigerrard
    Hi... I am running a multinode HPCC setup and had some doubts regarding workload distribution.

    Hi guys, I have 1 master node and 2 slave nodes.

    Processes running on Master :
    mydafilesrv ( pid 2252 ) is running ...
    myeclagent ( pid 3533 ) is running ...
    myesp ( pid 5159 ) is running ...
    mysasha ( pid 6796 ) is running ...
    mythor ( pid 21232 ) is running with 2 slave process(es) ...

    Processes running on Slave :
    mydafilesrv ( pid 2096 ) is running ...
    mydali ( pid 3349 ) is running ...
    myeclccserver ( pid 4779 ) is running ...

    Initially, the CPU utilization for all 3 nodes is 0% (100% idle state).

    When I start running a dataGeneration ECL script, the CPU utilization of master node reaches 40-50% however CPU utilization on Slaves continues to remain 0%.
    Even the network utilization on all 3 nodes is similar and the disk utilization for just the master seems very high.

    I am confused. I expected the slave processes to be doing all the data generation and writing it to the disk work. But low-level metrics don't indicate that. Can someone please share some insight on the same?

    Only HPCC related user processes are running on these 3 instances.
    I am consistently seeing eclagent to be the top process on master.
    I am periodically seeing daserver & thorslave to be the top process on slave

    johnholt
    @johnholt
    It sounds like you are running in an HTHOR queue, which is a single node.
    You will need to run in a THOR queue to run on the slaves.
    Rushikesh Ghatpande
    @rushigerrard
    Thanks a lot.
    johnholt
    @johnholt
    Welcome. Once you are running on a THOR queue, you may need to pay attention to how you generate the data such that it is appropriately distributed.
    Rushikesh Ghatpande
    @rushigerrard

    I have been trying to spray data using dfuplus command.

    Here are the details of my HPCC setup.

    Master IP Address :172.31.45.14
    Slave 1 IP Address : 172.31.33.152
    Slave 2 IP Address : 172.31.42.187

    On my master node, I have 30 GB of kmeans related dataset in csv format spread across 30 files.
    The directory is : /mnt/var/lib/HPCCSystems/dataset/kmeans_30GB/

    Using the dfuplus command, I am trying to spray the same data across my slave nodes.

    Here is the command I am running for an individual file.

    $ sudo dfuplus action=spray srcip=172.31.45.14 srcfile=/mnt/var/lib/HPCCSystems/dataset/kmeans_30GB/file1 dstname=kmeans::dataset::file1 dstcluster=mythor server=http://172.31.45.14:8010 format=csv

    Checking for local Dali File Server on port 7100

    Variable spraying from /mnt/var/lib/HPCCSystems/dataset/kmeans_30GB/file1 on 172.31.45.14:7100 to kmeans::dataset::file1
    Submitted WUID D20180423-152513
    D20180423-152513 status: queued
    Failed: No Drop Zone on '172.31.45.14' configured at '/mnt/var/lib/HPCCSystems/dataset/kmeans_30GB/file1'.

    The job fails stating that
    No Drop Zone on '172.31.45.14' configured at '/mnt/var/lib/HPCCSystems/dataset/kmeans_30GB/file1'.

    My ECL Watch console shows a single mydropzone entry of 172.31.33.152.

    Could someone please help me figure out the possible issue?
    johnholt
    @johnholt
    The easiest way to proceed is to upload the file to your landing zone via ECLWatch; and then initiate the Spray via ECL Watch. Do you really need to use a scriptable approach?
    Rushikesh Ghatpande
    @rushigerrard
    Yes, I have 30, 50 gb datasets... I would much prefer scriptable approach than using ECLWatch
    johnholt
    @johnholt
    OK. I’ll find out how to add your system into the landing zone list or to disable the check. Another alternative that would not require a configuration change is to use the SOAP services interface to the ESP which would preserve the scripting but is more complicated.
    johnholt
    @johnholt
    Rushikesh Ghatpande
    @rushigerrard
    Yes. It helped. Thanks.

    Working with flat text file :
    I sprayed a file as a csv with no delimiters and no quotes. The first 2 lines would be posted in the 1st reply.

    I also passed the maxRecordSize as 2147483646 (1 less than INT.MAX) because of the large record size / line.

    The ECL Record structure is :
    RECORD
    STRING field1;
    END;

    I had the following questions :

    1. Def : UNKNOWN char field1
      Why UNKNOWN?
      2.XML :
      <?xml version="1.0"?>

      <Table>
      <Field ecltype="string" label="field1" name="field1" position="0" rawtype="-983036" size="-15" type="string"></Field>
      <filename>file1</filename>
      </Table>

    Why is the rawtype negative?

    My corresponding ECL code is as follows:

    CODE: SELECT ALL
    dataset_name := '~benchmark::file1';
    rs := RECORD
    STRING field1;
    END;

    histogram_dataset2 := DATASET(dataset_name, rs, THOR);

    PATTERN histogramRule1 := PATTERN('(?<=_)[1]');
    RULE userRatingRule1 := histogramRule1;
    matchText1 := {STRING100 out3 := MATCHTEXT(userRatingRule1)};
    userRatingAnswerOne := PARSE(histogram_dataset2, field1, userRatingRule1, matchText1, BEST, MANY, NOCASE);

    ecl run HistogramRating.ecl --target=thor --server=master:8010

    W20180502-145535 failed

    <Result>
    <Exception><Code>4294967295</Code><Source>eclagent</Source><Message>System error: -1: Graph graph1[1], diskread[2]: SLAVE #1 [172.31.15.1:20100]: CFileSerialStream::get read past end of stream, CFileSerialStream::get read past end of stream - handling file: /var/lib/HPCCSystems/hpcc-data/thor/benchmark/file1._1_of_2 - caused by (-1, CFileSerialStream::get read past end of stream)</Message></Exception>
    </Result>

    How to resolve this error?

    johnholt
    @johnholt
    Do you mean no field delimiters or no record delimiters or both?
    If “~benchmark::file1” is the file that you sprayed as CSV, you need read it as CSV in the DATASET(…, CSV(…)) and not DATASET(…, THOR)