Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    bchukiat
    @bchukiat
    This message was deleted
    @bchukiat Sorry wrong post
    Rajath Shashidhara
    @rajathshashidhara
    Hello.. I am trying to build HPCC-Platform on Arch Linux.
    make package produces a tar.gz file and a shell script
    running the script extracts the tar.gz in the current directory.
    I have also tried make install.
    Gordon Smith
    @GordonSmith
    What format does Arch Linux normally use deb or rpm or other?
    Nitin Choudhary
    @nitinkgp23
    Hey. I am completely new to open source. Have no idea where to start. Can anyone help?
    Rushikesh Ghatpande
    @rushigerrard
    Hi... I am running a multinode HPCC setup and had some doubts regarding workload distribution.

    Hi guys, I have 1 master node and 2 slave nodes.

    Processes running on Master :
    mydafilesrv ( pid 2252 ) is running ...
    myeclagent ( pid 3533 ) is running ...
    myesp ( pid 5159 ) is running ...
    mysasha ( pid 6796 ) is running ...
    mythor ( pid 21232 ) is running with 2 slave process(es) ...

    Processes running on Slave :
    mydafilesrv ( pid 2096 ) is running ...
    mydali ( pid 3349 ) is running ...
    myeclccserver ( pid 4779 ) is running ...

    Initially, the CPU utilization for all 3 nodes is 0% (100% idle state).

    When I start running a dataGeneration ECL script, the CPU utilization of master node reaches 40-50% however CPU utilization on Slaves continues to remain 0%.
    Even the network utilization on all 3 nodes is similar and the disk utilization for just the master seems very high.

    I am confused. I expected the slave processes to be doing all the data generation and writing it to the disk work. But low-level metrics don't indicate that. Can someone please share some insight on the same?

    Only HPCC related user processes are running on these 3 instances.
    I am consistently seeing eclagent to be the top process on master.
    I am periodically seeing daserver & thorslave to be the top process on slave

    johnholt
    @johnholt
    It sounds like you are running in an HTHOR queue, which is a single node.
    You will need to run in a THOR queue to run on the slaves.
    Rushikesh Ghatpande
    @rushigerrard
    Thanks a lot.
    johnholt
    @johnholt
    Welcome. Once you are running on a THOR queue, you may need to pay attention to how you generate the data such that it is appropriately distributed.
    Rushikesh Ghatpande
    @rushigerrard

    I have been trying to spray data using dfuplus command.

    Here are the details of my HPCC setup.

    Master IP Address :172.31.45.14
    Slave 1 IP Address : 172.31.33.152
    Slave 2 IP Address : 172.31.42.187

    On my master node, I have 30 GB of kmeans related dataset in csv format spread across 30 files.
    The directory is : /mnt/var/lib/HPCCSystems/dataset/kmeans_30GB/

    Using the dfuplus command, I am trying to spray the same data across my slave nodes.

    Here is the command I am running for an individual file.

    $ sudo dfuplus action=spray srcip=172.31.45.14 srcfile=/mnt/var/lib/HPCCSystems/dataset/kmeans_30GB/file1 dstname=kmeans::dataset::file1 dstcluster=mythor server=http://172.31.45.14:8010 format=csv

    Checking for local Dali File Server on port 7100

    Variable spraying from /mnt/var/lib/HPCCSystems/dataset/kmeans_30GB/file1 on 172.31.45.14:7100 to kmeans::dataset::file1
    Submitted WUID D20180423-152513
    D20180423-152513 status: queued
    Failed: No Drop Zone on '172.31.45.14' configured at '/mnt/var/lib/HPCCSystems/dataset/kmeans_30GB/file1'.

    The job fails stating that
    No Drop Zone on '172.31.45.14' configured at '/mnt/var/lib/HPCCSystems/dataset/kmeans_30GB/file1'.

    My ECL Watch console shows a single mydropzone entry of 172.31.33.152.

    Could someone please help me figure out the possible issue?
    johnholt
    @johnholt
    The easiest way to proceed is to upload the file to your landing zone via ECLWatch; and then initiate the Spray via ECL Watch. Do you really need to use a scriptable approach?
    Rushikesh Ghatpande
    @rushigerrard
    Yes, I have 30, 50 gb datasets... I would much prefer scriptable approach than using ECLWatch
    johnholt
    @johnholt
    OK. I’ll find out how to add your system into the landing zone list or to disable the check. Another alternative that would not require a configuration change is to use the SOAP services interface to the ESP which would preserve the scripting but is more complicated.
    johnholt
    @johnholt
    Rushikesh Ghatpande
    @rushigerrard
    Yes. It helped. Thanks.

    Working with flat text file :
    I sprayed a file as a csv with no delimiters and no quotes. The first 2 lines would be posted in the 1st reply.

    I also passed the maxRecordSize as 2147483646 (1 less than INT.MAX) because of the large record size / line.

    The ECL Record structure is :
    RECORD
    STRING field1;
    END;

    I had the following questions :

    1. Def : UNKNOWN char field1
      Why UNKNOWN?
      2.XML :
      <?xml version="1.0"?>

      <Table>
      <Field ecltype="string" label="field1" name="field1" position="0" rawtype="-983036" size="-15" type="string"></Field>
      <filename>file1</filename>
      </Table>

    Why is the rawtype negative?

    My corresponding ECL code is as follows:

    CODE: SELECT ALL
    dataset_name := '~benchmark::file1';
    rs := RECORD
    STRING field1;
    END;

    histogram_dataset2 := DATASET(dataset_name, rs, THOR);

    PATTERN histogramRule1 := PATTERN('(?<=_)[1]');
    RULE userRatingRule1 := histogramRule1;
    matchText1 := {STRING100 out3 := MATCHTEXT(userRatingRule1)};
    userRatingAnswerOne := PARSE(histogram_dataset2, field1, userRatingRule1, matchText1, BEST, MANY, NOCASE);

    ecl run HistogramRating.ecl --target=thor --server=master:8010

    W20180502-145535 failed

    <Result>
    <Exception><Code>4294967295</Code><Source>eclagent</Source><Message>System error: -1: Graph graph1[1], diskread[2]: SLAVE #1 [172.31.15.1:20100]: CFileSerialStream::get read past end of stream, CFileSerialStream::get read past end of stream - handling file: /var/lib/HPCCSystems/hpcc-data/thor/benchmark/file1._1_of_2 - caused by (-1, CFileSerialStream::get read past end of stream)</Message></Exception>
    </Result>

    How to resolve this error?

    johnholt
    @johnholt
    Do you mean no field delimiters or no record delimiters or both?
    If “~benchmark::file1” is the file that you sprayed as CSV, you need read it as CSV in the DATASET(…, CSV(…)) and not DATASET(…, THOR)
    RicardoFernandez-UY
    @RicardoFernandez-UY
    Hi, I'm trying to install HPCCSystems on ubuntu 20.04, but it fails because Dali uses port 7070 by default, and it is already in use by other application. I tried changing the port in the environment.xml and restarting the system, but it insists in connecting to port 7070, and it show the corresponding error in the dali log. Does anyone know how to solve this? Thank you!
    Here is the error: 0000000F USR 2021-11-24 13:07:03.776 3866411 3866411 "DASERVER[0] starting - listening to port 7070"
    00000010 OPR 2021-11-24 13:07:03.778 3866411 3866411 "ERROR: -7: daserver.cpp(817) : Exception : port in use
    RicardoFernandez-UY
    @RicardoFernandez-UY
    Anybody?
    Gordon Smith
    @GordonSmith
    @RicardoFernandez-UY - if you checkout the "Using Configuration Manager" online documentation: https://hpccsystems.com/training/documentation/installation-and-administration it should show you the way....
    RicardoFernandez-UY
    @RicardoFernandez-UY
    @GordonSmith thank you Gordon!! I believe there is no documentation I haven't tried yet. I've done the configuration steps manually and with the GUI; I've checked that the port has indeed changed in the environment.xml file, restarted the server, cleaned the logs (just in case ;-)), uninstalled, re-installed the whole thing many times, and when Dali starts it goes looking for the default 7070 port and it fails with a timeout, because the port is already in use by other app. Just as an experiment I've done the installation following the same steps as per the manual in a vanilla server (without anything else installed) and it works like a charm. Of course in ther latter it uses the 7070 port which in this case is free.
    Gordon Smith
    @GordonSmith
    I suggest you post your query here: https://hpccsystems.com/bb/ things will be quiet until after folks get back from thanksgiving...
    RicardoFernandez-UY
    @RicardoFernandez-UY
    @GordonSmith Thanks again!!! I've tried. It's a little bit embarrassing, but I forgot my password and the reset password system of the forum is not working (at least not for me). I've tried it ten times, and never received the email with instructions. I've checked the spam folder also, just in case, and nothing. I'm working this problem with someone else and I hope I can solve it soon so I can post the issue where you suggested. It seems the universe is temporarily misaligned for my relationship with HPCCSystems :-) . Warm regards!!
    Gordon Smith
    @GordonSmith
    @RicardoFernandez-UY FYI here is the response I got internally:
    "you cannot via configuration at the moment.
    You can if it is started via the command line with --port <port>"
    I would suggest opening an issue here: https://track.hpccsystems.com/ if this is a serious issue for you?
    RicardoFernandez-UY
    @RicardoFernandez-UY
    @GordonSmith Thank you very much!!! It will become an issue whenever I have to deal with installations with multiple applications already installed, as there is always the possibility of that port being already in use. So, I'll follow your advice and open an issue. However, it is good news that there is a manual workaround for the problem. I'll try that too! I guess in this case I will have to start the components one by one in the correct order to end with the whole platform up and running. Am I right?
    Gordon Smith
    @GordonSmith
    Given Dali is the first component that needs to be started, you could try starting it manually and then use the usual start up service for the rest?
    RicardoFernandez-UY
    @RicardoFernandez-UY
    It sounds quite reasonable. I'll try that and let you know the results. I have no access to the servers until tomorrow.
    RicardoFernandez-UY
    @RicardoFernandez-UY

    @GordonSmith
    Well. This one is giving quite a fight ;-).
    I've tried:

    /opt/HPCCSystems/sbin/hpcc-run.sh -c dali@mydali.service --port 7047 start
    hpcc-run: unrecognized option '--port'
    Failure to parse commandline.

    Then:
    systemctl start dali@mydali.service --port 7047
    systemctl: unrecognized option '--port'

    Then:
    ./dafilesrv --port 7047
    And in the log files I got the usual
    0000000F USR 2021-11-30 21:26:48.958 856557 856557 "DASERVER[0] starting - listening to port 7070"
    00000010 OPR 2021-11-30 21:26:48.960 856557 856557 "ERROR: -7: daserver.cpp(817) : Exception : port in use

    And finally:

    ./daserver--port 7047
    ...
    0000000C USR 2021-11-30 21:46:23.082 858207 858207 "DASERVER[0] starting - listening to port 7070"
    0000000D OPR 2021-11-30 21:46:23.083 858207 858207 "ERROR: -7: daserver.cpp(817) : Exception : port in use
    ...

    I'm not giving up, but I might need some context for where to place the --port option.
    Perhaps the whole command :-)
    Thank you!! And thank you for your patience too!!!

    Rodrigo
    @rpastrana
    joining
    Stuart Ort
    @stuartort
    Welcome @rpastrana
    kenrowland
    @kenrowland
    Joined
    Greg Panagiotatos
    @g-pan
    Howdy!
    Michael Gardner
    @Michael-Gardner
    test
    Anthony Fishbeck
    @afishbeck
    Joined
    David de Hilster
    @dehilsterlexis
    Someone needs to update the HPCC icon
    Gavin Halliday
    @ghalliday
    BTW I'm trying to chase down why jira is down again...
    Stuart Ort
    @stuartort
    I can log into Jira, but when trying to accept and issue and add a comment, the comment section seems to be disabled for me. Anyone else that problem? Actually I can't add a comment anywhere in jira