These are chat archives for ManageIQ/manageiq/performance

1st
Feb 2019
Himanshu Roy
@hroyrh
Feb 01 09:54
Hi all, I am trying to understand the "parse_inventory" timing inside ems_refresh, what exactly happens during this time?. I am seeing seeing "parse_inventory" timings in the order of 950+ seconds for an OSP provider with around 100 each tenants, users, images, security-groups, 10 networks. Is it normal to take this much time for this volume of object-count
Beni Cherniavsky-Paskin
@cben
Feb 01 10:19

kbrock do you mean collecting ad-hoc metrics? ... I think that was an OpenShift provider feature

agrare:

that just lets you see the native provider metrics but we don't do anything e.g. rollups with ad-hoc metrics

Adam is right. We don't "collect" nor retain these at all. When you view them in UI, we make a request directly into openshift's hawkular/prometheus, pass on results to UI, and forget.
[https://github.com/ManageIQ/manageiq-ui-classic/search?q=get_data]
So irrelevant for C&U optimization discussion.

Himanshu Roy
@hroyrh
Feb 01 10:33
@cben can you take a look at my earlier comment
Peter McGowan
@pemcg
Feb 01 11:59
@hroyrh There’s a comment in the code:
        # legacy refreshers collect inventory during the parse phase
        # new refreshers should override this method to parse inventory
        # TODO: remove this call after all refreshers support retrieving
        # inventory separate from parsing
so my guess is that OSP counts as a legacy refresher and this time is spent actually pulling the inventory
Himanshu Roy
@hroyrh
Feb 01 12:37
@pemcg thanks. and from the looks of it, they plan to separate the collection from parsing in future ( TODO part)?
Adam Grare
@agrare
Feb 01 13:27
@hroyrh @pemcg OSP uses the newer graph refresh which also collects inventory during the parsing stage
It help keep memory down but does make it harder to diagnose where things are slow unfortunately
Adam Grare
@agrare
Feb 01 13:52
950 sec is a really long time though, try with admin credentials so we don't have to loop through each tenant maybe?
Himanshu Roy
@hroyrh
Feb 01 13:57
@agrare yes, and that indeed brought it down to ~550 sec
Nick LaMuro
@NickLaMuro
Feb 01 23:07
TIL learned about some of the sample scripts in the pg gem that are available:
$ bundle exec ruby -r "pg/../../sample/disk_usage_report" -e "report parse_args ['--database', 'vmdb_development']"
======================================================================
Disk usage information for vmdb_development: (2416 relations, 34 MB total)
======================================================================
Top twenty objects by size:
----------------------------------------------------------------------
                            pg_attribute    3712 kB (table)
                             pg_shdepend    1584 kB (table)
             pg_shdepend_reference_index    1504 kB (index)
              pg_shdepend_depender_index    1232 kB (index)
         pg_attribute_relid_attnam_index    1032 kB (index)
                               pg_depend     816 kB (table)
               pg_depend_reference_index     720 kB (index)
                pg_depend_depender_index     704 kB (index)
         pg_attribute_relid_attnum_index     608 kB (index)
                                 pg_proc     576 kB (table)
 ...
----------------------------------------------------------------------
Size per schema:
----------------------------------------------------------------------
          pg_catalog      19 MB
              public      16 MB
  information_schema     344 kB
----------------------------------------------------------------------
(more useful on a non-empty DB, FYI)