How does APEX get data from HPX?

[leonfse]

Hello,

For my Master Thesis I look into the Performance Analysis of HPX applications. In a best-case scenario, I would like to create an HPX-Adapter for Score-P. For that I want to understand, how APEX gets its data from HPX.
In my understanding so far HPX adds “external_timers” when it is built with APEX. But I could not close the gap between the timers and HPX functions jet. For example, the function “async_launch_policy_dispatch” which is shown in the APEX output.

Could someone point me towards the region in HPX where the HPX functions are measured by the external timers?

Thanks a lot!

Specifications

• HPX Version: 1.5.1
• CMake Flags: -DCMAKE_INSTALL_PREFIX=/... -DHPX_WITH_APEX=TRUE -DAPEX_WITH_OTF2=TRUE -DOTF2_ROOT=/...
• Platform (compiler, OS): GCC 7.5.0 , Ubuntu

[hkaiser]

Adding @khuck (the main APEX developer) to this discussion.

[dhinf]

Interesting for us are the user threads and tasks. Score-P records kernel threads, but can't record the user threads used by HPX. For the moment we try to figure out how and where APEX gets the information. Is it possible to get data about thread and task creation/destruction without changing the HPX source code?

[hkaiser]

Interesting for us are the user threads and tasks. Score-P records kernel threads, but can't record the user threads used by HPX. For the moment we try to figure out how and where APEX gets the information. Is it possible to get data about thread and task creation/destruction without changing the HPX source code?

As an alternative to APEX, you could rely on Intel's ITT notify interface that is supported by HPX. We have successfully used it to integrate Intel VTune to collect data related to HPX tasks (see here for details: https://software.intel.com/content/www/us/en/develop/articles/intel-itt-api-open-source.html). This needs to be enabled in HPX at compile time (cmake -DHPX_WITH_ITTNOTIFY=On) and additionally at runtime (command line option --hpx:ini=hpx.use_itt_notify=1).

In short - if enabled, HPX calls functions that have certain predefined names from an external library provided by some external tool. So far we have used it with the Intel tools only (i.e. our build system will support only that), but I don't see a reason why it shouldn't work (with some tweaks) with the code provided by Intel here: https://github.com/intel/ittapi.

[dhinf]

Thanks for the hint. We will try it.

[hkaiser]

Thanks for the hint. We will try it.

Please let us know how it goes. I'd be very interested in hearing what we can do to improve the integration. Also, please don't hesitate to get in contact if you have problems, or questions, etc.

[khuck]

TAU has the same problem that Score-P does, that it expects perfectly nested timers on each OS thread. APEX does have an adapter that will send the timer start/stop events to TAU if tau_exec is used with the executable. Going through APEX makes sure that the timer stacks are properly maintained. I could look into a Score-P adapter for APEX, if that makes sense.

On the other hand, if you were to build (as you suggest) a similar interface to HPX that APEX uses (which is possible, because the "external timer" in HPX is an abstraction of APEX to avoid circular dependencies) you could do something similar with Score-P...but I suspect that you will eventually run into problems with some tasks that start child tasks that are "direct actions" and are potentially pre-empted before the child is finished. Score-P will probably crash in that event (like TAU would).

Simplest solution: I would also suggest you look into APEX tracing to OTF2 files. That is the same trace output that Score-P generates, and I think you can generate Score-P "cube" files from those traces.

Please let me know if you want any other suggestions.