[RelVals] [GPU] 34634.XX failing with SIGABRT

[raoatifshad]

	kernel_obj=0x1512f5489a80, kernarg_address=0x0x1512c8306100
	completion_signal=0x0, correlation_id=0
	rptr=155, wptr=442
 Kernel Name: _ZN6alpaka6detail9gpuKernelIN17alpaka_rocm_async28caHitNtupletGeneratorKernels24Kernel_fillHitDetIndicesIN13pixelTopology8Phase2OTEEENS_22AccGpuUniformCudaHipRtINS_8ApiHipRtESt17integral_constantImLm1EEjEEJN4reco11TrackLayoutILm128ELb0EE22ViewTemplateFreeParamsILm128ELb0ELb1ELb1EEENSD_15TrackHitsLayoutILm128ELb0EE22ViewTemplateFreeParamsILm128ELb0ELb1ELb1EEEPN3cms11alpakatools24OneToManyAssocSequentialIjLj4294967295ELj4294967295EEENSD_18TrackingHitsLayoutILm128ELb0EE27ConstViewTemplateFreeParamsILm128ELb0ELb1ELb1EEEEEEvNS_3VecINS_5trait7DimTypeIT0_vE4typeENSW_7IdxTypeISY_vE4typeEEET_DpT1_
VGPU=0x15197ab9c400 SWq=0x151bff1fe000, HWq=0x151ba7100000, id=2
	Dispatch Header =0xb02 (type=2, barrier=1, acquire=1, release=1), setup=0
	grid=[123008, 1, 1], workgroup=[128, 1, 1]
	private_seg_size=64, group_seg_size=0
	kernel_obj=0x1512f5489a80, kernarg_address=0x0x1512c8306100
	completion_signal=0x0, correlation_id=0
	rptr=155, wptr=442
 Kernel Name: _ZN6alpaka6detail9gpuKernelIN17alpaka_rocm_async28caHitNtupletGeneratorKernels24Kernel_fillHitDetIndicesIN13pixelTopology8Phase2OTEEENS_22AccGpuUniformCudaHipRtINS_8ApiHipRtESt17integral_constantImLm1EEjEEJN4reco11TrackLayoutILm128ELb0EE22ViewTemplateFreeParamsILm128ELb0ELb1ELb1EEENSD_15TrackHitsLayoutILm128ELb0EE22ViewTemplateFreeParamsILm128ELb0ELb1ELb1EEEPN3cms11alpakatools24OneToManyAssocSequentialIjLj4294967295ELj4294967295EEENSD_18TrackingHitsLayoutILm128ELb0EE27ConstViewTemplateFreeParamsILm128ELb0ELb1ELb1EEEEEEvNS_3VecINS_5trait7DimTypeIT0_vE4typeENSW_7IdxTypeISY_vE4typeEEET_DpT1_
VGPU=0x15197fcfe400 SWq=0x151bff1fe000, HWq=0x151ba7100000, id=2
	Dispatch Header =0xb02 (type=2, barrier=1, acquire=1, release=1), setup=0
	grid=[123008, 1, 1], workgroup=[128, 1, 1]
	private_seg_size=64, group_seg_size=0
	kernel_obj=0x1512f5489a80, kernarg_address=0x0x1512c8306100
	completion_signal=0x0, correlation_id=0
	rptr=155, wptr=442
 :0:rocdevice.cpp            :3580: 3593109564906 us:  Callback: Queue 0x151ba7100000 aborting with error : HSA_STATUS_ERROR_EXCEPTION: An HSAIL operation resulted in a hardware exception. code: 0x1016

full logs here

[cmsbuild]

cms-bot internal usage

[cmsbuild]

A new Issue was created by @raoatifshad.

@Dr15Jones, @ftenchini, @makortel, @mandrenguyen, @sextonkennedy, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[makortel]

It is an assertion failure

src/RecoTracker/PixelSeeding/plugins/alpaka/CAHitNtupletGeneratorKernelsImpl.h:715: void alpaka_rocm_async::caHitNtupletGeneratorKernels::Kernel_fillHitDetIndices<pixelTopology::Phase2OT>::operator()(const Acc1D &, TkSoAView, TkHitSoAView, const HitContainer *__restrict, HitsConstView) const [TrackerTraits = pixelTopology::Phase2OT]: Device-side assertion `foundNtuplets->content[idx] < (uint32_t)hh.metadata().size()' failed.
Kernel Name: 

followed by

Kernel Name: _ZN6alpaka6detail9gpuKernelIN17alpaka_rocm_async28caHitNtupletGeneratorKernels24Kernel_fillHitDetIndicesIN13pixelTopology8Phase2OTEEENS_22AccGpuUniformCudaHipRtINS_8ApiHipRtESt17integral_constantImLm1EEjEEJN4reco11TrackLayoutILm128ELb0EE22ViewTemplateFreeParamsILm128ELb0ELb1ELb1EEENSD_15TrackHitsLayoutILm128ELb0EE22ViewTemplateFreeParamsILm128ELb0ELb1ELb1EEEPN3cms11alpakatools24OneToManyAssocSequentialIjLj4294967295ELj4294967295EEENSD_18TrackingHitsLayoutILm128ELb0EE27ConstViewTemplateFreeParamsILm128ELb0ELb1ELb1EEEEEEvNS_3VecINS_5trait7DimTypeIT0_vE4typeENSW_7IdxTypeISY_vE4typeEEET_DpT1_
VGPU=0x1515a3b2c800 SWq=0x1515c1096000, HWq=0x151569f00000, id=3
	Dispatch Header =0xb02 (type=2, barrier=1, acquire=1, release=1), setup=0
	grid=[123008, 1, 1], workgroup=[128, 1, 1]
	private_seg_size=64, group_seg_size=0
	kernel_obj=0x15115f489f80, kernarg_address=0x0x15103e506100
	completion_signal=0x0, correlation_id=0
	rptr=421, wptr=545

[makortel]

assign RecoTracker/PixelSeeding

[cmsbuild]

New categories assigned: reconstruction

@jfernan2,@mandrenguyen,@Moanwar,@srimanob you have been requested to review this Pull request/Issue and eventually sign? Thanks

[makortel]

assign heterogeneous

[cmsbuild]

New categories assigned: heterogeneous

@fwyzard,@makortel you have been requested to review this Pull request/Issue and eventually sign? Thanks

[akritkbehera]

The RelVals are still failing on amd_mi300x/amd_w7900 GPUs

[raoatifshad]

Hi everyone, just following up on this issue. We are still seeing these relvals failing. Is there any update on this?