GTPin: Itrace Sample Tool

The Itrace tool generates a dynamic control-flow trace of kernel execution

The trace is provided for each kernel, for each Draw/Enqueue granularity, and for each separate HW thread.

Running the Itrace tool

The Itrace tool (as well as all GTPin tracing tools) works in two phases, which should be run separately:

• Pre-processing phase (phase 1). A phase in which the tool determines the number of required records to be saved.
• Trace gathering phase (phase 2). A phase in which the actual trace is collected.

To run the pre-processing phase of the itrace tool (in its default configuration) use the following command:

Profilers/Bin/gtpin -t itrace --phase 1 -- app

NOTE: You can run this phase only once per application.

To run the trace-gathering phase of the itrace tool (in its default configuration), use the following command:

Profilers/Bin/gtpin -t itrace --phase 2 -- app

How to understand Itrace results

When you run the GTPin in-house Itrace tool in a default configuration for pre-processing (phase 1) GTPin generates the directory GTPIN_PROFILE_ITRACE0. In addition, the following two files are created within the current directory:

• itrace_pre_process.txt: Refers to a buffer for kernels dispatched to be executed on the device. The itrace_pre_process.txt file contains the maximum number of trace records generated by each kernel, out of all instances of Draw/Enqueue commands that this kernel executed (on the device). For example, if the kernel generates between 5 and 15 records when executed, the allocated buffer for that kernel should be large enough to hold 15 records.

This file is an input to the trace gathering phase. It has the following format:

VS_asm7cf5b819f3a88d8e_simd8___VS_asm7cf5b819f3a88d8e_simd8_10 2048
CS_asm3f2b2787381d7600_simd8___CS_asm3f2b2787381d7600_simd8_28 2048
VS_asmfab317bc037b0584_simd8___VS_asmfab317bc037b0584_simd8_30 4326
PS_asm967d2b3a9b5e0245_simd8___PS_asm967d2b3a9b5e0245_simd8_35 17888
PS_asm967d2b3a9b5e0245_simd16___PS_asm967d2b3a9b5e0245_simd16_37 95844
VS_asmfab317bc037b0584_simd8___VS_asmfab317bc037b0584_simd8_40 4326

where the numerical field indicates the number of required trace records, and the string field indicates a kernel.

• itrace_pre_process_sw_threads.txt: Refers to a information about the SW threads created by each kernel when the kernel is executed on the device. Each line in the .txt file contains the name of the kernel executed on the device; the number of SW threads generated by that kernel’s execution of a Draw or Enqueue command; the ID of the Draw or Enqueue command; and some other metadata.

This file contains informational data only, and has the following format:

VS_asmfab317bc037b0584_simd8___VS_asmfab317bc037b0584_simd8_30        101  DX12  0  34  318  3  0
VS_asmfab317bc037b0584_simd8___VS_asmfab317bc037b0584_simd8_30         99  DX12  0  34  319  3  0
VS_asmfab317bc037b0584_simd8___VS_asmfab317bc037b0584_simd8_30         28  DX12  0  34  320  3  0
VS_asmfab317bc037b0584_simd8___VS_asmfab317bc037b0584_simd8_30         38  DX12  0  34  321  3  0
VS_asmfab317bc037b0584_simd8___VS_asmfab317bc037b0584_simd8_30         76  DX12  0  34  322  3  0
VS_asmfab317bc037b0584_simd8___VS_asmfab317bc037b0584_simd8_30         22  DX12  0  34  323  3  0
VS_asmfab317bc037b0584_simd8___VS_asmfab317bc037b0584_simd8_30         13  DX12  0  34  324  3  0
VS_asmfab317bc037b0584_simd8___VS_asmfab317bc037b0584_simd8_30        104  DX12  0  34  325  3  0
VS_asmfab317bc037b0584_simd8___VS_asmfab317bc037b0584_simd8_30         97  DX12  0  34  326  3  0

where each line corresponds to a single kernel for a single Draw/Enqueue command. The fields have the following meaning (from left to right):

• Kernel name
• Number of SW threads generated for this Draw/Enqueue command
• The platform the application supports: Intel® oneAPI Level Zero* (Level Zero*), OpenCL*, Microsoft DX11*, Microsoft DX12*, or Vulkan*.
• The execution descriptor (per platform meta data)
  • Level Zero - Device Enqueue.
  • OpenCL - Device Enqueue.
  • DX11 - Device Draw.
  • DX12 - Device CommandList Draw PSO.
  • Vulkan - Device CommandList Draw PSO.
    where:
    • Device - The ID of the logical device on which the kernel was run.
    • Enqueue - The ID of the Enqueue command.
    • Draw - The ID of the Draw command.
    • CommandList - The ID of the Command List.
    • PSO - The ID of the PSO (Pipeline State Object).

When the Itrace tool is run for trace gathering (phase 2), the tool generates a directory: GTPIN_PROFILE_ITRACE1. GTPin saves the profiling results within the folder: GTPIN_PROFILE_ITRACE1\Session_Final. The traces for each kernel are saved in a separate sub-folder that has the same name as the kernel. Each Draw/Enqueue command has a separate trace, which is saved in a corresponding sub-directory, as shown in the following screenshot:

How to uncompress Itrace and read the trace

Each trace is saved in a compressed binary format, within a file called itrace_compressed.bin, as shown above. To uncompress the trace, you must run a Profilers\Scripts\uncompress_itrace.py Python Software Foundation Python* script, in the following manner (Python 3.5 or above is required):

python3 Profilers\Scripts\uncompress_itrace.py --input_dir GTPIN_PROFILE_ITRACE1\Session_Final\PS_asm967d2b3a9b5e0245_simd16\device_0__cl_34__draw_0__pso_3 --gen 9

Running this script opens the compressed trace into separate traces for each HW thread each, as shown in the following screenshot:

where the trace generated on each HW thread is saved in a text file named itrace___s_0_ss_0_eu_1_tid_5.out. The file name indicates the HW thread topology ID (Slice (S), DualSubSlice (DSS), SubSlice (SS), Execution Unit (EU), and HW thread ID (TID)). The resulting trace is provided in the following format:

  BBL ID             INS OFFSET
===============================
     0                0x00
     0                0xc8
    16                0x9c0
    16                0xa78
    17                0xa88
    17                0xae8
    19                0xb58
    20                0xbd8
    21                0xbe8
    22                0xcc8
    23                0xcd8
    24                0xdb8
    25                0xdc8
    26                0xdd0
--- EOT ---
     0                0x00
     0                0xc8
    16                0x9c0
    16                0xa78
    17                0xa88
    17                0xae8
    19                0xb58
    20                0xbd8
    21                0xbe8
    22                0xcc8
    23                0xcd8
    24                0xdb8
    25                0xdc8
    26                0xdd0
--- EOT ---

The left column indicates the basic block (BBL) ID, where the control flow goes. The right column indicates the offset of the first instruction of this BBL. In the case that a BBL consists of plural instructions, where the last one is a control flow instruction, then two lines are provided for the same BBL: One line for its first instruction, and one line for its last instruction. If the last instruction of a BBL is a control flow instruction that passes the control to the first instruction of the same basic block (single BBL loop), then the number of sequential repetitions of this BBL within the control flow is indicated.

An EOT indication separates the consequent dispatches of different SW threads of the kernel from the same HW thread.

To map the basic block IDs and instruction offsets to the kernel code, you must look in the Session_Final\ISA sub-folder, where the GEN assembly of all kernels are saved.

Control-flow graph

In addition to the trace, GTPin dumps the control-flow graph of the kernel into a text file called itrace_total.cfg. This file represents the CFG graph as a list of its edges, along with the frequencies of the control-flow transition on each edge, as accumulated over all HW threads traces. The itrace_total.cfg file has the following format:

 srcBBL, dstBBL, Frequency
=========================
0,1,0
0,16,262144
1,2,0
1,9,0
2,3,0
3,4,0
3,6,0
3,7,0
4,5,0
5,6,0
5,7,0
6,7,0
7,8,0
8,9,0
8,15,0
9,10,0
10,11,0
10,13,0
10,14,0
11,12,0
12,13,0
12,14,0
13,14,0
14,15,0
15,16,0
15,25,0
16,17,262144
16,18,0
17,18,0
17,19,262144
18,19,0
19,20,262144
20,21,262144
20,23,0
20,24,0
21,22,262144
22,23,262144
22,24,0
23,24,262144
24,25,262144
25,26,262144

where each line represents separate CFG edge and contains the following values:

• srcBBL - The ID of the source basic block.
• dstBBL - The ID of the destination basic block.
• frequency - The amount of time the control flow passes from the srcBBL to the dstBBL, accumulated over all kernel executions during current Draw/Enqueue command on any of the HW threads.

To map the basic block IDs and instruction offsets to the kernel code, you must look in the Session_Final\ISA sub-folder, where the GEN assembly of all kernels are saved.

(Back to the list of all GTPin Sample Tools)

itrace.h

/*========================== begin_copyright_notice ============================
Copyright (C) 2018-2023 Intel Corporation

SPDX-License-Identifier: MIT
============================= end_copyright_notice ===========================*/

/*!
 * @file Itrace tool definitions
 */

#ifndef ITRACE_H_
#define ITRACE_H_

#include <list>
#include <map>
#include <vector>
#include <set>

#include "gtpin_api.h"
#include "kernel_weight.h"

using namespace gtpin;

 /* ============================================================================================= */
 // Struct ItraceRecord
 /* ============================================================================================= */
 /*!
  * Structure of the trace record header.
  * The header details architectural state during execution of a BBL.
  */
struct ItraceRecord
{
    uint16_t bblId;     ///< BBL identifier
    uint16_t sr0;       ///< LSB-16 of the State register sr0.0:ud
    uint32_t tileId;    ///< TileId
};

/* ============================================================================================= */
// Class ItraceDispatch
/* ============================================================================================= */
/*!
 * Class that holds trace collected during a single kernel dispatch
 */
class ItraceDispatch
{
public:
    /// Construct a ItraceDispatch object with the empty trace
    explicit ItraceDispatch(const IGtKernelDispatch& dispatch) : _isTrimmed(false) { dispatch.GetExecDescriptor(_kernelExecDesc); }

    /// Read the entire trace from the specified profile buffer into this object
    bool ReadTrace(const GtProfileTrace& traceAccessor, const IGtProfileBuffer& profileBuffer);

    const GtKernelExecDesc& KernelExecDesc() const { return _kernelExecDesc; } ///< @return Descriptor of this kernel dispatch
    uint32_t        Size()      const { return (uint32_t)_rawTrace.size(); }   ///< @return Trace size in bytes
    const uint8_t*  Data()      const { return _rawTrace.data(); }             ///< @return Trace data collected in this dispatch
    uint8_t*        Data()            { return _rawTrace.data(); }             ///< @return Trace data collected in this dispatch
    bool            IsEmpty()   const;                                         ///< @return true if the trace is empty
    bool            IsTrimmed() const { return _isTrimmed; }                   ///< @return true if the trace has been trimmed

private:
    GtKernelExecDesc        _kernelExecDesc; ///< Kernel execution descriptor
    std::vector<uint8_t>    _rawTrace;       ///< Trace data collected in this kernel dispatch
    bool                    _isTrimmed;      ///< true if the trace has been trimmed to avoid buffer overflow
};

/* ============================================================================================= */
// Class ItraceKernel
/* ============================================================================================= */
/*!
 * Class that contains
 *  - Static information about basic block offsets in the kernel
 *  - Collection of instruction traces recorded by kernel dispatches
 */
class ItraceKernel
{
public:
    ItraceKernel() = default;

    /// Construct a ItraceKernel object intended to hold traces of the specified kernel
    explicit ItraceKernel(const IGtKernelInstrument& kernelInstrument, uint32_t numTiles);

    /*!
     * Read a trace recorded by the specified kernel dispatch. Create and add the corresponding ItraceDispatch
     * instance to this object
     */
    ItraceDispatch& AddItrace(IGtKernelDispatch& kernelDispatch);

    std::string           Name()            const { return _name; }               ///< @return Kernel's name
    std::string           ExtendedName()    const { return _extName; }            ///< @return Kernel's extended name
    std::string           UniqueName()      const { return _uniqueName; }         ///< @return Kernel's unique name
    const GtGpuPlatform   Platform()        const { return _platform; }           ///< @return Kernel's platform
    const IGtGenModel&    GenModel()        const { return GetGenModel(_genId); } ///< @return Kernel's GEN model
    const GtProfileTrace& TraceAccessor()   const { return _traceAccessor; }      ///< @return Trace accessor
    void  DumpAsm()                         const;                                ///< Dump kernel's assembly text to file

     /// @return true, if tracing of this kernel is enabled
    uint32_t IsEnabled() const { return (_traceAccessor.MaxTraceSize() != 0); }

    /// @return Control-Flow Graph edges
    typedef std::pair<BblId, BblId> Edge;
    typedef std::set<Edge>          Edges;
    const Edges& GetEdges() const { return _edges; }

    /// @return Traces collected in kernel's dispatches
    typedef std::list<ItraceDispatch> Traces;
    const Traces& GetTraces() const { return _traces; }

    typedef std::pair<ImgOffset, ImgOffset> BblBounds;    ///< Basic block's head and tail offsets
    typedef std::map<BblId, BblBounds>      BblBoundsMap; ///< Basic Block Id to Offsets map
    /// @return Basic Block to Offset map
    const BblBoundsMap& GetBblBounds() const { return _bblBoundsMap; }

    /// @return Number of tiles
    uint32_t NumTiles() const { return _numTiles; }

private:
    std::string       _name;              ///< Kernel's name
    std::string       _uniqueName;        ///< Kernel's unique name
    std::string       _extName;           ///< Kernel's extended name
    GtGpuPlatform     _platform;          ///< Kernel's platform
    GtGenModelId      _genId;             ///< Identifier of the GEN model, the kernel is compiled for
    std::string       _asmText;           ///< Kernel's assembly text
    Edges             _edges;             ///< Kernel's set of control-flow graph's edges
    GtProfileTrace    _traceAccessor;     ///< Trace accessor
    Traces            _traces;            ///< Traces collected in kernel's dispatches
    BblBoundsMap      _bblBoundsMap;      ///< Basic Block to BBlBounds map
    uint32_t          _numTiles;          ///< The number of supported tiles
};

/* ============================================================================================= */
// Class Itrace
/* ============================================================================================= */
/*!
 * Implementation of the IGtTool interface for the Itrace tool
 */
class Itrace : public GtTool
{
public:
    /// Implementation of the IGtTool interface
    const char* Name()       const { return "Itrace"; }
    uint32_t    ApiVersion() const { return GTPIN_API_VERSION; }

    void OnKernelBuild(IGtKernelInstrument& instrumentor);
    void OnKernelRun(IGtKernelDispatch& dispatcher);
    void OnKernelComplete(IGtKernelDispatch& dispatcher);

public:
    static void OnFini();                        ///< Callback function registered with atexit()

    static Itrace* Instance();                   ///< @return Single instance of this class

private:
    Itrace() {}
    Itrace(const Itrace&) = delete;
    Itrace& operator = (const Itrace&) = delete;
    ~Itrace() = default;

    /*!
     * Generate instrumentation for the specified basic block
     * @param[in] instrumentor      Interface of the kernel being instrumented
     * @param[in] bbl               Basic block to be instrumented
     * @param[in] ItraceKernel      Object that holds information about basic blocks in the kernel
     * @return success/failure status
     */
    bool InstrumentBbl(IGtKernelInstrument& instrumentor, const IGtBbl& bbl, const ItraceKernel& ItraceKernel);

    /*!
     * Generate procedure that allocates space for the new trace record in the trace and stores the record
     * for the specified basic block. The procedure sets _offsetReg equal to the offset of the location within the
     * profile buffer immediately following the record header.
     * If new record can not be allocated due to the trace capacity limitations, the procedure zeroes _offsetReg.
     *
     * @param[in, out]  proc            Procedure, the generated code is appended to
     * @param[in]       coder           GEN code generator
     * @param[in]       bbl             Basic block being instrumented
     * @param[in]       ItraceKernel    Object that holds information about basic blocks in the kernel
     * @param[in]       recordSize      Size of the new record, in bytes
     */
    void StoreRecord(GtGenProcedure& proc, const IGtGenCoder& coder, const IGtBbl& bbl,
        const ItraceKernel& ItraceKernel, uint32_t recordSize);

private:
    std::map<GtKernelId, ItraceKernel> _kernels;   ///< Collection of traces per kernel

    GtReg _addrReg;      ///< Virtual register that holds address within profile buffer
    GtReg _dataReg;      ///< Virtual register that holds data to be read from/written to profile buffer 
    GtReg _offsetReg;    ///< Virtual register that holds offset within the trace
    GtReg _tileIdReg;    ///< Virtual register that holds tile ID
};

/* ============================================================================================= */
// Class ItracePreProcessor
/* ============================================================================================= */
/*!
 * Class that computes per-kernel trace sizes in the preprocessing phase, and provides access to
 * this data in the trace gathering phase
 */
class ItracePreProcessor : public KernelWeight
{
public:
    uint64_t TraceSize(const std::string& extKernelName) const; ///< Given extended kernel name, return the trace size in bytes
    static ItracePreProcessor* Instance();                      ///< @return Single instance of this class
    static void OnFini();                                       ///< Callback function registered with atexit()

private:
    ItracePreProcessor();
    ItracePreProcessor(const ItracePreProcessor&) = delete;
    ItracePreProcessor& operator = (const ItracePreProcessor&) = delete;

private:
    /// KernelWeight interface overrides (@see description of KernelWeight functions)
    uint32_t GetBblWeight(IGtKernelInstrument& kernelInstrument, const IGtBbl& bbl) const;
    void AggregateDispatchCounters(KernelWeightCounters& kc, KernelWeightCounters dc) const;

private:
    KernelWeightProfileData _kernelCounters;        ///< Per-kernel counters of required trace records; collected in preprocessing phase
    static const char* _kernelPreProcessFileName;   ///< Name of the file that contains preprocessing data per kernel
    static const char* _dispatchPreProcessFileName; ///< Name of the file that contains preprocessing data per kernel dispatch
};

/* ============================================================================================= */
// Class ItracePostProcessor
/* ============================================================================================= */
/*!
 * Function object that processes kernel traces - stores them in files within the profile directory:
 *
 *    kernel_name
 *    |
 *        |- kernel_dispatch_1
 *           |- itrace_compressed.bin
 *           |- itrace_total.cfg
 *        |- kernel_dispatch_2
 *           |- itrace_compressed.bin
 *           |- itrace_total.cfg
 * The .bin trace files can be uncompressed by the uncompress_itrace.exe utility.
 *
 * Format of .bin trace files:
 * - Static information:
 *     - Number of BBLs in the kernel
 *     - For each BBL:
 *        - BBL ID
 *        - Offset:
 *          value = 0xFFFFXXXX for BBLs which do not end with control flow instruction (XXXX == offset of the BBL within the original binary)
 *          value = 0xYYYYXXXX for BBLs which end with control flow instruction (XXXX == offset of the BBL within the original binary, YYYY == the offset of the control flow instruction)
 * - Dynamic trace data:
 *      - Number of HW threads in which the trace was collected
 *      - For each HW thread:
 *          - HW Thread ID (in the format of sr0.0)
 *          - Number of records collected for this HW thread
 *          - All the records collected for this HW thread
 *
 * The format of *.cfg files:
 * Each line related to a separate edge of the control-flow graph:
 *     - source BBL ID, destination BBL ID, frequency of visiting the edge between source and destination BBLs.
 */
class ItracePostProcessor
{
public:
    /// Construct a ItracePostProcessor object for the specified collection of kernel traces
    ItracePostProcessor(const IGtCore& gtpinCore, const ItraceKernel& ItraceKernel);

    /// Process all kernel traces associated with this object - store them in files within the profile directory
    bool operator()();

private:

    /// Process the specified trace
    void ProcessTrace(const ItraceDispatch& trace);

    /// Store the processed trace in the specified file stream
    void StoreTrace(std::ofstream& fs);

    /// Store static information about BBL offsets in the kernel in the specified file stream
    void StoreBblBoundsInfo(std::ofstream& fs);

    /// Store Global Thread Identifier
    void StoreGlobalTid(uint32_t gtid, std::ofstream& fs);

    /// Store the specified value in the specified file stream in the binary format
    template <typename T> void Store(const T& val, std::ofstream& fs) { fs.write((const char*)&val, sizeof(val)); }

    /// Store kernel's Control Flow Graph
    void StoreCfg(std::ofstream& fs);

private:

    using TraceRecord = std::pair<uint32_t, uint32_t>;         ///< Trace record
    using TraceRecordList  = std::list<TraceRecord>;          ///< List of ItraceRecords per single HW thread
    using PerTileTraceRecords = std::vector<TraceRecordList>; ///< Per tile Itrace records
    typedef std::map<ItraceKernel::Edge, uint32_t> ItraceCfg;   ///< Weightened Control-flow graph
 
private:
    const ItraceKernel*              _kernel;            ///< Kernel&traces to be processed
    std::string                      _kernelDir;         ///< Directory to store kernel's trace files
    std::vector<PerTileTraceRecords> _itraceRecords;     ///< Map of tile ID to list of BBL trace with their frequencies, indexed by the thread ID
    std::vector<uint32_t>            _numProfiledThreads;///< Number of profiled (active) threads per tile
    ItraceCfg                        _cfg;               ///< Control-flow graph
    static const char*               _traceFileName;     ///< Name of the file to store trace in
};

#endif

itrace.cpp

/*========================== begin_copyright_notice ============================
Copyright (C) 2018-2025 Intel Corporation

SPDX-License-Identifier: MIT
============================= end_copyright_notice ===========================*/

/*!
 * @file Implementation of the Itrace tool
 */

#include <fstream>

#include "itrace.h"
#include "gtpin_tool_utils.h"

using namespace gtpin;
using namespace std;

/* ============================================================================================= */
// Configuration
/* ============================================================================================= */
Knob<int>  gKnobMaxTraceBufferInMB("max_buffer_mb", 3072, "itrace - the max allowed size of the trace buffer per kernel in MB");
Knob<int>  gKnobPhase("phase", 0, "tracing tool - processing phase\n { 1 - pre-processing, 2 - processing - trace gathering}");
Knob<bool> gKnobCfgOnly("cfg-only", false, "indicates that the collected trace should not be saved - only resulting cfg file");

/* ============================================================================================= */
// ItraceDispatch implementation
/* ============================================================================================= */
bool ItraceDispatch::ReadTrace(const GtProfileTrace& traceAccessor, const IGtProfileBuffer& profileBuffer)
{
    uint32_t traceSize = traceAccessor.Size(profileBuffer);
    _rawTrace.resize(traceSize);
    _isTrimmed = traceAccessor.IsTruncated(profileBuffer);
    return traceAccessor.Read(profileBuffer, _rawTrace.data(), 0, traceSize);
}

bool ItraceDispatch::IsEmpty() const
{
    return _rawTrace.size() < sizeof(ItraceRecord);
}

/* ============================================================================================= */
// ItraceKernel implementation
/* ============================================================================================= */
ItraceKernel::ItraceKernel(const IGtKernelInstrument& kernelInstrument, uint32_t numTiles) : _numTiles(numTiles)
{
    const IGtKernel& kernel = kernelInstrument.Kernel();
    const IGtCfg& cfg = kernelInstrument.Cfg();

    _name       = GlueString(kernel.Name());
    _extName    = ExtendedKernelName(kernel);
    _platform   = kernel.GpuPlatform();
    _genId      = kernel.GenModel().Id();
    _asmText    = CfgAsmText(cfg);
    _uniqueName = kernel.UniqueName();

    // Initialize trace accessor. The trace capacity is expected to be computed during the preprocessing phase.
    uint64_t traceCapacity = ItracePreProcessor::Instance()->TraceSize(_extName);
    if (traceCapacity == 0)
    {
        // Unknown trace capacity
        GTPIN_WARNING("ITRACE: unknown trace capacity for kernel " + _name + ". Assuming the kernel is filtered out. "
                      "Allocating a buffer of 8KB size. If the kernel is supposed to run, expect buffer overflow. "
                      "In this case, please re-run phase 1 and make sure the kernel is not filtered out.");
        traceCapacity = 0x2000;
    }
    else
    {
        traceCapacity += 0x2000; // Add some space to account for possible fluctuation of trace sizes between phases
        if (traceCapacity > UINT32_MAX)
        {
            GTPIN_WARNING("ITRACE: The kernel " + _name + " exceedeed maximum trace capacity.");
            traceCapacity = UINT32_MAX;
        }
    }
    if (traceCapacity > (uint64_t(gKnobMaxTraceBufferInMB) * 0x100000))
    {
        GTPIN_WARNING("ITRACE: required capacity (" + DecStr(traceCapacity) + ") for kernel " + _name + " is too big - cut to " + DecStr(gKnobMaxTraceBufferInMB) + "MB. "
                      "Expect the final trace to contain partial data.");
        traceCapacity = uint64_t(gKnobMaxTraceBufferInMB) * 0x100000;
    }
    uint32_t maxRecordSize = sizeof(ItraceRecord);
    _traceAccessor = GtProfileTrace((uint32_t)traceCapacity, maxRecordSize);
    _traceAccessor.Allocate(kernelInstrument.ProfileBufferAllocator());
    // Fill basic block offsets info
    for (auto bblPtr : cfg.Bbls())
    {
        const IGtBbl&     bbl = *bblPtr;
        BblId           bblId = bbl.Id();
        const IGtIns& insHead = bbl.FirstIns();
        const IGtIns& insTail = bbl.LastIns();
        uint32_t offsetHead   = cfg.GetInstructionOffset(insHead);
        uint32_t offsetTail   = insTail.IsChangingIP() ? uint32_t(cfg.GetInstructionOffset(insTail)) : 0xFFFFFFFF;
        _bblBoundsMap[bblId]  = BblBounds(offsetHead, offsetTail);
        const EdgeSpan& outgoingEdges = bbl.OutgoingEdges();
        for (auto outEdge : outgoingEdges)
        {
            const IGtBbl& dstBbl = outEdge->DstBbl();
            uint32_t dstBblId = dstBbl.Id();
            _edges.emplace(bblId, dstBblId);
        }
    }
}

ItraceDispatch& ItraceKernel::AddItrace(IGtKernelDispatch& kernelDispatch)
{
    // Create a new ItraceDispatch object and store the entire trace within this object
    _traces.emplace_back(kernelDispatch);
    ItraceDispatch& ItraceDispatch = _traces.back();
    if (!ItraceDispatch.ReadTrace(_traceAccessor, *kernelDispatch.GetProfileBuffer()))
    {
        GTPIN_ERROR_MSG("ITRACE: Failed to read profile buffer for kernel " + _name);
    }
    return ItraceDispatch;
}

void ItraceKernel::DumpAsm() const
{
    DumpKernelAsmText(_name, _uniqueName, _asmText);
}

/* ============================================================================================= */
// Itrace implementation
/* ============================================================================================= */
Itrace* Itrace::Instance()
{
    static Itrace instance;
    return &instance;
}

void Itrace::OnKernelBuild(IGtKernelInstrument& instrumentor)
{
    const IGtKernel& kernel = instrumentor.Kernel();
    uint32_t numTiles = (instrumentor.Coder().IsTileIdSupported()) ? GTPin_GetCore()->GenArch().MaxTiles(kernel.GpuPlatform()) : 1;

    // Create new KernelData object and add it to the data base
    auto ret = _kernels.emplace(piecewise_construct, forward_as_tuple(kernel.Id()), forward_as_tuple(instrumentor, numTiles));
    if (ret.second)
    {
        ItraceKernel& ItraceKernel = (*ret.first).second;
        if (!ItraceKernel.IsEnabled())
        {
            GTPIN_WARNING("ITRACE: The trace won't be generated for kernel " + ItraceKernel.Name());
            return;
        }

        const IGtCfg& cfg = instrumentor.Cfg();
        IGtVregFactory& vregs = instrumentor.Coder().VregFactory();

        // Initialize virtual registers
        _addrReg = vregs.MakeMsgAddrScratch();
        _dataReg = vregs.MakeMsgDataScratch();
        _offsetReg = vregs.MakeScratch(VREG_TYPE_DWORD);
        _tileIdReg = vregs.Make(VREG_TYPE_DWORD);

        GtGenProcedure preCode;
        instrumentor.Coder().LoadTileId(preCode, _tileIdReg);

        // Instrument kernel entries
        instrumentor.InstrumentEntries(preCode);

        // Instrument basic blocks
        for (auto bblPtr : cfg.Bbls())
        {
            InstrumentBbl(instrumentor, *bblPtr, ItraceKernel);
        }
    }
}

void Itrace::OnKernelRun(IGtKernelDispatch& dispatcher)
{
    bool isProfileEnabled = false;

    const IGtKernel& kernel = dispatcher.Kernel();
    GtKernelExecDesc execDesc; dispatcher.GetExecDescriptor(execDesc);
    if (kernel.IsInstrumented() && IsKernelExecProfileEnabled(execDesc, kernel.GpuPlatform(), kernel.Name().Get()))
    {
        auto it = _kernels.find(kernel.Id());
        if (it != _kernels.end())
        {
            const ItraceKernel& ItraceKernel = it->second;
            if (ItraceKernel.IsEnabled())
            {
                IGtProfileBuffer* buffer = dispatcher.CreateProfileBuffer(); GTPIN_ASSERT(buffer);
                const GtProfileTrace& traceAccessor = ItraceKernel.TraceAccessor();
                if (traceAccessor.Initialize(*buffer))
                {
                    isProfileEnabled = true;
                }
                else
                {
                    GTPIN_ERROR_MSG("ITRACE: Failed to write into memory buffer for kernel " + string(kernel.Name()));
                }
            }
        }
    }
    dispatcher.SetProfilingMode(isProfileEnabled);
}

void Itrace::OnKernelComplete(IGtKernelDispatch& dispatcher)
{
    if (!dispatcher.IsProfilingEnabled())
    {
        return; // Do nothing with unprofiled kernel dispatches
    }

    const IGtKernel& kernel = dispatcher.Kernel();
    auto it = _kernels.find(kernel.Id());
    if (it != _kernels.end())
    {
        // Read the trace from the profile buffer
        ItraceKernel& ItraceKernel = it->second;
        ItraceKernel.AddItrace(dispatcher);
    }
}

bool Itrace::InstrumentBbl(IGtKernelInstrument& instrumentor, const IGtBbl& bbl, const ItraceKernel& ItraceKernel)
{
    const IGtGenCoder& coder = instrumentor.Coder();
    const IGtCfg& cfg = instrumentor.Cfg();

    // Generate code that allocates space for the new record in the trace and stores the trace record.
    // Insert this procedure before the first instruction in the basic block.
    GtGenProcedure headerProc;
    auto firstInsIt = bbl.Instructions().begin();
    const IGtIns& firstIns = cfg.GetInstruction((*firstInsIt)->Id());
    StoreRecord(headerProc, coder, bbl, ItraceKernel, sizeof(ItraceRecord));
    instrumentor.InstrumentInstruction(firstIns, GtIpoint::Before(), headerProc);

    return true;
}

void Itrace::StoreRecord(GtGenProcedure& proc, const IGtGenCoder& coder, const IGtBbl& bbl,
    const ItraceKernel& ItraceKernel, uint32_t recordSize)
{
    IGtInsFactory& insF = coder.InstructionFactory();

    GtPredicate    predicate(FlagReg(0));

    // Set values of ItraceRecord fields in _dataReg
    proc += insF.MakeShl(_dataReg, StateReg(0), 16);                    // idFieldReg[16:31] = sr0.0
    proc += insF.MakeAdd(_dataReg, _dataReg, GtImmU32(bbl.Id()));       // idFieldReg[0:15]  = bbl.Id()

    // Allocate new record in the trace.
    // Set _offsetReg = offset of the allocated record in the profile buffer, _addrReg = address of the allocated record
    ItraceKernel.TraceAccessor().ComputeNewRecordOffset(coder, proc, recordSize, _offsetReg);
    coder.ComputeAddress(proc, _addrReg, _offsetReg);

    // Zero _offsetReg if the trace buffer is overflowed (predicate == true)
    proc += insF.MakeMov(_offsetReg, 0).SetPredicate(predicate);

    // Store Sr0.0 and BBL ID
    //if (!predicate) { STORE buffer[_addrReg] = _dataReg;
    proc += insF.MakeAtomicStore(_addrReg, _dataReg).SetPredicate(!predicate);

    // Store tile ID
    //if (!predicate) { STORE buffer[_addrReg] = _dataReg;
    proc += insF.MakeMov(_dataReg, _tileIdReg);
    coder.ComputeRelAddress(proc, _addrReg, _addrReg, offsetof(ItraceRecord, tileId));
    proc += insF.MakeAtomicStore(_addrReg, _dataReg).SetPredicate(!predicate);

    if (!proc.empty()) { proc.front()->AppendAnnotation(__func__); }
}

void Itrace::OnFini()
{
    Itrace& me = *Instance();
    IGtCore* gtpinCore = GTPin_GetCore();
    for (auto& ref : me._kernels)
    {
        const ItraceKernel& ItraceKernel = ref.second;
        ItracePostProcessor(*gtpinCore, ItraceKernel)();
        ItraceKernel.DumpAsm();
    }
}

/* ============================================================================================= */
// ItracePreProcessor implementation
/* ============================================================================================= */
const char* ItracePreProcessor::_kernelPreProcessFileName = "itrace_pre_process.txt";
const char* ItracePreProcessor::_dispatchPreProcessFileName = "itrace_pre_process_dispatch.txt";

ItracePreProcessor::ItracePreProcessor()
{
    if (gKnobPhase == 2)
    {
        // Read the data collected during the preprocessing phase
        std::ifstream is(_kernelPreProcessFileName);
        GTPIN_ASSERT_MSG(is, string("File ") + _kernelPreProcessFileName + " does not exist. The trace won't be generated");
        is >> _kernelCounters;
    }
    else if (gKnobPhase == 1)
    {
        // Create pre_process files or remove old pre_process files's content if they exist
        CreateCleanFile(_kernelPreProcessFileName);
        CreateCleanFile(_dispatchPreProcessFileName);
    }
}

ItracePreProcessor* ItracePreProcessor::Instance()
{
    static ItracePreProcessor instance;
    return &instance;
}

void ItracePreProcessor::OnFini()
{
    ItracePreProcessor& tool = *Instance();
    tool.DumpKernelProfiles(_kernelPreProcessFileName);
    tool.DumpDispatchProfiles(_dispatchPreProcessFileName);
}

uint64_t ItracePreProcessor::TraceSize(const string& extKernelName) const
{
    auto it = _kernelCounters.find(extKernelName);
    return ((it == _kernelCounters.end()) ? 0 : it->second.weight);
}

uint32_t ItracePreProcessor::GetBblWeight(IGtKernelInstrument&, const IGtBbl&) const
{
    return sizeof(ItraceRecord);
}

void ItracePreProcessor::AggregateDispatchCounters(KernelWeightCounters& kc, KernelWeightCounters dc) const
{
    kc.weight = std::max(kc.weight, dc.weight);
    kc.freq += dc.freq;
}

/* ============================================================================================= */
// ItracePostProcessor implementation
/* ============================================================================================= */
const char* ItracePostProcessor::_traceFileName = "itrace_compressed.bin";

ItracePostProcessor::ItracePostProcessor(const IGtCore& gtpinCore, const ItraceKernel& ItraceKernel) :
    _kernel(&ItraceKernel),
    _kernelDir(JoinPath(string(gtpinCore.ProfileDir()), ItraceKernel.UniqueName())) {}

bool ItracePostProcessor::operator()()
{
    if (!MakeDirectory(_kernelDir))
    {
        GTPIN_WARNING("ITRACE: Could not create directory " + _kernelDir);
        return false;
    }

    // Process traces recorded in kernel dispatches
    for (const ItraceDispatch& trace : _kernel->GetTraces())
    {
        if (!trace.IsEmpty())
        {
            if (trace.IsTrimmed())
            {
                GTPIN_WARNING("ITRACE: Detected trace buffer overflow in kernel " + _kernel->Name());
            }

            ProcessTrace(trace);

            string subdir = trace.KernelExecDesc().ToString(_kernel->Platform(), ExecDescFileNameFormat());
            string dir = MakeSubDirectory(_kernelDir, subdir);

            if (!gKnobCfgOnly)
            {
                string filePath = JoinPath(dir, _traceFileName);
                ofstream fs(filePath, std::ios::binary);
                if (!fs)
                {
                    GTPIN_WARNING("ITRACE: Could not create file " + filePath);
                    continue;
                }
                StoreTrace(fs);
            }

            string cfgFilePath = JoinPath(dir, "itrace_total.cfg");
            ofstream cfgfs(cfgFilePath);
            if (!cfgfs)
            {
                GTPIN_WARNING("ITRACE: Could not create file " + cfgFilePath);
                continue;
            }
            StoreCfg(cfgfs);
        }
    }

    return true;
}

void ItracePostProcessor::ProcessTrace(const ItraceDispatch& trace)
{
    const uint8_t* traceData = trace.Data();
    uint32_t       traceSize = trace.Size();

    // Associate trace records with threads - populate _threadTraceRecords array
    const GtStateRegAccessor& sra = _kernel->GenModel().StateRegAccessor();
    uint32_t maxThreads = _kernel->GenModel().MaxThreads(); // Max number of HW threads

    // Build control-flow graph
    _cfg.clear();
    ItraceKernel::Edges edges = _kernel->GetEdges();
    for (auto& edge : edges)
    {
        _cfg.emplace(edge, 0);
    }

    // Reference to the trace record
    struct Record
    {
        const ItraceRecord* header; ///< Pointer to the header of the record
        uint32_t              size; ///< Size of the record in bytes, including header
    };

    using RecordList = std::list<Record>;           ///< List of Records
    using PerTileRecords = std::vector<RecordList>; ///< Per tile Records
    std::vector<PerTileRecords> threadRecords;      ///< Vector of per tile records. Tile ID is an index to this vector

    threadRecords.resize(_kernel->NumTiles());
    _itraceRecords.resize(_kernel->NumTiles());
    _numProfiledThreads.resize(_kernel->NumTiles(), 0);

    for (uint32_t tile = 0; tile < _kernel->NumTiles(); tile++)
    {
        threadRecords[tile].resize(maxThreads);

        _itraceRecords[tile].clear();
        _itraceRecords[tile].resize(maxThreads);
    }

    for (uint32_t recordOffset = 0; recordOffset + sizeof(ItraceRecord) <= traceSize;)
    {
        // Retrive thread ID and BBL ID from the record
        const ItraceRecord* record = (const ItraceRecord*)(traceData + recordOffset);
        uint32_t tid = sra.GetGlobalTid(record->sr0);
        uint32_t tileId = record->tileId; GTPIN_ASSERT(tileId < _kernel->NumTiles());
        uint32_t recordSize = sizeof(ItraceRecord);
        if (recordOffset + recordSize > traceSize)
        {
            break; // end of trace
        }

        auto& tileRecords = threadRecords[tileId];
        auto& records = tileRecords[tid];

        // Add a new trace record reference to _threadTraceRecords
        if (records.empty()) { ++_numProfiledThreads[tileId]; } // Increment thread count on the first relevant record
        records.emplace_back(Record{ record, recordSize });

        recordOffset += recordSize;
    }

    for (uint32_t tileId = 0; tileId < threadRecords.size(); tileId++)
    {
        if (_numProfiledThreads[tileId] == 0)
        {
            continue;
        }

        const auto& tileRecords = threadRecords[tileId];
        auto& tileItraceRecords = _itraceRecords[tileId];

        // Store per-thread traces
        for (uint32_t tid = 0; tid < maxThreads; tid++)
        {
            const auto& threadRecordList = tileRecords[tid];
            auto& records = tileItraceRecords[tid];

            if (threadRecordList.empty())
            {
                continue;
            }

            // Store trace records
            BblId prevBblId;
            for (const auto& record : threadRecordList)
            {
                const auto& header = *(record.header);
                BblId       bblId  = header.bblId;

                if (prevBblId != bblId)
                {
                    records.emplace_back(bblId, 1);
                }
                else
                {
                    auto& r = records.back();
                    r.second++;
                }

                if (prevBblId.IsValid())
                {
                    auto it = _cfg.find(ItraceKernel::Edge(prevBblId, bblId));
                    if (it != _cfg.end())
                    {
                        it->second += 1;
                    }
                }

                prevBblId = bblId;
            }
        }
    }
}

void ItracePostProcessor::StoreTrace(std::ofstream& fs)
{
    StoreBblBoundsInfo(fs);

    // Compute and store the number of involved tiles
    uint32_t numOfTiles = 0;
    for (uint32_t i = 0; i < _numProfiledThreads.size(); i++)
    {
        numOfTiles += (_numProfiledThreads[i] == 0) ? 0 : 1;
    }
    Store(numOfTiles, fs);

    for (uint32_t tileId = 0; tileId < _itraceRecords.size(); tileId++)
    {
        if (_numProfiledThreads[tileId] == 0) { continue; }

        const auto& tileRecords = _itraceRecords[tileId];

        Store(tileId, fs);

        // Store the number of profiled threads
        Store(_numProfiledThreads[tileId], fs);

        // Store per-thread traces
        for (uint32_t tid = 0; tid < tileRecords.size(); tid++)
        {
            const auto& records = tileRecords[tid];

            if (records.empty())
            {
                continue;
            }

            StoreGlobalTid(tid, fs);    // Store Global Thread Identifier

            uint32_t numRecords = (uint32_t)records.size();
            Store(numRecords, fs);      // Store #records collected in the thread

            // Store trace records
            for (const auto& record : records)
            {
                uint32_t bblId = record.first;
                uint32_t loopCount = record.second;
                Store(bblId, fs);       // Store BBL ID
                Store(loopCount, fs);   // Store loopCount
            }
        }
    }
}

void ItracePostProcessor::StoreBblBoundsInfo(std::ofstream& fs)
{
    // Store static information about memory accesses in BBLs
    ItraceKernel::BblBoundsMap bblBoundsMap = _kernel->GetBblBounds();
    uint32_t numBbls = (uint32_t)bblBoundsMap.size();
    Store(numBbls, fs);                                 // Store the number of BBLs that access memory

    for (uint32_t bblId = 0; bblId < numBbls; bblId++)
    {
        auto bounds = bblBoundsMap[bblId];
        Store(bblId, fs);                               // Store BBL ID
        uint32_t val = bounds.first;
        Store(val, fs);
        val = bounds.second;
        Store(val, fs);
    }
}

void ItracePostProcessor::StoreGlobalTid(uint32_t gtid, std::ofstream& fs)
{
    const GtStateRegAccessor& sra = _kernel->GenModel().StateRegAccessor();
    uint32_t sr0 = sra.SetGlobalTid(0, gtid);

    auto storeSr0Field = [&](const ScatteredBitFieldU32& sbf)
    {
        uint32_t val = (sbf.IsEmpty() ? UINT32_MAX : sbf.GetValue(sr0));
        Store(val, fs);
    };

    storeSr0Field(sra.SliceIdField());
    storeSr0Field(sra.DualSubSliceIdField());
    storeSr0Field(sra.SubSliceIdField());
    storeSr0Field(sra.EuIdField());
    storeSr0Field(sra.ThreadSlotField());
}

void ItracePostProcessor::StoreCfg(std::ofstream& fs)
{
    ostringstream ostr;
    ostr << "srcBBL, dstBBL, Frequency" << std::endl;
    ostr << "=========================" << std::endl;
    for (const auto& it : _cfg)
    {
        ItraceKernel::Edge edge = it.first;
        uint32_t frequency = it.second;

        // print srcBBL, dstBBL, frequency
        ostr << std::dec << edge.first << "," << edge.second << "," << frequency << std::endl;
    }
    fs << ostr.str();
    fs.close();
}

/* ============================================================================================= */
// GTPin_Entry
/* ============================================================================================= */
EXPORT_C_FUNC void GTPin_Entry(int argc, const char* argv[])
{
    ConfigureGTPin(argc, argv);
    if (gKnobPhase == 1)
    {
        ItracePreProcessor::Instance()->Register();
        atexit(ItracePreProcessor::OnFini);
    }
    else
    {
        GTPIN_ASSERT_MSG((gKnobPhase == 2), "Itrace: Invalid phase value. Should be 1 or 2, provided " + std::to_string(gKnobPhase));
        Itrace::Instance()->Register();
        atexit(Itrace::OnFini);
    }
}

(Back to the list of all GTPin Sample Tools)