Interactive Diagram Lab

Computational Graph & Tensor Dataflow Engine

Attention Is All You Need (Vaswani et al., 2017)

Computes semantic token-to-token similarity ( $Q K^T$ ) and applies softmax weights over value representations

\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{Q K^T}{\sqrt{d_k}}\right) V

시퀀스 길이 (

N

)512헤드 차원 (

d_k

)64

K

512 × 64

transpose→0 FLOPs

K_T

64 × 512

01Transpose Key Matrix

Transposes the [ $N \times d_k$ ] Key matrix into [ $d_k \times N$ ] for dot-product multiplication

K^T = \text{Transpose}(K)

Total Computation 68.16 MFLOPsOutput [512 × 64]

Core Architectural & Hardware Specification

Definition

A computational framework abstracting neural equations into a Directed Acyclic Graph (DAG), mapping tensors as dataflow edges and tracking autograd dependencies.

Core Mechanics

•Decomposes math into primitive nodes (MatMul, Softmax, Norm) and tensor edges
•Tracks backward paths by retaining forward activations to apply the chain rule

Limitations & Bottlenecks

•Dynamic control flows limit ahead-of-time static JIT optimizations

Comparison

Unlike text equations, computational graphs explicitly expose operator independence, enabling hardware compilers (XLA, TensorRT) to execute automatic parallelization.