by all processing elements, but has high access latency ;
* read-only memory: smaller, low latency, writable by the host CPU but not the compute devices ;
* local memory: shared by a group of processing elements ;
* per-element private memory (registers; ).

Not every device needs to implement each level of this hierarchy in hardware. Consistency between the various levels in the hierarchy is relaxed, and only enforced by explicit synchronization