“…Maschinenindustrie von P. Steller. Elektrotechnische industrie von J. …”

Tabla de Contenidos: “…Communication bottlenecks in data parallel training -- Analyzing the communication workloads -- Parameter server architecture -- The All-Reduce architecture -- The inefficiency of state-of-the-art communication schemes -- Leveraging idle links and host resources -- Tree All-Reduce -- Hybrid data transfer over PCIe and NVLink -- On-device memory bottlenecks -- Recomputation and quantization -- Recomputation -- Quantization -- Summary -- Section 2 - Model Parallelism -- Chapter 5: Splitting the Model -- Technical requirements -- Single-node training error - out of memory -- Fine-tuning BERT on a single GPU -- Trying to pack a giant model inside one state-of-the-art GPU -- ELMo, BERT, and GPT -- Basic concepts -- RNN -- ELMo -- BERT -- GPT -- Pre-training and fine-tuning -- State-of-the-art hardware -- P100, V100, and DGX-1 -- NVLink -- A100 and DGX-2 -- NVSwitch -- Summary -- Chapter 6: Pipeline Input and Layer Split -- Vanilla model parallelism is inefficient -- Forward propagation -- Backward propagation -- GPU idle time between forward and backward propagation -- Pipeline input -- Pros and cons of pipeline parallelism -- Advantages of pipeline parallelism -- Disadvantages of pipeline parallelism -- Layer split -- Notes on intra-layer model parallelism -- Summary -- Chapter 7: Implementing Model Parallel Training and Serving Workflows -- Technical requirements -- Wrapping up the whole model parallelism pipeline -- A model parallel training overview -- Implementing a model parallel training pipeline -- Specifying communication protocol among GPUs -- Model parallel serving -- Fine-tuning transformers -- Hyperparameter tuning in model parallelism -- Balancing the workload among GPUs -- Enabling/disabling pipeline parallelism -- NLP model serving -- Summary -- Chapter 8: Achieving Higher Throughput and Lower Latency -- Technical requirements…”