Longformer Torch2Paddle

PaddlePaddle-Longformer-model-base-4096

PyTorch Shape Paddle Shape
embeddings.word_embeddings.weight [50265, 768] embeddings.word_embeddings.weight
embeddings.position_embeddings.weight [4098, 768] embeddings.position_embeddings.weight
embeddings.token_type_embeddings.weight [1, 768] embeddings.token_type_embeddings.weight
embeddings.LayerNorm.weight [768] embeddings.layer_norm.weight
embeddings.LayerNorm.bias [768] embeddings.layer_norm.bias
encoder.layer.0.attention.self.query.weight [768, 768] encoder.layers.0.self_attn.query.weight T
encoder.layer.0.attention.self.query.bias [768] encoder.layers.0.self_attn.query.bias
encoder.layer.0.attention.self.key.weight [768, 768] encoder.layers.0.self_attn.key.weight T
encoder.layer.0.attention.self.key.bias [768] encoder.layers.0.self_attn.key.bias
encoder.layer.0.attention.self.value.weight [768, 768] encoder.layers.0.self_attn.value.weight T
encoder.layer.0.attention.self.value.bias [768] encoder.layers.0.self_attn.value.bias
encoder.layer.0.attention.self.query_global.weight [768, 768] encoder.layers.0.self_attn.query_global.weight T
encoder.layer.0.attention.self.query_global.bias [768] encoder.layers.0.self_attn.query_global.bias
encoder.layer.0.attention.self.key_global.weight [768, 768] encoder.layers.0.self_attn.key_global.weight T
encoder.layer.0.attention.self.key_global.bias [768] encoder.layers.0.self_attn.key_global.bias
encoder.layer.0.attention.self.value_global.weight [768, 768] encoder.layers.0.self_attn.value_global.weight T
encoder.layer.0.attention.self.value_global.bias [768] encoder.layers.0.self_attn.value_global.bias
encoder.layer.0.attention.output.dense.weight [768, 768] encoder.layers.0.self_attn.out.weight T
encoder.layer.0.attention.output.dense.bias [768] encoder.layers.0.self_attn.out.bias
encoder.layer.0.attention.output.LayerNorm.weight [768] encoder.layers.0.norm1.weight
encoder.layer.0.attention.output.LayerNorm.bias [768] encoder.layers.0.norm1.bias
encoder.layer.0.intermediate.dense.weight [3072, 768] encoder.layers.0.linear1.weight T [768, 3072]
encoder.layer.0.intermediate.dense.bias [3072] encoder.layers.0.linear1.bias
encoder.layer.0.output.dense.weight [768, 3072] encoder.layers.0.linear2.weight T [3072, 768]
encoder.layer.0.output.dense.bias [768] encoder.layers.0.linear2.bias
encoder.layer.0.output.LayerNorm.weight [768] encoder.layers.0.norm2.weight
encoder.layer.0.output.LayerNorm.bias [768] encoder.layers.0.norm2.bias
pooler.dense.weight [768, 768] pooler.dense.weight T
pooler.dense.bias [768] pooler.dense.bias




pytorch.nn.functional.pad

torch

torch.nn.functional.pad

从输入input的最后一个维度向前padding

输入input的$\left\lfloor\frac{\text{len(pad)}}{2}\right\rfloor$个维度进行padding

  • 如果只padding输入张量input的最后1个维度,pad的形式如:(padding_left, padding_right)

  • 如果只padding输入张量input的最后2个维度,pad的形式如:(padding_left, padding_right, padding_top, padding_bottom)

  • 如果只padding输入张量input的最后3个维度,pad的形式如:(padding_left, padding_right, padding_top, padding_bottom, padding_front, padding_back)


Hierarchical Position Embedding

Date Tags Paddle

预训练模型 预训练的三种embedding

word_embedding: [vocab_size, hidden_size]

position_embedding: [max_len, hidden_size]

token_type_embedding: [token_type_size, hidden_size]

embedding




paddle implements torch.repeat_interleave/K.repeat_elements using paddle.reshape & paddle.tile

https://pytorch.org/docs/stable/generated/torch.Tensor.repeat.html#torch.Tensor.repeat

https://pytorch.org/docs/stable/generated/torch.repeat_interleave.html

If the repeats is tensor([n1, n2, n3, …]), then the output will be tensor([0, 0, …, 1, 1, …, 2, 2, …, …]) where 0 appears n1 times, 1 appears n2 times, 2 appears n3 times, etc.

torch.repeat

torch.repeat_interleave


Mac m1 install Paddle

Date Tags Paddle

Mac m1系统mini-conda,环境安装paddle问题?

conda activate paddle
python -m pip install paddlepaddle -i https://mirror.baidu.com/pypi/simple
Looking in indexes: https://mirror.baidu.com/pypi/simple
ERROR: Could not find a version that satisfies the requirement paddlepaddle
ERROR: No matching distribution found for paddlepaddle

https://www.paddlepaddle.org.cn/install/quick?docurl=/documentation/docs/zh/install/pip/macos-pip.html

https://github.com/PaddlePaddle/Paddle/issues/32377

macOS 版本 10.x/11.x (64 bit) (不支持GPU版本)

Python 版本 3.6/3.7/3.8/3.9 (64 bit)

pip 或 pip3 版本 20.2.2或更高版本 (64 bit)

CONDA_SUBDIR=osx-64 conda create -n paddle python==3.8.10 // create a new environment called pd_rosetta with intel packages.
conda activate paddle
python -c "import platform;print(platform.machine())"  // should be ‘x86_64’ not ‘arm64’
conda env config vars set CONDA_SUBDIR=osx-64 // # make sure that conda commands in this environment use intel packages
conda deactivate
conda activate paddle
echo "CONDA_SUBDIR: $CONDA_SUBDIR"
python -m pip install paddlepaddle -i https://mirror.baidu.com/pypi/simple
                    more ...