Mac m1 install Paddle

Date Tags Paddle

Mac m1系统mini-conda,环境安装paddle问题?

conda activate paddle
python -m pip install paddlepaddle -i https://mirror.baidu.com/pypi/simple
Looking in indexes: https://mirror.baidu.com/pypi/simple
ERROR: Could not find a version that satisfies the requirement paddlepaddle
ERROR: No matching distribution found for paddlepaddle

https://www.paddlepaddle.org.cn/install/quick?docurl=/documentation/docs/zh/install/pip/macos-pip.html

https://github.com/PaddlePaddle/Paddle/issues/32377

macOS 版本 10.x/11.x (64 bit) (不支持GPU版本)

Python 版本 3.6/3.7/3.8/3.9 (64 bit)

pip 或 pip3 版本 20.2.2或更高版本 (64 bit)

CONDA_SUBDIR=osx-64 conda create -n paddle python==3.8.10 // create a new environment called pd_rosetta with intel packages.
conda activate paddle
python -c "import platform;print(platform.machine())"  // should be ‘x86_64’ not ‘arm64’
conda env config vars set CONDA_SUBDIR=osx-64 // # make sure that conda commands in this environment use intel packages
conda deactivate
conda activate paddle
echo "CONDA_SUBDIR: $CONDA_SUBDIR"
python -m pip install paddlepaddle -i https://mirror.baidu.com/pypi/simple
                    more ...
                

Pandas-regex

Regex元字符

https://regex101.com/

https://www.runoob.com/regexp/regexp-metachar.html

定位符

^ : ^[^0-9][0-9]$ 第一个字符不能是数字

$ :

\b :

\B :

特殊单字符

  • . :

  • \d : [0-9]

  • \w : [0-9a-zA-Z_]

  • \s : [\r\n\t\f\v ]

  • \D : [^0-9]

  • \W : [^0-9a-zA-Z_]

  • \S : [^\r\n\t\f\v ]

空白符

\r \n \f \t \v 空格 等价 \s

修饰符

i :

g :

m :

s :

量词

* : 出现0到多次

+ : 出现1到多次

? : 出现0到1次

{m} : 出现m次

{m,} : 出现至少m次

{m,n} : 出现m到n次

范围

| : 或

[ ] more ...





Bert as Service

Date Tags NLP

git clone git@github.com:hanxiao/bert-as-service.git

https://bert-as-service.readthedocs.io/en/latest/section/get-start.html#start-the-bert-service-in-a-docker-container

docker build -t bert-as-service -f ./docker/Dockerfile .

docker run --runtime nvidia -itd -p 8022:5555 -p 8021:5556 -v /bert-as-service/server/model/:/model -t bert-as-service 1 128

usage: /usr/local/bin/bert-serving-start -http_port 8125 -num_worker=4 -max_seq_len=64 -max_batch_size=512 -model_dir /model
                 ARG   VALUE
__________________________________________________
           ckpt_name = bert_model.ckpt
         config_name = bert_config.json
                cors = *
                 cpu = False
          device_map = []
       do_lower_case = True
  fixed_embed_length = False
                fp16 = False
 gpu_memory_fraction = 0.5
       graph_tmp_dir = None
    http_max_connect = 10
           http_port = 8125
        mask_cls_sep = False
      max_batch_size = 512
         max_seq_len = 64
           model_dir = /model
no_position_embeddings = False
    no_special_token = False
          num_worker = 4
       pooling_layer = [-2]
    pooling_strategy = REDUCE_MEAN
                port = 5555
            port_out = 5556
       prefetch_size = 10
 priority_batch_size = 16
show_tokens_to_client = False
     tuned_model_dir = None
             verbose = False
                 xla = False
                    more ...
                

Faiss

0.faiss Mac m1 install源码安装?

master分支若不支持arm64,则下载使用下面 pr:matsui528:workaround-for-aarch64-gcc源码git clone -b branch_name url分支下载

https://github.com/facebookresearch/faiss/pull/1882

https://github.com/facebookresearch/faiss/wiki/Installing-Faiss#compiling-faiss-on-arm
$ brew install llvm
$ brew install swig
$ cd faiss
$ cmake -B build -DFAISS_ENABLE_GPU=OFF -DFAISS_ENABLE_C_API=ON -DBUILD_SHARED_LIBS=ON -DBUILD_TESTING=OFF -DFAISS_ENABLE_PYTHON=ON .
$ cmake  -B build -DCMAKE_CXX_COMPILER=clang++ -DFAISS_ENABLE_GPU=OFF  -DPython_EXECUTABLE=$(which python3) -DFAISS_OPT_LEVEL=generic -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTING=ON
$ make -C build -j faiss
$ make -C build -j swigfaiss
$ cd build/faiss/python/ && python3 setup.py build
$ export PYTHONPATH=$PWD/build/faiss/python/build/lib/
                    more ...
                

Docker Python Mysql

docker pull mysql:5.7

docker run -itd -p 8070:3306 -v /hadoop-data/work/sl/project/mysql/data:/var/lib/mysql -v /hadoop-data/work/sl/project/mysql/conf.d:/etc/mysql/conf.d --name=qa_mysql -e MYSQL_ROOT_PASSWORD=123456 mysql:5.7

mysql -h host_ip -P port …

more ...

Docker build

Date Tags Docker

问题描述:

>>> docker build -t image_name:v1 .
Sending build context to Docker daemon  2.229MB
Step 1/4 : FROM nvidia/cuda:10.0-cudnn7-runtime-centos7
 ---> Running in 7d171c998c6a
Removing intermediate container 7d171c998c6a
 ---> 494b7b9cbb47
Step 3/4 : RUN yum -y install vim
 ---> Running …
more ...

Docker export vs commit

Date Tags Docker

问题描述:

2021-09-04 14:12:08.686281: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Could not dlopen library 'libcuda.so.1'; dlerror: /lib64/libcuda.so.1: file too short
2021-09-04 14:12:08.686315: E tensorflow/stream_executor/cuda/cuda_driver …
more ...