GPT Training
by Jerry Su, post on Feb 20, 2024Teacher Forcing
Exposure bias
- 最优路径。一步错，步步错
Scheduled sampling
- 实现修正纠错。开始更大概率选取ground truth作为target，随着时间更多概率选取模型predict结果作为target，最终逐渐使 …
Nucleus Sampling Top-p Sampling
by Jerry Su, post on Feb 20, 20241. 温度调节（Temperature Scaling）
-
为了调整概率分布的“锐利度”，可以引入一个温度参数（Temperature）。温度较高时，概率分布变得更加平坦，增加了低概 …
Concurrent http requests using asyncio and aiohttp
by Jerry Su, post on Jul 13, 2023import aiohttp
import asyncio
from typing import Dict
async def async_post_request(url: str, data: Dict) -> Dict:
"""async post.
"""
async with aiohttp.ClientSession() as session:
async with session.post(url, json=data) as resp:
response = await resp.json()
return response
model_url_dict={
"scene1": "http://0.0.0.0:8781/classify/utcx",
"scene2 …SELF-INSTRUCT: Aligning Language Model with Self Generated Instructions
by Jerry Su, post on Apr 29, 2023Self-Instruct is a framework that helps language models improve their ability to follow natural language instructions. It does this by using the model’s own generations to create a large collection of instructional data. With Self-Instruct, it is possible to improve the instruction-following capabilities of language models without relying on …
Rust closure
by Jerry Su, post on Jan 05, 2023闭包：可以捕获其所在作用域中变量的匿名函数。
fn add_one_v1 (x: u32) -> u32 { x + 1 };
let add_one_v2 = |x: u32| -> u32 { x + 1 };
let add_one_v3 = |x| { x + 1 };
let add_one_v4 = |x| x + 1 ;
-
调用方式与函 …
Rust [] vs &[] vs vec![]
by Jerry Su, post on Jan 05, 2023What are the differences between [], &[], and vec![]?
Rust self vs &self vs &mut self
by Jerry Su, post on Jan 05, 2023-
self: (self: Self) Having a method that takes ownership of the instance by using just self as the first parameter is rare; this technique is usually used when the method transforms self into something else and you want to prevent the caller from using the original instance after the transformation …
Rust in JupyterLab
by Jerry Su, post on Dec 20, 20221.Install JupyterLab
2.Install Rust
-
curl https://sh.rustup.rs -sSf | sh -
source $HOME/.cargo/env -
export PATH="$HOME/.cargo/bin:$PATH"
3.Install Evcxr Jupyter Kernel
-
cargo install evcxr_jupyter -
evcxr_jupyter --install
4.Reference
https://github.com/google/evcxr/blob/main/evcxr_jupyter/README.md
https://datacrayon.com/posts/programming …
Brackets Matching
by Jerry Su, post on Dec 16, 2022brackets = {')': '(', '》': '《', '）': '（'}
brackets_open, brackets_close = brackets.values(), brackets.keys()
def brackets_match(text: str) -> int:
"""Brackets Match
"""
stack, error = [], []
for idx, char in enumerate(text):
if char in brackets_open:
stack.append((idx, char))
elif char in brackets_close:
if stack and stack[-1][-1] == brackets[char]:
stack.pop()
else:
error.append(idx)
if stack …