当前位置： 首页 > news >正文

网站开发费记什么会计科目,我的世界怎么做购买点卷网站,公众号 链接wordpress,做请柬网站快速方便地下载huggingface的模型库和数据集 方法一#xff1a;用于使用 aria2/wgetgit 下载 Huggingface 模型和数据集的 CLI 工具特点Usage 方法二#xff1a;模型下载【个人使用记录】保持目录结构数据集下载不足之处 方法一#xff1a;用于使用 aria2/wgetgit 下载 Hugg… 快速方便地下载huggingface的模型库和数据集 方法一用于使用 aria2/wgetgit 下载 Huggingface 模型和数据集的 CLI 工具特点Usage 方法二模型下载【个人使用记录】保持目录结构数据集下载不足之处 方法一用于使用 aria2/wgetgit 下载 Huggingface 模型和数据集的 CLI 工具 来自https://gist.github.com/padeoe/697678ab8e528b85a2a7bddafea1fa4f。 使用方法将hfd.sh拷贝过去然后参考下面的参考命令下载数据集或者模型 Huggingface 模型下载器 考虑到官方 huggingface-cli 缺乏多线程下载支持以及错误处理不足在 hf_transfer 中这个命令行工具巧妙地利用 wget 或 aria2 来处理 LFS 文件并使用 git clone 来处理其余文件。 特点 ⏯️ 从断点恢复您可以随时重新运行它或按 CtrlC。 多线程下载利用多线程加速下载过程。 文件排除使用–exclude或–include跳过或指定文件为具有重复格式的模型例如.bin或.safetensors节省时间。 身份验证支持对于需要 Huggingface 登录的门控模型请使用 –hf_username 和 –hf_token 进行身份验证。 镜像站点支持使用“HF_ENDPOINT”环境变量进行设置。代理支持使用“HTTPS_PROXY”环境变量进行设置。 简单仅依赖git、aria2c/wget。 Usage 首先下载 hfd.sh 或克隆此存储库然后授予脚本执行权限。 chmod ax hfd.sh为了方便起见您可以创建一个别名 alias hfd\(PWD/hfd.sh使用说明 \) ./hfd.sh -h Usage:hfd repo_id [–include include_pattern] [–exclude exclude_pattern] [–hf_username username] [–hf_token token] [–tool aria2c|wget] [-x threads] [–dataset] [–local-dir path]Description:Downloads a model or dataset from Hugging Face using the provided repo ID.Parameters:repo_id The Hugging Face repo ID in the format org/repo_name.–include (Optional) Flag to specify a string pattern to include files for downloading.–exclude (Optional) Flag to specify a string pattern to exclude files from downloading.include/exclude_pattern The pattern to match against filenames, supports wildcard characters. e.g., –exclude .safetensor, –include vae/.–hf_username (Optional) Hugging Face username for authentication. NOT EMAIL.–hf_token (Optional) Hugging Face token for authentication.–tool (Optional) Download tool to use. Can be aria2c (default) or wget.-x (Optional) Number of download threads for aria2c. Defaults to 4.–dataset (Optional) Flag to indicate downloading a dataset.–local-dir (Optional) Local directory path where the model or dataset will be stored.Example:hfd bigscience/bloom-560m –exclude *.safetensorshfd meta-llama/Llama-2-7b –hf_username myuser –hf_token mytoken -x 4hfd lavita/medical-qa-shared-task-v1-toy –dataset下载模型 hfd bigscience/bloom-560m下载模型需要登录 从https://huggingface.co/settings/tokens获取huggingface令牌然后 hfd meta-llama/Llama-2-7b –hf_username YOUR_HF_USERNAME_NOT_EMAIL –hf_token YOUR_HF_TOKEN下载模型并排除某些文件例如.safetensors hfd bigscience/bloom-560m –exclude *.safetensors使用 aria2c 和多线程下载 hfd bigscience/bloom-560m输出 下载过程中将显示文件 URL $ hfd bigscience/bloom-560m –tool wget –exclude *.safetensors … Start Downloading lfs files, bash script:wget -c https://huggingface.co/bigscience/bloom-560m/resolve/main/flax_model.msgpack

wget -c https://huggingface.co/bigscience/bloom-560m/resolve/main/model.safetensors

wget -c https://huggingface.co/bigscience/bloom-560m/resolve/main/onnx/decoder_model.onnx …# 安装包 apt update apt-get install aria2 apt-get install iftop apt-get install git-lfs #参考命令 bash /xxx/xxx/hfd.sh mmaaz60/ActivityNet-QA-Test-Videos –tool aria2c -x 16 –dataset –local-dir /xxx/xxx/ActivityNethfd.sh #!/usr/bin/env bash

Color definitions

RED\033[0;31m GREEN\033[0;32m YELLOW\033[1;33m NC\033[0m # No Colortrap printf \({YELLOW}\nDownload interrupted. If you re-run the command, you can resume the download from the breakpoint.\n\){NC}; exit 1 INTdisplay_help() {cat EOF Usage:hfd repo_id [–include include_pattern] [–exclude exclude_pattern] [–hf_username username] [–hf_token token] [–tool aria2c|wget] [-x threads] [–dataset] [–local-dir path] Description:Downloads a model or dataset from Hugging Face using the provided repo ID.Parameters:repo_id The Hugging Face repo ID in the format org/repo_name.–include (Optional) Flag to specify a string pattern to include files for downloading.–exclude (Optional) Flag to specify a string pattern to exclude files from downloading.include/exclude_pattern The pattern to match against filenames, supports wildcard characters. e.g., –exclude .safetensor, –include vae/.–hf_username (Optional) Hugging Face username for authentication. NOT EMAIL.–hf_token (Optional) Hugging Face token for authentication.–tool (Optional) Download tool to use. Can be aria2c (default) or wget.-x (Optional) Number of download threads for aria2c. Defaults to 4.–dataset (Optional) Flag to indicate downloading a dataset.–local-dir (Optional) Local directory path where the model or dataset will be stored.Example:hfd bigscience/bloom-560m –exclude .safetensorshfd meta-llama/Llama-2-7b –hf_username myuser –hf_token mytoken -x 4hfd lavita/medical-qa-shared-task-v1-toy –dataset EOFexit 1 }MODEL_ID\(1 shift# Default values TOOLaria2c THREADS4 HF_ENDPOINT\){HF_ENDPOINT:-https://hf-mirror.com}while [[ \(# -gt 0 ]]; docase \)1 in–include) INCLUDE_PATTERN\(2; shift 2 ;;--exclude) EXCLUDE_PATTERN\)2; shift 2 ;;–hf_username) HF_USERNAME\(2; shift 2 ;;--hf_token) HF_TOKEN\)2; shift 2 ;;–tool) TOOL\(2; shift 2 ;;-x) THREADS\)2; shift 2 ;;–dataset) DATASET1; shift ;;–local-dir) LOCAL_DIR$2; shift 2 ;;) shift ;;esac done# Check if aria2, wget, curl, git, and git-lfs are installed check_command() {if ! command -v \(1 /dev/null; thenecho -e \){RED}\(1 is not installed. Please install it first.\){NC}exit 1fi }# Mark current repo safe when using shared file system like samba or nfs ensure_ownership() {if git status 21 | grep fatal: detected dubious ownership in repository at /dev/null; thengit config –global –add safe.directory \({PWD}printf \){YELLOW}Detected dubious ownership in repository, mark \({PWD} safe using git, edit ~/.gitconfig if you want to reverse this.\n\){NC} fi }[[ \(TOOL aria2c ]] check_command aria2c [[ \)TOOL wget ]] check_command wget check_command curl; check_command git; check_command git-lfs[[ -z \(MODEL_ID || \)MODEL_ID ~ ^-h ]] display_helpif [[ -z \(LOCAL_DIR ]]; thenLOCAL_DIR\){MODEL_ID#*/} fiif [[ \(DATASET 1 ]]; thenMODEL_IDdatasets/\)MODEL_ID fi echo Downloading to \(LOCAL_DIRif [ -d \)LOCAL_DIR/.git ]; thenprintf \({YELLOW}%s exists, Skip Clone.\n\){NC} \(LOCAL_DIRcd \)LOCAL_DIR ensure_ownership GIT_LFS_SKIP_SMUDGE1 git pull || { printf \({RED}Git pull failed.\){NC}\n; exit 1; } elseREPO_URL\(HF_ENDPOINT/\)MODEL_IDGIT_REFS_URL\({REPO_URL}/info/refs?servicegit-upload-packecho Testing GIT_REFS_URL: \)GIT_REFS_URLresponse\((curl -s -o /dev/null -w %{http_code} \)GIT_REFS_URL)if [ \(response 401 ] || [ \)response 403 ]; thenif [[ -z \(HF_USERNAME || -z \)HF_TOKEN ]]; thenprintf \({RED}HTTP Status Code: \)response.\nThe repository requires authentication, but –hf_username and –hf_token is not passed. Please get token from https://huggingface.co/settings/tokens.nExiting.n${NC}exit 1fiREPO_URLhttps://\(HF_USERNAME:\)HF_TOKEN\({HF_ENDPOINT#https://}/\)MODEL_IDelif [ \(response ! 200 ]; thenprintf \){RED}Unexpected HTTP Status Code: \(response\n\){NC}printf \({YELLOW}Executing debug command: curl -v %s\nOutput:\){NC}\n \(GIT_REFS_URLcurl -v \)GIT_REFS_URL; printf \n\({RED}Git clone failed.\n\){NC}; exit 1fiecho GIT_LFS_SKIP_SMUDGE1 git clone \(REPO_URL \)LOCAL_DIRGIT_LFS_SKIP_SMUDGE1 git clone \(REPO_URL \)LOCAL_DIR cd \(LOCAL_DIR || { printf \){RED}Git clone failed.\n\({NC}; exit 1; }ensure_ownershipwhile IFS read -r file; dotruncate -s 0 \)filedone \((git lfs ls-files | cut -d -f 3-) fiprintf \nStart Downloading lfs files, bash script:\ncd \)LOCAL_DIR\n files\((git lfs ls-files | cut -d -f 3-) declare -a urlswhile IFS read -r file; dourl\)HF_ENDPOINT/\(MODEL_ID/resolve/main/\)filefile_dir\((dirname \)file)mkdir -p \(file_dirif [[ \)TOOL wget ]]; thendownload_cmdwget -c $url\ -O $file[[ -n \(HF_TOKEN ]] download_cmdwget --header\Authorization: Bearer \){HF_TOKEN}\ -c $url\ -O $file\elsedownload_cmdaria2c –console-log-levelerror –file-allocationnone -x \(THREADS -s \)THREADS -k 1M -c $url\ -d $file_dir\ -o $(basename \(file)\[[ -n \)HF_TOKEN ]] download_cmdaria2c –header\Authorization: Bearer \({HF_TOKEN}\ --console-log-levelerror --file-allocationnone -x \)THREADS -s \(THREADS -k 1M -c \\)url\ -d $file_dir\ -o $(basename \(file)\fi[[ -n \)INCLUDE_PATTERN ! \(file \)INCLUDE_PATTERN ]] printf # %s\n \(download_cmd continue[[ -n \)EXCLUDE_PATTERN \(file \)EXCLUDE_PATTERN ]] printf # %s\n \(download_cmd continueprintf %s\n \)download_cmdurls(\(url|\)file) done \(filesfor url_file in \){urls[]}; doIFS| read -r url file \(url_fileprintf \){YELLOW}Start downloading \({file}.\n\){NC} file_dir\((dirname \)file)if [[ \(TOOL wget ]]; then[[ -n \)HF_TOKEN ]] wget –headerAuthorization: Bearer \({HF_TOKEN} -c \)url -O \(file || wget -c \)url -O \(fileelse[[ -n \)HF_TOKEN ]] aria2c –headerAuthorization: Bearer \({HF_TOKEN} --console-log-levelerror --file-allocationnone -x \)THREADS -s \(THREADS -k 1M -c \)url -d \(file_dir -o \)(basename \(file) || aria2c --console-log-levelerror --file-allocationnone -x \)THREADS -s \(THREADS -k 1M -c \)url -d \(file_dir -o \)(basename \(file)fi[[ \)? -eq 0 ]] printf Downloaded %s successfully.\n \(url || { printf \){RED}Failed to download %s.\n\({NC} \)url; exit 1; } doneprintf \({GREEN}Download completed successfully.\n\){NC} 方法二模型下载【个人使用记录】 这个代码不能保持目录结构见下面的改进版 import datetime import os import threadingfrom huggingface_hub import hf_hub_url from huggingface_hub.hf_api import HfApi from huggingface_hub.utils import filter_repo_objects# 执行命令 def execCmd(cmd):print(命令%s开始运行%s % (cmd, datetime.datetime.now()))os.system(cmd)print(命令%s结束运行%s % (cmd, datetime.datetime.now()))if name main:# 需下载的hf库名称repo_id Salesforce/blip2-opt-2.7b# 本地存储路径save_path ./blip2-opt-2.7b# 获取项目信息_api HfApi()repo_info _api.repo_info(repo_idrepo_id,repo_typemodel,revisionmain,tokenNone,)# 获取文件信息filtered_repo_files list(filter_repo_objects(items[f.rfilename for f in repo_info.siblings],allow_patternsNone,ignore_patternsNone,))cmds []threads []# 需要执行的命令列表for file in filtered_repo_files:# 获取路径url hf_hub_url(repo_idrepo_id, filenamefile)# 断点下载指令cmds.append(fwget -c {url} -P {save_path})print(cmds)print(程序开始%s % datetime.datetime.now())for cmd in cmds:th threading.Thread(targetexecCmd, args(cmd,))th.start()threads.append(th)for th in threads:th.join()print(程序结束%s % datetime.datetime.now())保持目录结构 import datetime import os import threading from pathlib import Pathfrom huggingface_hub import hf_hub_url from huggingface_hub.hf_api import HfApi from huggingface_hub.utils import filter_repo_objects# 执行命令 def execCmd(cmd):print(命令%s开始运行%s % (cmd, datetime.datetime.now()))os.system(cmd)print(命令%s结束运行%s % (cmd, datetime.datetime.now()))if name main:# 需下载的hf库名称repo_id Salesforce/blip2-opt-2.7b# 本地存储路径save_path ./blip2-opt-2.7b# 创建本地保存目录Path(save_path).mkdir(parentsTrue, exist_okTrue)# 获取项目信息_api HfApi()repo_info _api.repo_info(repo_idrepo_id,repo_typemodel,revisionmain,tokenNone,)# 获取文件信息filtered_repo_files list(filter_repo_objects(items[f.rfilename for f in repo_info.siblings],allow_patternsNone,ignore_patternsNone,))cmds []threads []# 需要执行的命令列表for file in filtered_repo_files:# 获取路径url hf_hub_url(repo_idrepo_id, filenamefile)# 在本地创建子目录local_file os.path.join(save_path, file)local_dir os.path.dirname(local_file)Path(local_dir).mkdir(parentsTrue, exist_okTrue)# 断点下载指令cmds.append(fwget -c {url} -P {local_dir})print(cmds)print(程序开始%s % datetime.datetime.now())for cmd in cmds:th threading.Thread(targetexecCmd, args(cmd,))th.start()threads.append(th)for th in threads:th.join()print(程序结束%s % datetime.datetime.now())数据集下载 import datetime import os import threading from pathlib import Pathfrom huggingface_hub import HfApi from huggingface_hub.utils import filter_repo_objects# 执行命令 def execCmd(cmd):print(命令%s开始运行%s % (cmd, datetime.datetime.now()))os.system(cmd)print(命令%s结束运行%s % (cmd, datetime.datetime.now()))if name main:# 需下载的数据集IDdataset_id openai/webtext# 本地存储路径save_path ./webtext# 创建本地保存目录Path(save_path).mkdir(parentsTrue, exist_okTrue)# 获取数据集信息_api HfApi()dataset_info _api.dataset_info(dataset_iddataset_id,revisionmain,tokenNone,)# 获取文件信息filtered_dataset_files list(filter_repo_objects(items[f.rfilename for f in dataset_info.siblings],allow_patternsNone,ignore_patternsNone,))cmds []threads []# 需要执行的命令列表for file in filtered_dataset_files:# 获取路径url dataset_info.get_file_url(file)# 在本地创建子目录local_file os.path.join(save_path, file)local_dir os.path.dirname(local_file)Path(local_dir).mkdir(parentsTrue, exist_okTrue)# 断点下载指令cmds.append(fwget -c {url} -P {local_dir})print(cmds)print(程序开始%s % datetime.datetime.now())for cmd in cmds:th threading.Thread(targetexecCmd, args(cmd,))th.start()threads.append(th)for th in threads:th.join()print(程序结束%s % datetime.datetime.now())不足之处 不支持需要授权的库。 文件太多可能会开很多线程。 创作不易观众老爷们请留步… 动起可爱的小手点个赞再走呗 (๑◕ܫ๑) 欢迎大家关注笔者你的关注是我持续更博的最大动力 原创文章转载告知盗版必究 ♠ ⊕ ♠ ⊕ ♠ ⊕ ♠ ⊕ ♠ ⊕ ♠ ⊕ ♠ ⊕ ♠ ⊕ ♠ ⊕ ♠ ⊕ ♠ ⊕ ♠ ⊕ ♠ ⊕ ♠ ⊕ ♠ ⊕ ♠ ⊕ ♠ ⊕ ♠ ⊕ ♠ ⊕ ♠ ⊕ ♠ ⊕ ♠ ⊕ ♠ ⊕ ♠ ⊕ ♠ ⊕ ♠ ⊕ ♠ ⊕ ♠ ⊕ ♠ ⊕ ♠ ⊕ ♠