The default is, that app model files, which are found locally (in only one app), get automatically `pulled-back` into the `"/workspace/shared_models"` folder.<br>
From there they will be re-linked back not only to their own `pulled-back` model-type folder, but also will be linked back into all other corresponding app model-type folders.<br>
So the `pulled-back` model is automatically shared to all installed apps.
All apps can be provisioned in at least 2 Virtual Environment versions:<br>
- 'official' - This setup is "to the point' as defined and recommended by the app owners on GitHub.<br>
- 'latest' - This setup extends the 'official' Setup with the latest PyTorch and Cuda libraries, or<br>
- in the case of ComfyUI - provides also an additional set of pre-installed Custom-Nodes.
The user can choose from all available versions during Setup, or pre-select the VENV_VERSION, which should be installed via following ENV vars in the format `VENV_VERSION_<app_id>`.
If these ENV vars are not set/passed into the container, the App-Manager will provide an UI for selecting them during Setup:
**NOTE**: If it makes sense, additional app versions will be added to the MANIFEST later, e.g. 'experimental' versions ;-)<br>
## Future plans
**We also plan for additional Apps**:<br>
-@ostris`ai-toolkit` - another popular Trainer app, see https://github.com/ostris/ai-toolkit
-@MNeMoNiCuZ`joy-caption-batch` - a popular captioning app, see https://github.com/MNeMoNiCuZ/joy-caption-batch
for captioning with https://www.aimodels.fyi/models/huggingFace/llama-joycaption-alpha-two-hf-llava-fancyfeast
Such a captioning app adds very nicely with the need to have good captions for Trainers like `kohya_ss` and `ai-toolkit`, specifically when training custom LoRA `Flux.1` models.
## Better AI-Launcher Features
All Apps can now be also `refreshed` any time - with their `'Refresh Application'` button - to the latest GitHub state of their corresponding Repos. This will include refreshing repo sub-modules (as in the case with 'kohya_ss'), and also will refresh 'custom_nodes' (in the case of 'ComfyUI's installed 12 custom nodes). In the case of 'ComfyUI' also all custom module requirements will be updated to their latest definitions.<br>
Note however, that refreshing an app needs to `reset` its status to the state, as when it was last installed/cloned!<br>
That means that any changes in the `app_path` (existing files edited or new files added) get lost, including local model downloads into the various `models` sub-folders of the app!<br>
Before refreshing, the `Refresh Symlinks` code will be called to `pull-back` any locally downloaded model files,
and save them into the `'shared_models'` workspace folder, before the actual `reset` is done.<br>
So this operation is not 'light' and you should plan for that accordingly!<br>
Every App also can be `deleted` and installed as another version with its `'Delete Application'` button.<br>
When `deleting` an app, the same logic applies as during `refreshing` and app, and the same `Refresh Symlinks` code will be called to `pull-back` any locally downloaded model files, and save them into the `'shared_models'` workspace folder, before the actual deletion of the app is done.<br>
This should make it easier to switch between app versions, if needed ;-)<br>
`Downloading` and `Unpacking` of app versions runs with the fastest available options:
- Downloads:
ATM we use `aria2c --max-connection-per-server=16 --max-concurrent-downloads=16 --split=16` for downloading app version `TAR.gz` archives from a central `S3 bucket location`.<br>
- Unpacking:
The `TAR.gz` archives are compressed with `'gzip -9'` option to achieve the lowest possible archive file size during download, which at the same time still provides fast Unpacking rates.<br>
Unpacking the archives is done also as fast as possible with `PIGZ`, a parallel version of gzip. Although it only uses a single thread for decompression, it starts 3 additional threads for reading, writing, and check calculation.
- Verification:
All downloaded TAR archives are `SHA256 hash` checked for possible download corruptions.<br>
After Unpacking and after Cloning/Installing the app, both the `app_path` and also the `venv_path` of the app are checked for correct and expected folder sizes. That should help to detect corrupted installations, which - for any possible reason - did not finish their corresponding stage.<br>
This last verification part can also be done later at any time with the `'Check Application'` button of the app.<br>
If the check code detects wrong sizes for the APP or VENV folders, which are UNDER an expected minimum size of the app_path and venv_path, it offers to `delete` the app. `'Check Application'` shows a verification summary of the expected and actual APP and VENV folder sizes, and it also shows which version is currently installed and when it was last refeshed. It even shows you, when an updated app version exists online.
### Shared Models
`'Shared Models'` provides a very powerful and completely configurable `'mapping'` for all kind of 'model files, be it Checkpoints, LoRAs, Embeddings and many more, between a `'shared_models'` workspace folder, and **all** installed applications, be it the currently supported applications or **any custom app**.
The mapping is completely transparent, and can be configures with 3 different kind of `mapping JSON files`.
One map for the kinds of model types to share, another map for the installed app-path locations, and the third map `connecting` these two other maps. This allows **any** mapping to **any** app ;-)<br>
`'Shared Models'` supports file-symlinks for single-file models, but also folder-symlinks for folder-based models (e.g. most LLM models are provided as folders). The mapping supports both types of symlinks.
To further 'get started' with `'Shared Models'`, make sure to read the separate `README-SHARED-MODELS.txt` which also provides 3 sample scenarios in the form of 3 installable small TAR archives with 'test-dummy' models and a bash-script to install these test data files into your `'/workspace'` folder.
This readme file, bash-script and 3 archives can be found in the `'/app/tests'` folder within the container (or source-code):
$ tree /app/tests
/app/tests
├── README-SHARED_MODELS.txt
├── populate_testdata.sh
├── testdata_installed_apps_pull.tar.gz
├── testdata_shared_models_link.tar.gz
└── testdata_stable-diffusion-webui_pull.tar.gz
1 directory, 5 files
### Model Downloader
We also provide an intelligent `Model Downloader` to download all types of models directly into the `'shared_models'` workspace, from where these models will be automatically shared across all installed application, and mapped intelligently into their according (different named) local app model folders.<br>
This `Model Downloader` currently supports `HuggingFace` and `CivitAI` download Urls and - in the case of CivitAI - has a very smart `CivitAi Model and Version Picker Dialog`, to choose between all available 'Versions', and from any selected Version between all its available 'Files', of a specified given CivitAI Model Id Url.<br>
On the `'Models'` tab of the App-Manager, some `Example URLs` for popular Models are provided both for `HuggingFace` and for `CivitAI`.
The `Model Downloader` supports also the use of `HuggingFace` and/or `CivitAI``security tokens`, which can be provided as `ENV vars` (see below), or stored in hidden files in the workspace, or as one-time security tokens only available in memory in the web-form during model download.<br>
This allows downloading `private models` and also `gated models` from both `HuggingFace` and `CivitAI`.
The HF_TOKEN need to be a `READ/WRITE` scoped token, if you plan also to **UPLOAD** models to `HuggingFace` later, when we support direct uploads of your trained models from Trainer Apps like `kohya_ss` or later from `ai-toolkit`.
These three, user-specific and **security sensitive environment vars**, should be stored as RUNPOD **`SECRETS`** and be referenced directly in your POD Template in the format `{{ RUNPOD_SECRET_MYENVVAR }}`.
You can reference your Secret directly in the Environment Variables section of your Pod template. To reference your Secret, reference it's key appended to the `RUNPOD_SECRET_` prefix.
