Usage
Options
number
default:"0.0"
minimum floor load in perf units/s (token/s for LLms)
number
default:"0.0"
minimum floor load in perf units/s (token/s for LLms), but allow handling with cold workers
number
default:"0.9"
target capacity utilization (fraction, max 1.0, default 0.9)
number
default:"2.5"
cold/stopped instance capacity target as multiple of hot capacity target (default 2.5)
integer
default:"5"
min number of workers to keep ‘cold’ when you have no load (default 5)
integer
default:"20"
max number of workers your endpoint group can have (default 20)
string
deployment endpoint name (allows multiple autoscale groups to share same deployment endpoint)
number
maximum seconds requests may be queued on each worker (default 30.0)
number
target seconds for the queue to be cleared (default 10.0)
integer
seconds of no traffic before the endpoint can scale to zero active workers
Description
Create a new endpoint group to manage many autoscaling groups Example: vastai create endpoint--target_util 0.9 --cold_mult 2.0 --endpoint_name “LLama”