Community
modelplane
By modelplane
Last changed 4 days ago
Source Code

Notice something off about this package? Help us keep the marketplace safe and trustworthy by reporting inappropriate content or behavior.

Report this package
Overview
An open-source AI inference platform built on Crossplane.

Modelplane v0.7.0

Modelplane now provisions inference clusters on CoreWeave. Set an InferenceCluster's source to CoreWeave and Modelplane creates a CoreWeave Kubernetes Service cluster with its VPC and node pools, then installs the serving stack. A ModelDeployment runs there unchanged. Thanks to CoreWeave for the GPU capacity we built and tested this on, and to their engineers for working through the bugs with us.

v0.7 also fixes ways a model could stop serving. A Standalone engine's rollout used to start a new pod before stopping an old one, which deadlocks on a full GPU pool. It now replaces one copy at a time, so an engine with one copy stops serving while its replacement starts. Each replica used to run one endpoint picker, and the cluster's gateway won't route without it, so draining the picker's node took the model offline. Replicas now run two pickers, spread across nodes where there's room, and a drain evicts only one at a time. The cluster's gateway proxy is still one pod, so a drain can still interrupt serving. A new InferenceGateway no longer returns HTTP 500s for its first model.

On EKS, AKS, Civo, Nebius and CoreWeave, GPU metrics now have the deployment, replica, role and engine labels of the pod using the GPU, so you can join them to that deployment's engine metrics.

Modelplane used to place replicas on AMD GPU pools a whole node at a time. Now small models can share AMD nodes, as they already could NVIDIA nodes.

Check out https://docs.modelplane.ai/getting-started/ to get started, and the v0.7 docs for the rest.

To try Modelplane without a cloud account, @tr0njavolta's local demo runs it on your laptop with fake GPUs and models small enough for a CPU.

Highlights

What's Changed

New Contributors

Full Changelog: https://github.com/modelplaneai/modelplane/compare/v0.6.0...v0.7.0