metadata
base_model: sentence-transformers/all-MiniLM-L6-v2
datasets: []
language:
- en
library_name: sentence-transformers
license: apache-2.0
metrics:
- cosine_accuracy@1
- cosine_accuracy@3
- cosine_accuracy@5
- cosine_accuracy@10
- cosine_precision@1
- cosine_precision@3
- cosine_precision@5
- cosine_precision@10
- cosine_recall@1
- cosine_recall@3
- cosine_recall@5
- cosine_recall@10
- cosine_ndcg@10
- cosine_mrr@10
- cosine_map@100
pipeline_tag: sentence-similarity
tags:
- sentence-transformers
- sentence-similarity
- feature-extraction
- generated_from_trainer
- dataset_size:1490
- loss:MatryoshkaLoss
- loss:MultipleNegativesRankingLoss
widget:
- source_sentence: >-
Can you explain how to configure the credentials for authentication to a
remote MLflow tracking server in ZenML?
sentences:
- >-
w_bucket=gs://my_bucket --provider=<YOUR_PROVIDER>You can pass other
configurations specific to the stack components as key-value arguments.
If you don't provide a name, a random one is generated for you. For more
information about how to work use the CLI for this, please refer to the
dedicated documentation section.
Authentication Methods
You need to configure the following credentials for authentication to a
remote MLflow tracking server:
tracking_uri: The URL pointing to the MLflow tracking server. If using
an MLflow Tracking Server managed by Databricks, then the value of this
attribute should be "databricks".
tracking_username: Username for authenticating with the MLflow tracking
server.
tracking_password: Password for authenticating with the MLflow tracking
server.
tracking_token (in place of tracking_username and tracking_password):
Token for authenticating with the MLflow tracking server.
tracking_insecure_tls (optional): Set to skip verifying the MLflow
tracking server SSL certificate.
databricks_host: The host of the Databricks workspace with the
MLflow-managed server to connect to. This is only required if the
tracking_uri value is set to "databricks". More information: Access the
MLflow tracking server from outside Databricks
Either tracking_token or tracking_username and tracking_password must be
specified.
This option configures the credentials for the MLflow tracking service
directly as stack component attributes.
This is not recommended for production settings as the credentials won't
be stored securely and will be clearly visible in the stack
configuration.
zenml experiment-tracker register mlflow_experiment_tracker
--flavor=mlflow \
--tracking_uri=<URI> --tracking_token=<token>
--flavor=mlflow \
--tracking_password=<PASSWORD>
- >-
token_hex
token_hex(32)or:Copyopenssl rand -hex 32Important: If you configure
encryption for your SQL database secrets store, you should keep the
ZENML_SECRETS_STORE_ENCRYPTION_KEY value somewhere safe and secure, as
it will always be required by the ZenML server to decrypt the secrets in
the database. If you lose the encryption key, you will not be able to
decrypt the secrets in the database and will have to reset them.
These configuration options are only relevant if you're using the AWS
Secrets Manager as the secrets store backend.
ZENML_SECRETS_STORE_TYPE: Set this to aws in order to set this type of
secret store.
The AWS Secrets Store uses the ZenML AWS Service Connector under the
hood to authenticate with the AWS Secrets Manager API. This means that
you can use any of the authentication methods supported by the AWS
Service Connector to authenticate with the AWS Secrets Manager API.
"Version": "2012-10-17",
"Statement": [
"Sid": "ZenMLSecretsStore",
"Effect": "Allow",
"Action": [
"secretsmanager:CreateSecret",
"secretsmanager:GetSecretValue",
"secretsmanager:DescribeSecret",
"secretsmanager:PutSecretValue",
"secretsmanager:TagResource",
"secretsmanager:DeleteSecret"
],
"Resource":
"arn:aws:secretsmanager:<AWS-region>:<AWS-account-id>:secret:zenml/*"
The following configuration options are supported:
ZENML_SECRETS_STORE_AUTH_METHOD: The AWS Service Connector
authentication method to use (e.g. secret-key or iam-role).
ZENML_SECRETS_STORE_AUTH_CONFIG: The AWS Service Connector
configuration, in JSON format (e.g.
{"aws_access_key_id":"<aws-key-id>","aws_secret_access_key":"<aws-secret-key>","region":"<aws-region>"}).
Note: The remaining configuration options are deprecated and may be
removed in a future release. Instead, you should set the
ZENML_SECRETS_STORE_AUTH_METHOD and ZENML_SECRETS_STORE_AUTH_CONFIG
variables to use the AWS Service Connector authentication method.
- >-
tive Directory credentials or generic OIDC tokens.This authentication
method only requires a GCP workload identity external account JSON file
that only contains the configuration for the external account without
any sensitive credentials. It allows implementing a two layer
authentication scheme that keeps the set of permissions associated with
implicit credentials down to the bare minimum and grants permissions to
the privilege-bearing GCP service account instead.
This authentication method can be used to authenticate to GCP services
using credentials from other cloud providers or identity providers. When
used with workloads running on AWS or Azure, it involves automatically
picking up credentials from the AWS IAM or Azure AD identity associated
with the workload and using them to authenticate to GCP services. This
means that the result depends on the environment where the ZenML server
is deployed and is thus not fully reproducible.
When used with AWS or Azure implicit in-cloud authentication, this
method may constitute a security risk, because it can give users access
to the identity (e.g. AWS IAM role or Azure AD principal) implicitly
associated with the environment where the ZenML server is running. For
this reason, all implicit authentication methods are disabled by default
and need to be explicitly enabled by setting the
ZENML_ENABLE_IMPLICIT_AUTH_METHODS environment variable or the helm
chart enableImplicitAuthMethods configuration option to true in the
ZenML deployment.
By default, the GCP connector generates temporary OAuth 2.0 tokens from
the external account credentials and distributes them to clients. The
tokens have a limited lifetime of 1 hour. This behavior can be disabled
by setting the generate_temporary_tokens configuration option to False,
in which case, the connector will distribute the external account
credentials JSON to clients instead (not recommended).
- source_sentence: What is an example of a ZenML server YAML configuration file?
sentences:
- >-
sing a type annotation.
Tuple vs multiple outputsIt is impossible for ZenML to detect whether
you want your step to have a single output artifact of type Tuple or
multiple output artifacts just by looking at the type annotation.
We use the following convention to differentiate between the two: When
the return statement is followed by a tuple literal (e.g. return 1, 2 or
return (value_1, value_2)) we treat it as a step with multiple outputs.
All other cases are treated as a step with a single output of type
Tuple.
from zenml import step
from typing_extensions import Annotated
from typing import Tuple
@step
def my_step() -> Tuple[int, int]:
output_value = (0, 1)
return output_value
@step
def my_step(condition) -> Tuple[int, ...]:
if condition:
output_value = (0, 1)
else:
output_value = (0, 1, 2)
return output_value
@step
def my_step() -> Annotated[Tuple[int, ...], "my_output"]:
return 0, 1
@step
def my_step() -> Tuple[int, int]:
return 0, 1
@step
def my_step() -> Tuple[int, ...]:
return 0, 1
Step output names
By default, ZenML uses the output name output for single output steps
and output_0, output_1, ... for steps with multiple outputs. These
output names are used to display your outputs in the dashboard and fetch
them after your pipeline is finished.
If you want to use custom output names for your steps, use the Annotated
type annotation:
from typing_extensions import Annotated
Annotated on Python 3.9+
from typing import Tuple
from zenml import step
@step
def square_root(number: int) -> Annotated[float, "custom_output_name"]:
return number ** 0.5
@step
def divide(a: int, b: int) -> Tuple[
Annotated[int, "quotient"],
Annotated[int, "remainder"]
]:
return a // b, a % b
- >-
HyperAI Orchestrator
Orchestrating your pipelines to run on HyperAI.ai instances.
HyperAI is a cutting-edge cloud compute platform designed to make AI
accessible for everyone. The HyperAI orchestrator is an orchestrator
flavor that allows you to easily deploy your pipelines on HyperAI
instances.
This component is only meant to be used within the context of a remote
ZenML deployment scenario. Usage with a local ZenML deployment may lead
to unexpected behavior!
When to use it
You should use the HyperAI orchestrator if:
you're looking for a managed solution for running your pipelines.
you're a HyperAI customer.
Prerequisites
You will need to do the following to start using the HyperAI
orchestrator:
Have a running HyperAI instance. It must be accessible from the internet
(or at least from the IP addresses of your ZenML users) and allow SSH
key based access (passwords are not supported).
Ensure that a recent version of Docker is installed. This version must
include Docker Compose, meaning that the command docker compose works.
Ensure that the appropriate NVIDIA Driver is installed on the HyperAI
instance (if not already installed by the HyperAI team).
Ensure that the NVIDIA Container Toolkit is installed and configured on
the HyperAI instance.
Note that it is possible to omit installing the NVIDIA Driver and NVIDIA
Container Toolkit. However, you will then be unable to use the GPU from
within your ZenML pipeline. Additionally, you will then need to disable
GPU access within the container when configuring the Orchestrator
component, or the pipeline will not start correctly.
How it works
- >-
fied, or a string, in which case it must be a path# to a CA certificate
bundle to use or the CA bundle value itself
verify_ssl:
Here is an example of a ZenML server YAML configuration file:
url:
https://ac8ef63af203226194a7725ee71d85a-7635928635.us-east-1.elb.amazonaws.com/zenml
verify_ssl: |
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
To disconnect from the current ZenML server and revert to using the
local default database, use the following command:
zenml disconnect
How does it work?
Here's an architecture diagram that shows how the workflow looks like
when you do zenml deploy.
The deploy CLI makes use of a "recipe" inside the zenml-io/zenml
repository to deploy the server on the right cloud. Any configuration
that you pass with the CLI, is sent to the recipe as input variables.
PreviousDeploying ZenML
NextDeploy with Docker
Last updated 15 days ago
- source_sentence: When should I update my service account name to ensure security?
sentences:
- >-
y <SERVICE_ACCOUNT_NAME> update.
Important noticeEvery API key issued is a potential gateway to access
your data, secrets and infrastructure. It's important to regularly
rotate API keys and deactivate or delete service accounts and API keys
that are no longer needed.
PreviousConnect in with your User (interactive)
NextInteract with secrets
Last updated 15 days ago
- >-
Connect in with your User (interactive)
You can authenticate your clients with the ZenML Server using the ZenML
CLI and the web based login. This can be executed with the command:
zenml connect --url https://...
This command will start a series of steps to validate the device from
where you are connecting that will happen in your browser. You can
choose whether to mark your respective device as trusted or not. If you
choose not to click Trust this device, a 24-hour token will be issued
for authentication services. Choosing to trust the device will issue a
30-day token instead.
To see all devices you've permitted, use the following command:
zenml authorized-device list
Additionally, the following command allows you to more precisely inspect
one of these devices:
zenml authorized-device describe <DEVICE_ID>
For increased security, you can invalidate a token using the zenml
device lock command followed by the device ID. This helps provide an
extra layer of security and control over your devices.
zenml authorized-device lock <DEVICE_ID>
To keep things simple, we can summarize the steps:
Use the zenml connect --url command to start a device flow and connect
to a zenml server.
Choose whether to trust the device when prompted.
Check permitted devices with zenml devices list.
Invalidate a token with zenml device lock ....
Important notice
Using the ZenML CLI is a secure and comfortable way to interact with
your ZenML tenants. It's important to always ensure that only trusted
devices are used to maintain security and privacy.
Don't forget to manage your device trust levels regularly for optimal
security. Should you feel a device trust needs to be revoked, lock the
device immediately. Every token issued is a potential gateway to access
your data, secrets and infrastructure.
PreviousConnect to a server
NextConnect with a Service Account
Last updated 19 days ago
- >-
━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━┷━━━━━━━┷━━━━━━━━┛A lot more is hidden
behind a Service Connector Type than a name and a simple list of
resource types. Before using a Service Connector Type to configure a
Service Connector, you probably need to understand what it is, what it
can offer and what are the supported authentication methods and their
requirements. All this can be accessed directly through the CLI. Some
examples are included here.
Showing information about the gcp Service Connector Type:
zenml service-connector describe-type gcp
Example Command Output
╔══════════════════════════════════════════════════════════════════════════════╗
║ 🔵 GCP Service Connector (connector type:
gcp) ║
╚══════════════════════════════════════════════════════════════════════════════╝
Authentication methods:
🔒 implicit
🔒 user-account
🔒 service-account
🔒 oauth2-token
🔒 impersonation
Resource types:
🔵 gcp-generic
📦 gcs-bucket
🌀 kubernetes-cluster
🐳 docker-registry
Supports auto-configuration: True
Available locally: True
Available remotely: True
The ZenML GCP Service Connector facilitates the authentication and
access to
managed GCP services and resources. These encompass a range of
resources,
including GCS buckets, GCR container repositories and GKE clusters. The
connector provides support for various authentication methods, including
GCP
user accounts, service accounts, short-lived OAuth 2.0 tokens and
implicit
authentication.
To ensure heightened security measures, this connector always issues
short-lived
OAuth 2.0 tokens to clients instead of long-lived credentials.
Furthermore, it
includes automatic configuration and detection of credentials locally
configured through the GCP CLI.
This connector serves as a general means of accessing any GCP service by
issuing
OAuth 2.0 credential objects to clients. Additionally, the connector can
handle
specialized authentication for GCS, Docker and Kubernetes Python
clients. It
- source_sentence: >-
Where can I find the instructions to clone the ZenML quickstart repository
and set up the stack?
sentences:
- >-
into play when the component is ultimately in use.The design behind this
interaction lets us separate the configuration of the flavor from its
implementation. This way we can register flavors and components even
when the major dependencies behind their implementation are not
installed in our local setting (assuming the CustomArtifactStoreFlavor
and the CustomArtifactStoreConfig are implemented in a different
module/path than the actual CustomArtifactStore).
Enabling Artifact Visualizations with Custom Artifact Stores
ZenML automatically saves visualizations for many common data types and
allows you to view these visualizations in the ZenML dashboard. Under
the hood, this works by saving the visualizations together with the
artifacts in the artifact store.
In order to load and display these visualizations, ZenML needs to be
able to load and access the corresponding artifact store. This means
that your custom artifact store needs to be configured in a way that
allows authenticating to the back-end without relying on the local
environment, e.g., by embedding the authentication credentials in the
stack component configuration or by referencing a secret.
Furthermore, for deployed ZenML instances, you need to install the
package dependencies of your artifact store implementation in the
environment where you have deployed ZenML. See the Documentation on
deploying ZenML with custom Docker images for more information on how to
do that.
PreviousAzure Blob Storage
NextContainer Registries
Last updated 19 days ago
- >-
t_repository: str
user: Optional[str]
resources:cpu_count: Optional[PositiveFloat]
gpu_count: Optional[NonNegativeInt]
memory: Optional[ConstrainedStrValue]
step_operator: Optional[str]
success_hook_source:
attribute: Optional[str]
module: str
type: SourceType
train_model:
enable_artifact_metadata: Optional[bool]
enable_artifact_visualization: Optional[bool]
enable_cache: Optional[bool]
enable_step_logs: Optional[bool]
experiment_tracker: Optional[str]
extra: Mapping[str, Any]
failure_hook_source:
attribute: Optional[str]
module: str
type: SourceType
model:
audience: Optional[str]
description: Optional[str]
ethics: Optional[str]
license: Optional[str]
limitations: Optional[str]
name: str
save_models_to_registry: bool
suppress_class_validation_warnings: bool
tags: Optional[List[str]]
trade_offs: Optional[str]
use_cases: Optional[str]
version: Union[ModelStages, int, str, NoneType]
was_created_in_this_run: bool
name: Optional[str]
outputs: {}
parameters: {}
settings:
docker:
apt_packages: List[str]
build_context_root: Optional[str]
build_options: Mapping[str, Any]
copy_files: bool
copy_global_config: bool
dockerfile: Optional[str]
dockerignore: Optional[str]
environment: Mapping[str, Any]
install_stack_requirements: bool
parent_image: Optional[str]
python_package_installer: PythonPackageInstaller
replicate_local_python_environment: Union[List[str],
PythonEnvironmentExportMethod,
NoneType]
required_hub_plugins: List[str]
required_integrations: List[str]
requirements: Union[NoneType, str, List[str]]
skip_build: bool
source_files: SourceFileMode
target_repository: str
user: Optional[str]
resources:
cpu_count: Optional[PositiveFloat]
gpu_count: Optional[NonNegativeInt]
memory: Optional[ConstrainedStrValue]
step_operator: Optional[str]
success_hook_source:
attribute: Optional[str]
module: str
type: SourceType
- >-
as the ZenML quickstart. You can clone it like so:git clone --depth 1
git@github.com:zenml-io/zenml.git
cd zenml/examples/quickstart
pip install -r requirements.txt
zenml init
To run a pipeline using the new stack:
Set the stack as active on your clientCopyzenml stack set
a_new_local_stack
Run your pipeline code:Copypython run.py --training-pipeline
Keep this code handy as we'll be using it in the next chapters!
PreviousDeploying ZenML
NextConnecting remote storage
Last updated 19 days ago
- source_sentence: >-
How do I register and connect an S3 artifact store in ZenML using the
interactive mode?
sentences:
- >-
hich Resource Name to use in the interactive mode:zenml artifact-store
register s3-zenfiles --flavor s3 --path=s3://zenfiles
zenml service-connector list-resources --resource-type s3-bucket
--resource-id s3://zenfiles
zenml artifact-store connect s3-zenfiles --connector aws-multi-type
Example Command Output
$ zenml artifact-store register s3-zenfiles --flavor s3
--path=s3://zenfiles
Running with active workspace: 'default' (global)
Running with active stack: 'default' (global)
Successfully registered artifact_store `s3-zenfiles`.
$ zenml service-connector list-resources --resource-type s3-bucket
--resource-id zenfiles
The 's3-bucket' resource with name 'zenfiles' can be accessed by
service connectors configured in your workspace:
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┓
┃ CONNECTOR ID │ CONNECTOR NAME │
CONNECTOR TYPE │ RESOURCE TYPE │ RESOURCE NAMES ┃
┠──────────────────────────────────────┼──────────────────────┼────────────────┼───────────────┼────────────────┨
┃ 4a550c82-aa64-4a48-9c7f-d5e127d77a44 │ aws-multi-type │ 🔶
aws │ 📦 s3-bucket │ s3://zenfiles ┃
┠──────────────────────────────────────┼──────────────────────┼────────────────┼───────────────┼────────────────┨
┃ 66c0922d-db84-4e2c-9044-c13ce1611613 │ aws-multi-instance │ 🔶
aws │ 📦 s3-bucket │ s3://zenfiles ┃
┠──────────────────────────────────────┼──────────────────────┼────────────────┼───────────────┼────────────────┨
┃ 65c82e59-cba0-4a01-b8f6-d75e8a1d0f55 │ aws-single-instance │ 🔶
aws │ 📦 s3-bucket │ s3://zenfiles ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┛
$ zenml artifact-store connect s3-zenfiles --connector aws-multi-type
Running with active workspace: 'default' (global)
Running with active stack: 'default' (global)
Successfully connected artifact store `s3-zenfiles` to the following
resources:
- >-
👣Step Operators
Executing individual steps in specialized environments.
The step operator enables the execution of individual pipeline steps in
specialized runtime environments that are optimized for certain
workloads. These specialized environments can give your steps access to
resources like GPUs or distributed processing frameworks like Spark.
Comparison to orchestrators: The orchestrator is a mandatory stack
component that is responsible for executing all steps of a pipeline in
the correct order and providing additional features such as scheduling
pipeline runs. The step operator on the other hand is used to only
execute individual steps of the pipeline in a separate environment in
case the environment provided by the orchestrator is not feasible.
When to use it
A step operator should be used if one or more steps of a pipeline
require resources that are not available in the runtime environments
provided by the orchestrator. An example would be a step that trains a
computer vision model and requires a GPU to run in a reasonable time,
combined with a Kubeflow orchestrator running on a Kubernetes cluster
that does not contain any GPU nodes. In that case, it makes sense to
include a step operator like SageMaker, Vertex, or AzureML to execute
the training step with a GPU.
Step Operator Flavors
Step operators to execute steps on one of the big cloud providers are
provided by the following ZenML integrations:
Step Operator Flavor Integration Notes SageMaker sagemaker aws Uses
SageMaker to execute steps Vertex vertex gcp Uses Vertex AI to execute
steps AzureML azureml azure Uses AzureML to execute steps Spark spark
spark Uses Spark on Kubernetes to execute steps in a distributed manner
Custom Implementation custom Extend the step operator abstraction and
provide your own implementation
If you would like to see the available flavors of step operators, you
can use the command:
zenml step-operator flavor list
How to use it
- >-
Azure Container Registry
Storing container images in Azure.
The Azure container registry is a container registry flavor that comes
built-in with ZenML and uses the Azure Container Registry to store
container images.
When to use it
You should use the Azure container registry if:
one or more components of your stack need to pull or push container
images.
you have access to Azure. If you're not using Azure, take a look at the
other container registry flavors.
How to deploy it
Go here and choose a subscription, resource group, location, and
registry name. Then click on Review + Create and to create your
container registry.
How to find the registry URI
The Azure container registry URI should have the following format:
<REGISTRY_NAME>.azurecr.io
zenmlregistry.azurecr.io
myregistry.azurecr.io
To figure out the URI for your registry:
Go to the Azure portal.
In the search bar, enter container registries and select the container
registry you want to use. If you don't have any container registries
yet, check out the deployment section on how to create one.
Use the name of your registry to fill the template
<REGISTRY_NAME>.azurecr.io and get your URI.
How to use it
To use the Azure container registry, we need:
Docker installed and running.
The registry URI. Check out the previous section on the URI format and
how to get the URI for your registry.
We can then register the container registry and use it in our active
stack:
zenml container-registry register <NAME> \
--flavor=azure \
--uri=<REGISTRY_URI>
zenml stack update -c <NAME>
You also need to set up authentication required to log in to the
container registry.
Authentication Methods
model-index:
- name: zenml/finetuned-all-MiniLM-L6-v2
results:
- task:
type: information-retrieval
name: Information Retrieval
dataset:
name: dim 384
type: dim_384
metrics:
- type: cosine_accuracy@1
value: 0.3132530120481928
name: Cosine Accuracy@1
- type: cosine_accuracy@3
value: 0.6144578313253012
name: Cosine Accuracy@3
- type: cosine_accuracy@5
value: 0.7168674698795181
name: Cosine Accuracy@5
- type: cosine_accuracy@10
value: 0.7891566265060241
name: Cosine Accuracy@10
- type: cosine_precision@1
value: 0.3132530120481928
name: Cosine Precision@1
- type: cosine_precision@3
value: 0.20481927710843373
name: Cosine Precision@3
- type: cosine_precision@5
value: 0.1433734939759036
name: Cosine Precision@5
- type: cosine_precision@10
value: 0.0789156626506024
name: Cosine Precision@10
- type: cosine_recall@1
value: 0.3132530120481928
name: Cosine Recall@1
- type: cosine_recall@3
value: 0.6144578313253012
name: Cosine Recall@3
- type: cosine_recall@5
value: 0.7168674698795181
name: Cosine Recall@5
- type: cosine_recall@10
value: 0.7891566265060241
name: Cosine Recall@10
- type: cosine_ndcg@10
value: 0.5579120329651274
name: Cosine Ndcg@10
- type: cosine_mrr@10
value: 0.48292933639319197
name: Cosine Mrr@10
- type: cosine_map@100
value: 0.4907452723782479
name: Cosine Map@100
- task:
type: information-retrieval
name: Information Retrieval
dataset:
name: dim 256
type: dim_256
metrics:
- type: cosine_accuracy@1
value: 0.2891566265060241
name: Cosine Accuracy@1
- type: cosine_accuracy@3
value: 0.6144578313253012
name: Cosine Accuracy@3
- type: cosine_accuracy@5
value: 0.7108433734939759
name: Cosine Accuracy@5
- type: cosine_accuracy@10
value: 0.7650602409638554
name: Cosine Accuracy@10
- type: cosine_precision@1
value: 0.2891566265060241
name: Cosine Precision@1
- type: cosine_precision@3
value: 0.20481927710843373
name: Cosine Precision@3
- type: cosine_precision@5
value: 0.14216867469879516
name: Cosine Precision@5
- type: cosine_precision@10
value: 0.07650602409638553
name: Cosine Precision@10
- type: cosine_recall@1
value: 0.2891566265060241
name: Cosine Recall@1
- type: cosine_recall@3
value: 0.6144578313253012
name: Cosine Recall@3
- type: cosine_recall@5
value: 0.7108433734939759
name: Cosine Recall@5
- type: cosine_recall@10
value: 0.7650602409638554
name: Cosine Recall@10
- type: cosine_ndcg@10
value: 0.5394043126982406
name: Cosine Ndcg@10
- type: cosine_mrr@10
value: 0.46553595333715836
name: Cosine Mrr@10
- type: cosine_map@100
value: 0.4739275972429515
name: Cosine Map@100
- task:
type: information-retrieval
name: Information Retrieval
dataset:
name: dim 128
type: dim_128
metrics:
- type: cosine_accuracy@1
value: 0.28313253012048195
name: Cosine Accuracy@1
- type: cosine_accuracy@3
value: 0.5481927710843374
name: Cosine Accuracy@3
- type: cosine_accuracy@5
value: 0.6506024096385542
name: Cosine Accuracy@5
- type: cosine_accuracy@10
value: 0.7168674698795181
name: Cosine Accuracy@10
- type: cosine_precision@1
value: 0.28313253012048195
name: Cosine Precision@1
- type: cosine_precision@3
value: 0.1827309236947791
name: Cosine Precision@3
- type: cosine_precision@5
value: 0.1301204819277108
name: Cosine Precision@5
- type: cosine_precision@10
value: 0.07168674698795179
name: Cosine Precision@10
- type: cosine_recall@1
value: 0.28313253012048195
name: Cosine Recall@1
- type: cosine_recall@3
value: 0.5481927710843374
name: Cosine Recall@3
- type: cosine_recall@5
value: 0.6506024096385542
name: Cosine Recall@5
- type: cosine_recall@10
value: 0.7168674698795181
name: Cosine Recall@10
- type: cosine_ndcg@10
value: 0.5067699591037801
name: Cosine Ndcg@10
- type: cosine_mrr@10
value: 0.43858529355517323
name: Cosine Mrr@10
- type: cosine_map@100
value: 0.44791284428498435
name: Cosine Map@100
- task:
type: information-retrieval
name: Information Retrieval
dataset:
name: dim 64
type: dim_64
metrics:
- type: cosine_accuracy@1
value: 0.24096385542168675
name: Cosine Accuracy@1
- type: cosine_accuracy@3
value: 0.46987951807228917
name: Cosine Accuracy@3
- type: cosine_accuracy@5
value: 0.5843373493975904
name: Cosine Accuracy@5
- type: cosine_accuracy@10
value: 0.6807228915662651
name: Cosine Accuracy@10
- type: cosine_precision@1
value: 0.24096385542168675
name: Cosine Precision@1
- type: cosine_precision@3
value: 0.1566265060240964
name: Cosine Precision@3
- type: cosine_precision@5
value: 0.11686746987951806
name: Cosine Precision@5
- type: cosine_precision@10
value: 0.06807228915662648
name: Cosine Precision@10
- type: cosine_recall@1
value: 0.24096385542168675
name: Cosine Recall@1
- type: cosine_recall@3
value: 0.46987951807228917
name: Cosine Recall@3
- type: cosine_recall@5
value: 0.5843373493975904
name: Cosine Recall@5
- type: cosine_recall@10
value: 0.6807228915662651
name: Cosine Recall@10
- type: cosine_ndcg@10
value: 0.45307543718220417
name: Cosine Ndcg@10
- type: cosine_mrr@10
value: 0.3806679097341751
name: Cosine Mrr@10
- type: cosine_map@100
value: 0.389050349953244
name: Cosine Map@100
zenml/finetuned-all-MiniLM-L6-v2
This is a sentence-transformers model finetuned from sentence-transformers/all-MiniLM-L6-v2. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.
Model Details
Model Description
- Model Type: Sentence Transformer
- Base model: sentence-transformers/all-MiniLM-L6-v2
- Maximum Sequence Length: 256 tokens
- Output Dimensionality: 384 tokens
- Similarity Function: Cosine Similarity
- Language: en
- License: apache-2.0
Model Sources
Full Model Architecture
SentenceTransformer(
(0): Transformer({'max_seq_length': 256, 'do_lower_case': False}) with Transformer model: BertModel
(1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
(2): Normalize()
)
Usage
Direct Usage (Sentence Transformers)
First install the Sentence Transformers library:
pip install -U sentence-transformers
Then you can load this model and run inference.
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("zenml/finetuned-all-MiniLM-L6-v2")
sentences = [
'How do I register and connect an S3 artifact store in ZenML using the interactive mode?',
"hich Resource Name to use in the interactive mode:zenml artifact-store register s3-zenfiles --flavor s3 --path=s3://zenfiles\n\nzenml service-connector list-resources --resource-type s3-bucket --resource-id s3://zenfiles\n\nzenml artifact-store connect s3-zenfiles --connector aws-multi-type\n\nExample Command Output\n\n$ zenml artifact-store register s3-zenfiles --flavor s3 --path=s3://zenfiles\n\nRunning with active workspace: 'default' (global)\n\nRunning with active stack: 'default' (global)\n\nSuccessfully registered artifact_store `s3-zenfiles`.\n\n$ zenml service-connector list-resources --resource-type s3-bucket --resource-id zenfiles\n\nThe 's3-bucket' resource with name 'zenfiles' can be accessed by service connectors configured in your workspace:\n\n┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┓\n\n┃ CONNECTOR ID │ CONNECTOR NAME │ CONNECTOR TYPE │ RESOURCE TYPE │ RESOURCE NAMES ┃\n\n┠──────────────────────────────────────┼──────────────────────┼────────────────┼───────────────┼────────────────┨\n\n┃ 4a550c82-aa64-4a48-9c7f-d5e127d77a44 │ aws-multi-type │ 🔶 aws │ 📦 s3-bucket │ s3://zenfiles ┃\n\n┠──────────────────────────────────────┼──────────────────────┼────────────────┼───────────────┼────────────────┨\n\n┃ 66c0922d-db84-4e2c-9044-c13ce1611613 │ aws-multi-instance │ 🔶 aws │ 📦 s3-bucket │ s3://zenfiles ┃\n\n┠──────────────────────────────────────┼──────────────────────┼────────────────┼───────────────┼────────────────┨\n\n┃ 65c82e59-cba0-4a01-b8f6-d75e8a1d0f55 │ aws-single-instance │ 🔶 aws │ 📦 s3-bucket │ s3://zenfiles ┃\n\n┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┛\n\n$ zenml artifact-store connect s3-zenfiles --connector aws-multi-type\n\nRunning with active workspace: 'default' (global)\n\nRunning with active stack: 'default' (global)\n\nSuccessfully connected artifact store `s3-zenfiles` to the following resources:",
"Azure Container Registry\n\nStoring container images in Azure.\n\nThe Azure container registry is a container registry flavor that comes built-in with ZenML and uses the Azure Container Registry to store container images.\n\nWhen to use it\n\nYou should use the Azure container registry if:\n\none or more components of your stack need to pull or push container images.\n\nyou have access to Azure. If you're not using Azure, take a look at the other container registry flavors.\n\nHow to deploy it\n\nGo here and choose a subscription, resource group, location, and registry name. Then click on Review + Create and to create your container registry.\n\nHow to find the registry URI\n\nThe Azure container registry URI should have the following format:\n\n<REGISTRY_NAME>.azurecr.io\n\n# Examples:\n\nzenmlregistry.azurecr.io\n\nmyregistry.azurecr.io\n\nTo figure out the URI for your registry:\n\nGo to the Azure portal.\n\nIn the search bar, enter container registries and select the container registry you want to use. If you don't have any container registries yet, check out the deployment section on how to create one.\n\nUse the name of your registry to fill the template <REGISTRY_NAME>.azurecr.io and get your URI.\n\nHow to use it\n\nTo use the Azure container registry, we need:\n\nDocker installed and running.\n\nThe registry URI. Check out the previous section on the URI format and how to get the URI for your registry.\n\nWe can then register the container registry and use it in our active stack:\n\nzenml container-registry register <NAME> \\\n\n--flavor=azure \\\n\n--uri=<REGISTRY_URI>\n\n# Add the container registry to the active stack\n\nzenml stack update -c <NAME>\n\nYou also need to set up authentication required to log in to the container registry.\n\nAuthentication Methods",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
Evaluation
Metrics
Information Retrieval
| Metric |
Value |
| cosine_accuracy@1 |
0.3133 |
| cosine_accuracy@3 |
0.6145 |
| cosine_accuracy@5 |
0.7169 |
| cosine_accuracy@10 |
0.7892 |
| cosine_precision@1 |
0.3133 |
| cosine_precision@3 |
0.2048 |
| cosine_precision@5 |
0.1434 |
| cosine_precision@10 |
0.0789 |
| cosine_recall@1 |
0.3133 |
| cosine_recall@3 |
0.6145 |
| cosine_recall@5 |
0.7169 |
| cosine_recall@10 |
0.7892 |
| cosine_ndcg@10 |
0.5579 |
| cosine_mrr@10 |
0.4829 |
| cosine_map@100 |
0.4907 |
Information Retrieval
| Metric |
Value |
| cosine_accuracy@1 |
0.2892 |
| cosine_accuracy@3 |
0.6145 |
| cosine_accuracy@5 |
0.7108 |
| cosine_accuracy@10 |
0.7651 |
| cosine_precision@1 |
0.2892 |
| cosine_precision@3 |
0.2048 |
| cosine_precision@5 |
0.1422 |
| cosine_precision@10 |
0.0765 |
| cosine_recall@1 |
0.2892 |
| cosine_recall@3 |
0.6145 |
| cosine_recall@5 |
0.7108 |
| cosine_recall@10 |
0.7651 |
| cosine_ndcg@10 |
0.5394 |
| cosine_mrr@10 |
0.4655 |
| cosine_map@100 |
0.4739 |
Information Retrieval
| Metric |
Value |
| cosine_accuracy@1 |
0.2831 |
| cosine_accuracy@3 |
0.5482 |
| cosine_accuracy@5 |
0.6506 |
| cosine_accuracy@10 |
0.7169 |
| cosine_precision@1 |
0.2831 |
| cosine_precision@3 |
0.1827 |
| cosine_precision@5 |
0.1301 |
| cosine_precision@10 |
0.0717 |
| cosine_recall@1 |
0.2831 |
| cosine_recall@3 |
0.5482 |
| cosine_recall@5 |
0.6506 |
| cosine_recall@10 |
0.7169 |
| cosine_ndcg@10 |
0.5068 |
| cosine_mrr@10 |
0.4386 |
| cosine_map@100 |
0.4479 |
Information Retrieval
| Metric |
Value |
| cosine_accuracy@1 |
0.241 |
| cosine_accuracy@3 |
0.4699 |
| cosine_accuracy@5 |
0.5843 |
| cosine_accuracy@10 |
0.6807 |
| cosine_precision@1 |
0.241 |
| cosine_precision@3 |
0.1566 |
| cosine_precision@5 |
0.1169 |
| cosine_precision@10 |
0.0681 |
| cosine_recall@1 |
0.241 |
| cosine_recall@3 |
0.4699 |
| cosine_recall@5 |
0.5843 |
| cosine_recall@10 |
0.6807 |
| cosine_ndcg@10 |
0.4531 |
| cosine_mrr@10 |
0.3807 |
| cosine_map@100 |
0.3891 |
Training Details
Training Dataset
Unnamed Dataset
- Size: 1,490 training samples
- Columns:
positive and anchor
- Approximate statistics based on the first 1000 samples:
|
positive |
anchor |
| type |
string |
string |
| details |
- min: 9 tokens
- mean: 21.23 tokens
- max: 49 tokens
|
- min: 23 tokens
- mean: 237.64 tokens
- max: 256 tokens
|
- Samples:
| positive |
anchor |
How can you leverage MLflow for tracking and visualizing experiment results in ZenML? |
MLflow
Logging and visualizing experiments with MLflow.
The MLflow Experiment Tracker is an Experiment Tracker flavor provided with the MLflow ZenML integration that uses the MLflow tracking service to log and visualize information from your pipeline steps (e.g. models, parameters, metrics).
When would you want to use it?
MLflow Tracking is a very popular tool that you would normally use in the iterative ML experimentation phase to track and visualize experiment results. That doesn't mean that it cannot be repurposed to track and visualize the results produced by your automated pipeline runs, as you make the transition toward a more production-oriented workflow.
You should use the MLflow Experiment Tracker:
if you have already been using MLflow to track experiment results for your project and would like to continue doing so as you are incorporating MLOps workflows and best practices in your project through ZenML.
if you are looking for a more visually interactive way of navigating the results produced from your ZenML pipeline runs (e.g. models, metrics, datasets)
if you or your team already have a shared MLflow Tracking service deployed somewhere on-premise or in the cloud, and you would like to connect ZenML to it to share the artifacts and metrics logged by your pipelines
You should consider one of the other Experiment Tracker flavors if you have never worked with MLflow before and would rather use another experiment tracking tool that you are more familiar with.
How do you deploy it?
The MLflow Experiment Tracker flavor is provided by the MLflow ZenML integration, you need to install it on your local machine to be able to register an MLflow Experiment Tracker and add it to your stack:
zenml integration install mlflow -y
The MLflow Experiment Tracker can be configured to accommodate the following MLflow deployment scenarios: |
What are the required integrations for running pipelines with a Docker-based orchestrator in ZenML? |
ctivated by installing the respective integration:Integration Materializer Handled Data Types Storage Format bentoml BentoMaterializer bentoml.Bento .bento deepchecks DeepchecksResultMateriailzer deepchecks.CheckResult , deepchecks.SuiteResult .json evidently EvidentlyProfileMaterializer evidently.Profile .json great_expectations GreatExpectationsMaterializer great_expectations.ExpectationSuite , great_expectations.CheckpointResult .json huggingface HFDatasetMaterializer datasets.Dataset , datasets.DatasetDict Directory huggingface HFPTModelMaterializer transformers.PreTrainedModel Directory huggingface HFTFModelMaterializer transformers.TFPreTrainedModel Directory huggingface HFTokenizerMaterializer transformers.PreTrainedTokenizerBase Directory lightgbm LightGBMBoosterMaterializer lgbm.Booster .txt lightgbm LightGBMDatasetMaterializer lgbm.Dataset .binary neural_prophet NeuralProphetMaterializer NeuralProphet .pt pillow PillowImageMaterializer Pillow.Image .PNG polars PolarsMaterializer pl.DataFrame , pl.Series .parquet pycaret PyCaretMaterializer Any sklearn , xgboost , lightgbm or catboost model .pkl pytorch PyTorchDataLoaderMaterializer torch.Dataset , torch.DataLoader .pt pytorch PyTorchModuleMaterializer torch.Module .pt scipy SparseMaterializer scipy.spmatrix .npz spark SparkDataFrameMaterializer pyspark.DataFrame .parquet spark SparkModelMaterializer pyspark.Transformer pyspark.Estimator tensorflow KerasMaterializer tf.keras.Model Directory tensorflow TensorflowDatasetMaterializer tf.Dataset Directory whylogs WhylogsMaterializer whylogs.DatasetProfileView .pb xgboost XgboostBoosterMaterializer xgb.Booster .json xgboost XgboostDMatrixMaterializer xgb.DMatrix .binary
If you are running pipelines with a Docker-based orchestrator, you need to specify the corresponding integration as required_integrations in the DockerSettings of your pipeline in order to have the integration materializer available inside your Docker container. See the pipeline configuration documentation for more information. |
What is the difference between the stack component settings at registration time and runtime for ZenML? |
ettings to specify AzureML step operator settings.Difference between stack component settings at registration-time vs real-time
For stack-component-specific settings, you might be wondering what the difference is between these and the configuration passed in while doing zenml stack-component register --config1=configvalue --config2=configvalue, etc. The answer is that the configuration passed in at registration time is static and fixed throughout all pipeline runs, while the settings can change.
A good example of this is the MLflow Experiment Tracker, where configuration which remains static such as the tracking_url is sent through at registration time, while runtime configuration such as the experiment_name (which might change every pipeline run) is sent through as runtime settings.
Even though settings can be overridden at runtime, you can also specify default values for settings while configuring a stack component. For example, you could set a default value for the nested setting of your MLflow experiment tracker: zenml experiment-tracker register --flavor=mlflow --nested=True
This means that all pipelines that run using this experiment tracker use nested MLflow runs unless overridden by specifying settings for the pipeline at runtime.
Using the right key for Stack-component-specific settings
When specifying stack-component-specific settings, a key needs to be passed. This key should always correspond to the pattern: .
For example, the SagemakerStepOperator supports passing in estimator_args. The way to specify this would be to use the key step_operator.sagemaker
@step(step_operator="nameofstepoperator", settings= {"step_operator.sagemaker": {"estimator_args": {"instance_type": "m7g.medium"}}})
def my_step():
...
# Using the class
@step(step_operator="nameofstepoperator", settings= {"step_operator.sagemaker": SagemakerStepOperatorSettings(instance_type="m7g.medium")})
def my_step():
...
or in YAML:
steps:
my_step: |
- Loss:
MatryoshkaLoss with these parameters:{
"loss": "MultipleNegativesRankingLoss",
"matryoshka_dims": [
384,
256,
128,
64
],
"matryoshka_weights": [
1,
1,
1,
1
],
"n_dims_per_step": -1
}
Training Hyperparameters
Non-Default Hyperparameters
eval_strategy: epochper_device_train_batch_size: 32per_device_eval_batch_size: 16gradient_accumulation_steps: 16learning_rate: 2e-05num_train_epochs: 4lr_scheduler_type: cosinewarmup_ratio: 0.1bf16: Truetf32: Trueload_best_model_at_end: Trueoptim: adamw_torch_fusedbatch_sampler: no_duplicates
All Hyperparameters
Click to expand
overwrite_output_dir: Falsedo_predict: Falseeval_strategy: epochprediction_loss_only: Trueper_device_train_batch_size: 32per_device_eval_batch_size: 16per_gpu_train_batch_size: Noneper_gpu_eval_batch_size: Nonegradient_accumulation_steps: 16eval_accumulation_steps: Nonelearning_rate: 2e-05weight_decay: 0.0adam_beta1: 0.9adam_beta2: 0.999adam_epsilon: 1e-08max_grad_norm: 1.0num_train_epochs: 4max_steps: -1lr_scheduler_type: cosinelr_scheduler_kwargs: {}warmup_ratio: 0.1warmup_steps: 0log_level: passivelog_level_replica: warninglog_on_each_node: Truelogging_nan_inf_filter: Truesave_safetensors: Truesave_on_each_node: Falsesave_only_model: Falserestore_callback_states_from_checkpoint: Falseno_cuda: Falseuse_cpu: Falseuse_mps_device: Falseseed: 42data_seed: Nonejit_mode_eval: Falseuse_ipex: Falsebf16: Truefp16: Falsefp16_opt_level: O1half_precision_backend: autobf16_full_eval: Falsefp16_full_eval: Falsetf32: Truelocal_rank: 0ddp_backend: Nonetpu_num_cores: Nonetpu_metrics_debug: Falsedebug: []dataloader_drop_last: Falsedataloader_num_workers: 0dataloader_prefetch_factor: Nonepast_index: -1disable_tqdm: Trueremove_unused_columns: Truelabel_names: Noneload_best_model_at_end: Trueignore_data_skip: Falsefsdp: []fsdp_min_num_params: 0fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}fsdp_transformer_layer_cls_to_wrap: Noneaccelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}deepspeed: Nonelabel_smoothing_factor: 0.0optim: adamw_torch_fusedoptim_args: Noneadafactor: Falsegroup_by_length: Falselength_column_name: lengthddp_find_unused_parameters: Noneddp_bucket_cap_mb: Noneddp_broadcast_buffers: Falsedataloader_pin_memory: Truedataloader_persistent_workers: Falseskip_memory_metrics: Trueuse_legacy_prediction_loop: Falsepush_to_hub: Falseresume_from_checkpoint: Nonehub_model_id: Nonehub_strategy: every_savehub_private_repo: Falsehub_always_push: Falsegradient_checkpointing: Falsegradient_checkpointing_kwargs: Noneinclude_inputs_for_metrics: Falseeval_do_concat_batches: Truefp16_backend: autopush_to_hub_model_id: Nonepush_to_hub_organization: Nonemp_parameters: auto_find_batch_size: Falsefull_determinism: Falsetorchdynamo: Noneray_scope: lastddp_timeout: 1800torch_compile: Falsetorch_compile_backend: Nonetorch_compile_mode: Nonedispatch_batches: Nonesplit_batches: Noneinclude_tokens_per_second: Falseinclude_num_input_tokens_seen: Falseneftune_noise_alpha: Noneoptim_target_modules: Nonebatch_eval_metrics: Falsebatch_sampler: no_duplicatesmulti_dataset_batch_sampler: proportional
Training Logs
| Epoch |
Step |
dim_128_cosine_map@100 |
dim_256_cosine_map@100 |
dim_384_cosine_map@100 |
dim_64_cosine_map@100 |
| 0.6667 |
1 |
0.4153 |
0.4312 |
0.4460 |
0.3779 |
| 2.0 |
3 |
0.4465 |
0.4643 |
0.4824 |
0.3832 |
| 2.6667 |
4 |
0.4479 |
0.4739 |
0.4907 |
0.3891 |
- The bold row denotes the saved checkpoint.
Framework Versions
- Python: 3.10.14
- Sentence Transformers: 3.0.1
- Transformers: 4.41.2
- PyTorch: 2.3.1+cu121
- Accelerate: 0.31.0
- Datasets: 2.19.1
- Tokenizers: 0.19.1
Citation
BibTeX
Sentence Transformers
@inproceedings{reimers-2019-sentence-bert,
title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
author = "Reimers, Nils and Gurevych, Iryna",
booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
month = "11",
year = "2019",
publisher = "Association for Computational Linguistics",
url = "https://arxiv.org/abs/1908.10084",
}
MatryoshkaLoss
@misc{kusupati2024matryoshka,
title={Matryoshka Representation Learning},
author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
year={2024},
eprint={2205.13147},
archivePrefix={arXiv},
primaryClass={cs.LG}
}
MultipleNegativesRankingLoss
@misc{henderson2017efficient,
title={Efficient Natural Language Response Suggestion for Smart Reply},
author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
year={2017},
eprint={1705.00652},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
