@/all we just released v0.28.0! Here is the full changelog, and here's a summary:
New features
Breaking changes
Bug fixes
downscale_stabilization_period
to be disregarded during downscalingMisc
operator_load_balancer_scheme: internal
in your cluster configuration file, and set up VPC Peering. We plan in supporting a new auth strategy in an upcoming release.http://api-<api_name>:8888/predict
is not the correct endpoint for my local replicas?
@WebPerfTest_twitter if you don't spin up a cluster with GPU instances, then it won't use GPUs and you won't be taxed by AWS for that. You will only be taxed for the resources you use. Check out the instance_type
field on the Cortex Cloud on AWS page.
You could even subscribe to EKS Optimized AMI with GPU Support and if you're not specifying GPU instances, then you won't be billed for that.
All you have to do is specify the CPU-only instance type in your cluster config and you should be good to go (i.e. c5.xlarge
).
ClientError: An error occurred (InvalidParameterValue) when calling the DeleteMessage operation: Value <XXXXXXXXX> for parameter ReceiptHandle is invalid. Reason: The receipt handle has...
File "batch.py", line 330, in <module>
start()
File "batch.py", line 326, in start
sqs_loop()
File "batch.py", line 190, in sqs_loop
handle_batch_message(message)
File "batch.py", line 230, in handle_batch_message
sqs_client.delete_message(QueueUrl=queue_url, ReceiptHandle=receipt_handle)
File "botocore/client.py", line 337, in _api_call
return self._make_api_call(operation_name, kwargs)
File "botocore/client.py", line 656, in _make_api_call
raise error_class(parsed_response, operation_name)
Hey folks - another question from me again :)
Our cortex endpoints would like to use custom caching or config fetches from other machines. Think of something like Redis being deployed on an EC2 instance that has the VPC peering setup. I have followed this guide, but I can't seem to reach out to my EC2 instance from Cortex (Ec2 -> cortex works fine). i was wondering if there was an additional step that allowed two way comms.
cortex v0.23
@/all we just released v0.29.0! Here is the full changelog, and here's a summary:
New features
Breaking changes
requirements.txt
(docs) and/or dependencies.sh
(docs).Bug fixes
Docs
dependencies.sh
(docs) or custom images (docs)Misc
vpc peering
to find the relevant docs for that version. If there are no vpc peering docs for your version, let us know.
Hi all - I'm using Cortex 0.29 to do batch prediction using a PythonPredictor and Tensorflow. It's working fine on the CPU with basic requirements.txt, but now I'm trying to use the GPU and getting errors in the worker setup logs:
*"log": "2021-03-02 19:18:05.940507: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64\n",
For this explanation, what you’ve referred to as Cortex Core, the ability to install Cortex on your own cluster, I am going to refer to as Cortex BYOCluster
to make it more verbose. For the main version of Cortex that provisions its own cluster, I am going to refer to as Cortex Managed
.
BYOCluster is more flexible because it can be installed on your own EKS/GKE. This flexibility comes at the cost of not having some of the cluster aware functionality such as automatic GPU/ASIC setup, spot instances and upcoming functionality such as supporting multiple instance types. We’ve removed Cortex BYOCluster because it makes it harder for us to support and build some of the more complicated cluster-aware features because the cluster is no longer in Cortex’s control.
Rather than providing a product that is more flexible but supports a subset of the current and upcoming features, we leaned towards improving Cortex Managed to more easily integrate into existing develops workflows. We could maintain two separate products, but the idea of focusing on a single product, a managed cluster optimized for model inference that makes at scale model deployment and management in production easy and integrates into your devops stack appealed to us more.
Given that you’ve used Cortex Managed, it would be great to hear your thoughts on Cortex BYOCluster and Cortex Managed. Feel free to reach out to me at vishal@cortexlabs.com if you would like to have a chat.