philschmid HF staff commited on
Commit
5bca4e6
1 Parent(s): d3ce09e

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +54 -1
README.md CHANGED
@@ -1,3 +1,56 @@
1
  ---
2
- license: mit
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
+ tags:
3
+ - endpoints-template
4
+ license: apache-2.0
5
  ---
6
+
7
+ # Multi-Model GPU Inference with Hugging Face Inference Endpoints
8
+
9
+ Multi-model Inference Endpoints provide a way to deploy multiple models onto the same infrastructure for a scalable and cost-effective inference. On multi-model Inference Endpoints, we load a list of models into memory, either CPU or GPU, and dynamically use them during inference time.
10
+
11
+ The following diagram shows how multi-model inference endpoints look.
12
+
13
+ ![asset](mmie.png)
14
+
15
+ This repository includes a [custom handler](handler.py) of a sample multi-model `EndpointHandler` implementation. This multi-model handler loads 5 different models for inference including:
16
+ - `DistilBERT` model for `sentiment-analysis`
17
+ - `Marian` model `translation`
18
+ - `BART` model for `summarization`
19
+ - `BERT` model for `token-classification`
20
+ - `BERT` model for `text-classification`
21
+
22
+ If you want to learn more about multi-model inference endpoints checkout https://www.philschmid.de/multi-model-inference-endpoints
23
+
24
+
25
+ # Use with Inference Endpoints
26
+
27
+ Hugging Face Inference endpoints can be used with an HTTP client in any language. We will use Python and the `requests` library to send our requests. (make your you have it installed `pip install requests`)
28
+
29
+ ![result](inference.png)
30
+
31
+ ## Send requests with Pyton
32
+
33
+ ```python
34
+ import json
35
+ import requests as r
36
+
37
+ ENDPOINT_URL = "" # url of your endpoint
38
+ HF_TOKEN = "" # token of the account you deployed
39
+
40
+ # define model and payload
41
+ model_id = "facebook/bart-large-cnn"
42
+ text = "The tower is 324 metres (1,063 ft) tall, about the same height as an 81-storey building, and the tallest structure in Paris. Its base is square, measuring 125 metres (410 ft) on each side. During its construction, the Eiffel Tower surpassed the Washington Monument to become the tallest man-made structure in the world, a title it held for 41 years until the Chrysler Building in New York City was finished in 1930. It was the first structure to reach a height of 300 metres. Due to the addition of a broadcasting aerial at the top of the tower in 1957, it is now taller than the Chrysler Building by 5.2 metres (17 ft). Excluding transmitters, the Eiffel Tower is the second tallest free-standing structure in France after the Millau Viaduct."
43
+ request_body = {"inputs": text, "model_id": model_id}
44
+
45
+ # HTTP headers for authorization
46
+ headers= {
47
+ "Authorization": f"Bearer {HF_TOKEN}",
48
+ "Content-Type": "application/json"
49
+ }
50
+
51
+ # send request
52
+ response = r.post(ENDPOINT_URL, headers=headers, json=request_body)
53
+ prediction = response.json()
54
+
55
+ # [{'summary_text': 'The tower is 324 metres (1,063 ft) tall, about the same height as an 81-storey building. Its base is square, measuring 125 metres (410 ft) on each side. During its construction, the Eiffel Tower surpassed the Washington Monument to become the tallest man-made structure in the world.'}]
56
+ ```