triton_cpp v1.2.1
Header-only C++ wrapper for NVIDIA Triton Inference Server clients
Loading...
Searching...
No Matches
Public Member Functions | List of all members
triton_cpp::TritonInterface Class Reference

Synchronous, typed interface to one model served by Triton. More...

#include <triton_interface.hpp>

Public Member Functions

 TritonInterface (const std::string &model_name, const std::string &model_version, const std::string &server_url, bool shm, bool variable_input_size=false, bool retry_connection=false, double client_timeout_s=0.0, bool cuda_input_shm=false)
 Connect to Triton and query metadata for one model.
 
void initInOutputs (std::optional< std::map< std::string, std::vector< int64_t > > > special_output_shapes, std::optional< std::map< std::string, std::vector< int64_t > > > special_input_shapes=std::nullopt)
 Creates all input and output buffers for the model, based on the model metadata.
 
void infer ()
 Run synchronous inference with the data currently stored in the input buffers.
 
template<typename T >
VectorType< T > getInputTensor (const std::string &name, int64_t rows)
 Get a view to the InputBuffer, interpreted as a Vector.
 
template<typename T >
MatrixType< T > getInputTensor (const std::string &name, int64_t rows, int64_t cols)
 Get a view to the InputBuffer, interpreted as a Matrix.
 
template<typename T , typename... DimType>
TensorType< T, sizeof...(DimType)+3 > getInputTensor (const std::string &name, int64_t dim0, int64_t dim1, int64_t dim2, DimType... dims)
 Get a view to the InputBuffer, interpreted as a Tensor of rank >=3.
 
std::pair< uint8_t *, std::size_t > getInputTensor (const std::string &name)
 Get the raw input buffer.
 
bool usesCudaInputSharedMemory () const
 Whether input tensors are currently backed by Triton CUDA shared memory.
 
std::pair< uint8_t *, std::size_t > getInputTensorDevice (const std::string &name)
 Get the device pointer for a CUDA shared-memory-backed input tensor.
 
void copyInputTensorToDevice (const std::string &name, const void *host_data, std::size_t bytes)
 Copy host data into a CUDA shared-memory-backed input tensor.
 
template<typename T >
VectorType< const T > getOutputTensor (const std::string &name, int64_t rows) const
 Get a view to the OutputBuffer, interpreted as a Vector.
 
template<typename T >
MatrixType< const T > getOutputTensor (const std::string &name, int64_t rows, int64_t cols) const
 Get a view to the OutputBuffer, interpreted as a Matrix.
 
template<typename T , typename... DimType>
TensorType< const T, sizeof...(DimType)+3 > getOutputTensor (const std::string &name, int64_t dim0, int64_t dim1, int64_t dim2, DimType... dims) const
 Get a view to the OutputBuffer, interpreted as a Tensor of rank >=3.
 
std::pair< const uint8_t *, std::size_t > getOutputTensor (const std::string &name) const
 Get the raw output buffer and its size.
 
std::string getModelInfo () const
 Get a description of the model's in and outputs as human-readable text.
 
std::size_t nInputs () const
 Get the number of inputs.
 
std::size_t nOutputs () const
 Get the number of outputs.
 
std::vector< int64_t > getInputShape (const std::string &name) const
 Get the Input Shape of a tensor.
 
std::vector< int64_t > getOutputShape (const std::string &name) const
 Get the Output Shape of a tensor.
 
Lifetime
 TritonInterface (const TritonInterface &)=delete
 Copy construction is disabled because the interface owns registrations and buffers.
 
TritonInterface & operator= (const TritonInterface &)=delete
 Copy assignment is disabled because the interface owns registrations and buffers.
 
 TritonInterface (TritonInterface &&)=delete
 Move construction is disabled to keep registered buffer addresses stable.
 
TritonInterface & operator= (TritonInterface &&)=delete
 Move assignment is disabled to keep registered buffer addresses stable.
 
 ~TritonInterface ()
 Unregister shared-memory regions owned by this interface.
 

Detailed Description

Synchronous, typed interface to one model served by Triton.

The interface owns the Triton gRPC client, model input/output descriptors, and optional system or CUDA shared-memory registrations. Call initInOutputs() before accessing tensors or performing inference.

Definition at line 36 of file triton_interface.hpp.

Constructor & Destructor Documentation

◆ TritonInterface() [1/3]

triton_cpp::TritonInterface::TritonInterface ( const std::string & model_name,
const std::string & model_version,
const std::string & server_url,
bool shm,
bool variable_input_size = false,
bool retry_connection = false,
double client_timeout_s = 0.0,
bool cuda_input_shm = false )
inline

Connect to Triton and query metadata for one model.

Parameters
model_nameName of the served model.
model_versionModel version, such as "1".
server_urlTriton gRPC endpoint, such as "127.0.0.1:8001".
shmUse POSIX shared memory for inputs and outputs.
variable_input_sizeRecreate standard input buffers when callers request new shapes.
retry_connectionRetry client creation and model metadata queries once per second.
client_timeout_sPer-inference client timeout in seconds; zero disables the timeout.
cuda_input_shmUse CUDA IPC shared memory for inputs. Outputs still follow shm.
Exceptions
std::invalid_argumentfor incompatible options or a negative timeout.
std::runtime_errorif the server connection, model query, or requested CUDA support fails.

Definition at line 52 of file triton_interface.hpp.

60 : options_{model_name}, shm_{shm}, variable_input_size_{variable_input_size}, cuda_input_shm_requested_{cuda_input_shm} {
61 if ((shm || cuda_input_shm) && variable_input_size) {
62 throw std::invalid_argument("Variable input size and shared memory cannot be combined");
63 }
64 options_.model_version_ = model_version;
65 if (client_timeout_s < 0.0) {
66 throw std::invalid_argument("client_timeout_s must be >= 0.0");
67 }
68 options_.client_timeout_ = static_cast<decltype(options_.client_timeout_)>(client_timeout_s * 1e6);
69 triton::client::Error err;
70 err = triton::client::InferenceServerGrpcClient::Create(&triton_client_, server_url, false);
71 while (retry_connection && !err.IsOk()) {
72 std::cerr << "Failed to create Triton client: " << err.Message() << ". Retrying..." << std::endl;
73 std::this_thread::sleep_for(std::chrono::seconds(1));
74 err = triton::client::InferenceServerGrpcClient::Create(&triton_client_, server_url, false);
75 }
76 if (!err.IsOk()) {
77 throw std::runtime_error("Failed to create Triton client: " + err.Message());
78 }
79 inference::ModelConfigResponse model_config;
80 err = triton_client_->ModelConfig(&model_config, model_name, model_version);
81 while (retry_connection && !err.IsOk()) {
82 std::cerr << "Failed to get model config from Triton server: " << err.Message() << ". Retrying..." << std::endl;
83 std::this_thread::sleep_for(std::chrono::seconds(1));
84 err = triton_client_->ModelConfig(&model_config, model_name, model_version);
85 }
86 if (!err.IsOk()) {
87 throw std::runtime_error("Failed to get model config from Triton server: " + err.Message());
88 }
89
90 std::tie(input_metadata_, output_metadata_) = build_model_info(model_config);
91
92 if (cuda_input_shm_requested_) {
93#if defined(TRITON_CPP_ENABLE_CUDA_SHM)
94 inference::ServerMetadataResponse server_metadata;
95 const auto metadata_status = triton_client_->ServerMetadata(&server_metadata);
96 if (!metadata_status.IsOk()) {
97 throw std::runtime_error(
98 "CUDA input shared memory was requested, but Triton server metadata could not be "
99 "queried: " +
100 metadata_status.Message());
101 } else {
102 const bool server_supports_cuda_shm = std::find(server_metadata.extensions().begin(), server_metadata.extensions().end(),
103 "cuda_shared_memory") != server_metadata.extensions().end();
104 std::string reason;
105 if (!server_supports_cuda_shm) {
106 throw std::runtime_error(
107 "CUDA input shared memory was requested, but the Triton server does not advertise "
108 "cuda_shared_memory support");
109 } else if (!LocalCudaSharedMemorySupported(&reason)) {
110 throw std::runtime_error(
111 "CUDA input shared memory was requested, but local CUDA shared memory is not "
112 "available: " +
113 reason);
114 } else {
115 cuda_input_shm_enabled_ = true;
116 }
117 }
118#else
119 throw std::runtime_error(
120 "CUDA input shared memory was requested, but triton_cpp was built without CUDA SHM "
121 "support");
122#endif
123 }
124 }
triton_cpp::LocalCudaSharedMemorySupported
bool LocalCudaSharedMemorySupported(std::string *reason=nullptr)
Check whether this process can allocate CUDA IPC shared memory.
Definition cuda_shm.hpp:36

References triton_cpp::LocalCudaSharedMemorySupported().

◆ TritonInterface() [2/3]

triton_cpp::TritonInterface::TritonInterface ( const TritonInterface & )
delete

Copy construction is disabled because the interface owns registrations and buffers.

◆ TritonInterface() [3/3]

triton_cpp::TritonInterface::TritonInterface ( TritonInterface && )
delete

Move construction is disabled to keep registered buffer addresses stable.

◆ ~TritonInterface()

triton_cpp::TritonInterface::~TritonInterface ( )
inline

Unregister shared-memory regions owned by this interface.

Definition at line 137 of file triton_interface.hpp.

137{ releaseSharedMemoryRegistrations(); };

Member Function Documentation

◆ copyInputTensorToDevice()

void triton_cpp::TritonInterface::copyInputTensorToDevice ( const std::string & name,
const void * host_data,
std::size_t bytes )
inline

Copy host data into a CUDA shared-memory-backed input tensor.

Parameters
nameName of the input tensor.
host_dataPointer to the host buffer to copy from.
bytesNumber of bytes to copy. Must exactly match the input tensor size.
Exceptions
std::invalid_argumentif the tensor is not CUDA-backed, the byte size does not match, or triton_cpp was built without CUDA SHM support.
std::runtime_errorif the underlying CUDA copy fails.

Definition at line 376 of file triton_interface.hpp.

376 {
377#if defined(TRITON_CPP_ENABLE_CUDA_SHM)
378 auto& input = inputs_.at(name);
379 if (!input.isDeviceBacked()) {
380 throw std::invalid_argument("Input tensor '" + name + "' is not backed by CUDA shared memory");
381 }
382 if (bytes != input.data_raw_size) {
383 throw std::invalid_argument("Input tensor '" + name + "' byte size mismatch for host-to-device copy");
384 }
385 throw_on_cuda_error(cudaMemcpy(input.device_data_raw, host_data, bytes, cudaMemcpyHostToDevice), "cudaMemcpy");
386#else
387 // Suppress unused-parameter warnings when CUDA SHM support is not compiled in.
388 (void)name;
389 (void)host_data;
390 (void)bytes;
391 throw std::invalid_argument("triton_cpp was built without CUDA SHM support");
392#endif
393 }
triton_cpp::throw_on_cuda_error
void throw_on_cuda_error(cudaError_t status, const char *operation)
Throw a descriptive exception when a CUDA Runtime API call fails.
Definition cuda_shm.hpp:24

References triton_cpp::throw_on_cuda_error().

◆ getInputShape()

std::vector< int64_t > triton_cpp::TritonInterface::getInputShape ( const std::string & name) const
inline

Get the Input Shape of a tensor.

Parameters
namename of the input tensor
Returns
std::vector<int64_t> shape of the input tensor
Exceptions
std::invalid_argumentif the input name is not found in the model metadata

Definition at line 512 of file triton_interface.hpp.

512 {
513 auto it = input_metadata_.find(name);
514 if (it == input_metadata_.end()) {
515 throw std::invalid_argument("Input name not found in model metadata: " + name);
516 }
517 return it->second.shape;
518 }

◆ getInputTensor() [1/4]

std::pair< uint8_t *, std::size_t > triton_cpp::TritonInterface::getInputTensor ( const std::string & name)
inline

Get the raw input buffer.

Parameters
namename of the input
Returns
std::pair<uint8_t*, std::size_t> Pointer to the host-mappable input buffer and its size in bytes.
Exceptions
std::invalid_argumentif variable input size is enabled and this input has not been created yet, or if the input is not host-mappable.

Definition at line 333 of file triton_interface.hpp.

333 {
334 if (variable_input_size_ && inputs_.count(name) == 0) {
335 throw(std::invalid_argument("Variable input size is enabled, but no size was provided for input " + name));
336 }
337 auto& input = inputs_[name];
338 if (!input.isHostMappable()) {
339 throw std::invalid_argument("Input tensor '" + name + "' is backed by CUDA shared memory and is not host-mappable");
340 }
341 return {input.data_raw, input.data_raw_size};
342 }

◆ getInputTensor() [2/4]

template<typename T , typename... DimType>
TensorType< T, sizeof...(DimType)+3 > triton_cpp::TritonInterface::getInputTensor ( const std::string & name,
int64_t dim0,
int64_t dim1,
int64_t dim2,
DimType... dims )
inline

Get a view to the InputBuffer, interpreted as a Tensor of rank >=3.

Template Parameters
TScalar data type
Parameters
namename of the input
dim0size of the tensor.
dim1size of the tensor.
dim2size of the tensor.
dimssize of the tensor (arbitrary count).
Exceptions
std::invalid_argumentif the size of the input buffer does not match the requested size
Returns
TensorType<T> the requested Eigen::TensorMap

Definition at line 305 of file triton_interface.hpp.

306 {
307 if (variable_input_size_) {
308 CreateInputTensor(name, dim0, dim1, dim2, dims...);
309 }
310 auto& input = inputs_[name];
311 if (!input.isHostMappable()) {
312 throw std::invalid_argument("Input tensor '" + name + "' is backed by CUDA shared memory and is not host-mappable");
313 }
314 if (((dim0 * dim1 * dim2) * ... * dims) * sizeof(T) != input.data_raw_size) {
315 std::stringstream ss;
316 // Print the further dims, separated by commas
317 ss << name << ": dims: " << dim0 << ", " << dim1 << ", " << dim2;
318 ((ss << ", " << dims), ...);
319 ss << " sizeof(T): " << sizeof(T) << " input.data_raw_size: " << input.data_raw_size << std::endl;
320 throw std::invalid_argument("Invalid input tensor size: " + ss.str());
321 }
322 return TensorType<T, sizeof...(dims) + 3>{reinterpret_cast<T*>(input.data_raw), dim0, dim1, dim2, dims...};
323 }
triton_cpp::TensorType
typename std::conditional_t< std::is_const_v< T >, Eigen::TensorMap< const Eigen::Tensor< typename std::remove_const_t< T >, rank, Eigen::RowMajor, Eigen::Index > >, Eigen::TensorMap< Eigen::Tensor< T, rank, Eigen::RowMajor, Eigen::Index > > > TensorType
Helper type template for Eigen::Tensor compatible with Triton.
Definition types.hpp:55

◆ getInputTensor() [3/4]

template<typename T >
VectorType< T > triton_cpp::TritonInterface::getInputTensor ( const std::string & name,
int64_t rows )
inline

Get a view to the InputBuffer, interpreted as a Vector.

Template Parameters
TScalar data type
Parameters
namename of the input
rowssize of the vector.
Exceptions
std::invalid_argumentif the size of the input buffer does not match the requested size
Returns
VectorType<T> the requested Eigen::Map

Definition at line 247 of file triton_interface.hpp.

247 {
248 if (variable_input_size_) {
249 CreateInputTensor(name, rows);
250 }
251 auto& input = inputs_[name];
252 if (!input.isHostMappable()) {
253 throw std::invalid_argument("Input tensor '" + name + "' is backed by CUDA shared memory and is not host-mappable");
254 }
255 if (rows * sizeof(T) != input.data_raw_size) {
256 std::stringstream ss;
257 ss << name << ": rows: " << rows << " sizeof(T): " << sizeof(T) << " input.data_raw_size: " << input.data_raw_size
258 << std::endl;
259 throw std::invalid_argument("Invalid input tensor size: " + ss.str());
260 }
261 return VectorType<T>{reinterpret_cast<T*>(input.data_raw), rows};
262 }
triton_cpp::VectorType
typename std::conditional_t< std::is_const_v< T >, Eigen::Map< const Eigen::VectorX< typename std::remove_const_t< T > > >, Eigen::Map< Eigen::VectorX< T > > > VectorType
Helper type template for Eigen::Vector compatible with Triton.
Definition types.hpp:31

◆ getInputTensor() [4/4]

template<typename T >
MatrixType< T > triton_cpp::TritonInterface::getInputTensor ( const std::string & name,
int64_t rows,
int64_t cols )
inline

Get a view to the InputBuffer, interpreted as a Matrix.

Template Parameters
TScalar data type
Parameters
namename of the input
rowssize of the matrix.
colssize of the matrix.
Exceptions
std::invalid_argumentif the size of the input buffer does not match the requested size
Returns
MatrixType<T> the requested Eigen::Map

Definition at line 275 of file triton_interface.hpp.

275 {
276 if (variable_input_size_) {
277 CreateInputTensor(name, rows, cols);
278 }
279 auto& input = inputs_[name];
280 if (!input.isHostMappable()) {
281 throw std::invalid_argument("Input tensor '" + name + "' is backed by CUDA shared memory and is not host-mappable");
282 }
283 if (rows * cols * sizeof(T) != input.data_raw_size) {
284 std::stringstream ss;
285 ss << name << ": rows: " << rows << " cols: " << cols << " sizeof(T): " << sizeof(T)
286 << " input.data_raw_size: " << input.data_raw_size << std::endl;
287 throw std::invalid_argument("Invalid input tensor size: " + ss.str());
288 }
289 return MatrixType<T>{reinterpret_cast<T*>(input.data_raw), rows, cols};
290 }
triton_cpp::MatrixType
typename std::conditional_t< std::is_const_v< T >, Eigen::Map< const Eigen::Matrix< typename std::remove_const_t< T >, Eigen::Dynamic, Eigen::Dynamic, Eigen::RowMajor > >, Eigen::Map< Eigen::Matrix< T, Eigen::Dynamic, Eigen::Dynamic, Eigen::RowMajor > > > MatrixType
Helper type template for Eigen::Matrix compatible with Triton.
Definition types.hpp:41

◆ getInputTensorDevice()

std::pair< uint8_t *, std::size_t > triton_cpp::TritonInterface::getInputTensorDevice ( const std::string & name)
inline

Get the device pointer for a CUDA shared-memory-backed input tensor.

Parameters
nameName of the input tensor.
Returns
std::pair<uint8_t*, std::size_t> Device pointer and tensor size in bytes.
Exceptions
std::invalid_argumentif the input is not backed by CUDA shared memory.

Definition at line 358 of file triton_interface.hpp.

358 {
359 auto& input = inputs_.at(name);
360 if (!input.isDeviceBacked()) {
361 throw std::invalid_argument("Input tensor '" + name + "' is not backed by CUDA shared memory");
362 }
363 return {input.device_data_raw, input.data_raw_size};
364 }

◆ getModelInfo()

std::string triton_cpp::TritonInterface::getModelInfo ( ) const
inline

Get a description of the model's in and outputs as human-readable text.

Returns
std::string model info

Definition at line 489 of file triton_interface.hpp.

489{ return model_info_; }

◆ getOutputShape()

std::vector< int64_t > triton_cpp::TritonInterface::getOutputShape ( const std::string & name) const
inline

Get the Output Shape of a tensor.

Parameters
namename of the output tensor
Returns
std::vector<int64_t> shape of the output tensor
Exceptions
std::invalid_argumentif the output name is not found in the model metadata

Definition at line 527 of file triton_interface.hpp.

527 {
528 auto it = output_metadata_.find(name);
529 if (it == output_metadata_.end()) {
530 throw std::invalid_argument("Output name not found in model metadata: " + name);
531 }
532 return it->second.shape;
533 }

◆ getOutputTensor() [1/4]

std::pair< const uint8_t *, std::size_t > triton_cpp::TritonInterface::getOutputTensor ( const std::string & name) const
inline

Get the raw output buffer and its size.

Parameters
nameName of the output tensor.
Returns
Pointer to immutable output data and its size in bytes.
Exceptions
std::out_of_rangeif name is unknown when shared memory is enabled.

Definition at line 469 of file triton_interface.hpp.

469 {
470 const uint8_t* raw_data_buf{nullptr};
471 std::size_t raw_data_size;
472 if (shm_) {
473 auto shm = outputs_.at(name);
474 std::size_t offset;
475 std::string shm_name;
476 shm->SharedMemoryInfo(&shm_name, &raw_data_size, &offset);
477 raw_data_buf = output_shm_->getAddress() + offset;
478 } else {
479 results_->RawData(name, &raw_data_buf, &raw_data_size);
480 }
481 return {raw_data_buf, raw_data_size};
482 }

◆ getOutputTensor() [2/4]

template<typename T , typename... DimType>
TensorType< const T, sizeof...(DimType)+3 > triton_cpp::TritonInterface::getOutputTensor ( const std::string & name,
int64_t dim0,
int64_t dim1,
int64_t dim2,
DimType... dims ) const
inline

Get a view to the OutputBuffer, interpreted as a Tensor of rank >=3.

Template Parameters
TScalar data type
Parameters
namename of the output
dim0size of the tensor.
dim1size of the tensor.
dim2size of the tensor.
dimssize of the tensor (arbitrary count).
Exceptions
std::invalid_argumentif the size of the output buffer does not match the requested size
Returns
TensorType<const T> the requested Eigen::TensorMap

Definition at line 450 of file triton_interface.hpp.

451 {
452 auto [raw_data_buf, raw_data_size] = getOutputTensor(name);
453 if (raw_data_size != ((dim0 * dim1 * dim2) * ... * dims) * sizeof(T)) {
454 std::stringstream ss;
455 ss << name << ": dims: " << dim0 << ", " << dim1 << ", " << dim2;
456 ((ss << ", " << dims), ...);
457 ss << " sizeof(T): " << sizeof(T) << " raw_data_size: " << raw_data_size << std::endl;
458 throw std::invalid_argument("Invalid output tensor size: " + ss.str());
459 }
460 return TensorType<const T, sizeof...(dims) + 3>{reinterpret_cast<const T*>(raw_data_buf), dim0, dim1, dim2, dims...};
461 }
triton_cpp::TritonInterface::getOutputTensor
VectorType< const T > getOutputTensor(const std::string &name, int64_t rows) const
Get a view to the OutputBuffer, interpreted as a Vector.
Definition triton_interface.hpp:405

References getOutputTensor().

◆ getOutputTensor() [3/4]

template<typename T >
VectorType< const T > triton_cpp::TritonInterface::getOutputTensor ( const std::string & name,
int64_t rows ) const
inline

Get a view to the OutputBuffer, interpreted as a Vector.

Template Parameters
TScalar data type
Parameters
namename of the output
rowssize of the vector.
Exceptions
std::invalid_argumentif the size of the input buffer does not match the requested size
Returns
VectorType<const T> the requested Eigen::Map

Definition at line 405 of file triton_interface.hpp.

405 {
406 auto [raw_data_buf, raw_data_size] = getOutputTensor(name);
407 if (raw_data_size != rows * sizeof(T)) {
408 std::stringstream ss;
409 ss << name << ": rows: " << rows << " sizeof(T): " << sizeof(T) << " raw_data_size: " << raw_data_size << std::endl;
410 throw std::invalid_argument("Invalid output tensor size: " + ss.str());
411 }
412 return VectorType<const T>{reinterpret_cast<const T*>(raw_data_buf), rows};
413 }

References getOutputTensor().

Referenced by getOutputTensor(), getOutputTensor(), and getOutputTensor().

◆ getOutputTensor() [4/4]

template<typename T >
MatrixType< const T > triton_cpp::TritonInterface::getOutputTensor ( const std::string & name,
int64_t rows,
int64_t cols ) const
inline

Get a view to the OutputBuffer, interpreted as a Matrix.

Template Parameters
TScalar data type
Parameters
namename of the output
rowssize of the matrix.
colssize of the matrix.
Exceptions
std::invalid_argumentif the size of the output buffer does not match the requested size
Returns
MatrixType<const T> the requested Eigen::Map

Definition at line 426 of file triton_interface.hpp.

426 {
427 auto [raw_data_buf, raw_data_size] = getOutputTensor(name);
428 if (raw_data_size != rows * cols * sizeof(T)) {
429 std::stringstream ss;
430 ss << name << ": rows: " << rows << " cols: " << cols << " sizeof(T): " << sizeof(T) << " raw_data_size: " << raw_data_size
431 << std::endl;
432 throw std::invalid_argument("Invalid output tensor size: " + ss.str());
433 }
434 return MatrixType<const T>{reinterpret_cast<const T*>(raw_data_buf), rows, cols};
435 }

References getOutputTensor().

◆ infer()

void triton_cpp::TritonInterface::infer ( )
inline

Run synchronous inference with the data currently stored in the input buffers.

Returned output views remain valid until the next call to infer().

Exceptions
std::runtime_errorif Triton rejects or fails the inference request.

Definition at line 222 of file triton_interface.hpp.

222 {
223 if (variable_input_size_) {
224 raw_inputs_.clear();
225 for (const auto& input : inputs_) {
226 raw_inputs_.push_back(input.second.input.get());
227 }
228 }
229 triton::client::InferResult* raw_results{nullptr};
230 auto status = triton_client_->Infer(&raw_results, options_, raw_inputs_, raw_outputs_);
231 if (!status.IsOk()) {
232 throw(std::runtime_error("ModelInfer failed: " + status.Message()));
233 }
234 results_.reset(raw_results);
235 };

◆ initInOutputs()

void triton_cpp::TritonInterface::initInOutputs ( std::optional< std::map< std::string, std::vector< int64_t > > > special_output_shapes,
std::optional< std::map< std::string, std::vector< int64_t > > > special_input_shapes = std::nullopt )
inline

Creates all input and output buffers for the model, based on the model metadata.

Parameters
special_output_shapesSome models don't know their output shape, i.e. it is given as -1. In this case, you must provide the correct shape here.
special_input_shapesSome models don't know their input shape, i.e. it is given as -1. In this case, you must provide the correct shape here.
Exceptions
std::invalid_argumentif a provided input or output name is unknown.
std::runtime_errorif shared-memory setup was requested but initialization fails.

Definition at line 148 of file triton_interface.hpp.

149 {
150 releaseSharedMemoryRegistrations();
151 input_shm_.reset();
152#if defined(TRITON_CPP_ENABLE_CUDA_SHM)
153 input_cuda_shm_.reset();
154#endif
155 output_shm_.reset();
156 inputs_.clear();
157 outputs_.clear();
158 if (special_input_shapes.has_value()) {
159 for (const auto& [name, shape] : special_input_shapes.value()) {
160 auto it = input_metadata_.find(name);
161 if (it == input_metadata_.end()) {
162 throw std::invalid_argument("Input name not found in model metadata: " + name);
163 }
164 const auto datatype = it->second.datatype;
165 input_metadata_.erase(it);
166 input_metadata_.emplace(name, InputOutputMetaData{shape, datatype});
167 }
168 }
169
170 if (special_output_shapes.has_value()) {
171 for (const auto& [name, shape] : special_output_shapes.value()) {
172 auto it = output_metadata_.find(name);
173 if (it == output_metadata_.end()) {
174 throw std::invalid_argument("Output name not found in model metadata: " + name);
175 }
176 const auto datatype = it->second.datatype;
177 output_metadata_.erase(it);
178 output_metadata_.emplace(name, InputOutputMetaData{shape, datatype});
179 }
180 }
181 if (!variable_input_size_) {
182 if (cuda_input_shm_enabled_) {
183 try {
184 setup_cuda_shm_inputs(input_metadata_);
185 } catch (const std::exception& e) {
186 throw std::runtime_error("CUDA input shared memory was requested, but initialization failed: " + std::string(e.what()));
187 }
188 } else if (shm_) {
189 setup_shm_inputs(input_metadata_);
190 } else {
191 setup_standard_inputs(input_metadata_);
192 }
193 }
194
195 if (shm_) {
196 setup_shm_outputs(output_metadata_);
197 } else {
198 setup_standard_outputs(output_metadata_);
199 }
200
201 // Also store the raw pointers to in/outputs in vectors, as required by the Triton client
202 // We don't have to delete these anywhere, as they are managed by the respective shared pointers
203 if (!variable_input_size_) {
204 raw_inputs_.clear();
205 for (const auto& input : inputs_) {
206 raw_inputs_.push_back(input.second.input.get());
207 }
208 }
209
210 raw_outputs_.clear();
211 for (const auto& output : outputs_) {
212 raw_outputs_.push_back(output.second.get());
213 }
214 }

◆ nInputs()

std::size_t triton_cpp::TritonInterface::nInputs ( ) const
inline

Get the number of inputs.

Returns
std::size_t

Definition at line 496 of file triton_interface.hpp.

496{ return input_metadata_.size(); }

◆ nOutputs()

std::size_t triton_cpp::TritonInterface::nOutputs ( ) const
inline

Get the number of outputs.

Returns
std::size_t

Definition at line 503 of file triton_interface.hpp.

503{ return output_metadata_.size(); }

◆ operator=() [1/2]

TritonInterface & triton_cpp::TritonInterface::operator= ( const TritonInterface & )
delete

Copy assignment is disabled because the interface owns registrations and buffers.

◆ operator=() [2/2]

TritonInterface & triton_cpp::TritonInterface::operator= ( TritonInterface && )
delete

Move assignment is disabled to keep registered buffer addresses stable.

◆ usesCudaInputSharedMemory()

bool triton_cpp::TritonInterface::usesCudaInputSharedMemory ( ) const
inline

Whether input tensors are currently backed by Triton CUDA shared memory.

Returns
true if CUDA shared memory is enabled for inputs, otherwise false.

Definition at line 349 of file triton_interface.hpp.

349{ return cuda_input_shm_enabled_; }
The documentation for this class was generated from the following file: