--- title: "Models and performance" output: rmarkdown::html_vignette vignette: > %\VignetteIndexEntry{Models and performance} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- ``` r library(layaR) ``` ## Where models live layaR keeps models in a per-user folder that follows CRAN's rules (`tools::R_user_dir()`), one subfolder per checkpoint: ``` r sub(path.expand("~"), "~", laya_model_root(), fixed = TRUE) #> [1] "~/.cache/R/layaR/onnx" ``` Set the `layaR.model_root` option to keep them elsewhere, for example on a larger disk: `options(layaR.model_root = "/data/laya-models")` in your `.Rprofile`. There are three ways to get a model: * `laya_download_model("multilingual")` downloads an export from Hugging Face (`gueyenono/laya-onnx` by default; the `layaR.model_repo` option or the `repo` argument point elsewhere). The download resumes if it is interrupted, and nothing counts as installed until every file is there. * `laya_import_model(path, "multilingual")` adds an export you made yourself with the Laya repository's `export_onnx.py`. By default it links to the folder instead of copying 1.5 GB; `method = "copy"` copies it. * A model exported for Laya's Rust or TypeScript SDK, in their shared cache (`~/.cache/laya/onnx`), is found automatically. layaR reads it but never writes there. `options(layaR.shared_cache = FALSE)` turns this off. `laya_models()` lists the three checkpoints, whether each is installed, and where it was found; `laya_remove_model()` deletes one (an imported link is removed without touching the original files). ## Engines and memory An engine is a loaded checkpoint. Loading takes a few seconds and holds the model in memory, so create one engine and reuse it. The prediction functions do that for you: they default to `laya_default_engine()`, which loads the checkpoint named by the `layaR.checkpoint` option (`"multilingual"` unless you set it) on first use and keeps it for the session. ``` r before <- rss_mb() load_time <- system.time(engine <- laya_engine("multilingual"))[["elapsed"]] c(load_seconds = round(load_time, 1), memory_mb = round(rss_mb() - before)) #> load_seconds memory_mb #> 2 868 engine #> multilingual · fused ONNX · max_len 1024 · head_max_len 256 #> /home/gueyenono/.cache/R/layaR/onnx/multilingual ``` Each engine holds its own copy of the model, so two engines take twice the memory. `laya_engine_close()` frees it right away instead of waiting for garbage collection: ``` r laya_engine_close(engine) ``` To use a different checkpoint by default, set the option before the first prediction, for example `options(layaR.checkpoint = "english")`, or pass `engine = laya_engine("english")` to the calls that need it. ## Threads ONNX Runtime runs one forward pass on several cores. `intra_op_threads` sets how many; by default ONNX Runtime picks, which usually means one per physical core. Set the `layaR.threads` option to change the default for every engine, for example to leave cores free for other work. ## Parallel work **Don't fork.** ONNX Runtime's thread pools don't survive `fork()`, so an engine can't be used inside `parallel::mclapply()` or `future::multicore`. layaR detects this and raises a `laya_error_forked` error rather than letting the child process hang. **One engine is usually enough.** A single engine already spreads each forward pass over all cores. Running several engines in parallel on the same machine mostly makes them compete for the same cores, while multiplying memory use. **Separate processes work.** In PSOCK workers (`parallel::makePSOCKcluster()`, `future::multisession`, or `callr`), each process loads its own engine. An engine sent to a worker, or saved with `saveRDS()` and read back, reconnects to its model folder on first use, which loads the model again in that process. This is useful across machines, or with `intra_op_threads` set so the workers share the cores between them. ## Speed The rest of this vignette measures where the time goes, on the machine below, with the multilingual checkpoint. ``` r c(cpu = sub(".*: *", "", grep("model name", readLines("/proc/cpuinfo"), value = TRUE)[1]), cores = parallel::detectCores()) #> cpu cores #> "Intel(R) Core(TM) i7-4790K CPU @ 4.00GHz" "8" ``` ``` r qs <- laya_questions( department = laya_choice("Which department should handle this request?", billing = "invoices, payments, refunds", technical = "bugs, outages, system errors", sales = "pricing, new contracts", other = "everything else"), urgency = laya_score("How urgent is this request?", c("not urgent", "soon", "critical")), churn_risk = laya_noul("Does the customer say they will cancel or leave?") ) set.seed(1) openings <- c("Hi team,", "Hello,", "Urgent:", "Quick question:", "Dear support,") issues <- c("we were billed twice for March", "the app crashes when I log in", "we need a quote for 50 seats", "please close my account", "the export to CSV fails", "our production server is down") endings <- c("Thanks.", "Please fix this today or we will cancel.", "Any update?", paste(rep("We have tried everything in your documentation.", 3), collapse = " ")) texts <- paste(sample(openings, 256, TRUE), sample(issues, 256, TRUE), sample(endings, 256, TRUE)) summary(nchar(texts)) #> Min. 1st Qu. Median Mean 3rd Qu. Max. #> 38.0 47.0 71.5 87.7 87.0 190.0 engine <- laya_engine("multilingual") invisible(laya_predict(texts[1], qs, engine = engine)) # warm up ``` ### The R layer adds almost nothing Every call crosses from R to Rust as JSON. Comparing `laya_predict()` with a call straight into the Rust bridge, with the JSON prepared in advance, shows what the R side costs: ``` r state <- texts[1] ptr <- layaR:::engine_ptr(engine) state_json <- as.character(layaR:::to_json(state)) qs_json <- layaR:::questions_json(qs) opts <- layaR:::predict_options(NULL, NULL, NULL, NULL) overhead <- bench::mark( laya_predict = laya_predict(state, qs, engine = engine), rust_only = layaR:::rs_predict(ptr, state_json, qs_json, opts), check = FALSE, min_iterations = 30 ) overhead[, c("expression", "min", "median")] #> # A tibble: 2 × 3 #> expression min median #> #> 1 laya_predict 170ms 172ms #> 2 rust_only 169ms 171ms ``` The two are within measurement noise of each other: building the JSON, checking arguments and turning the result into a list cost almost nothing next to the forward pass. ### Batches: padding costs more than it saves on a CPU A forward pass can take several texts at once, but every text in it is padded to the longest one. On a GPU batching usually wins; on this CPU it doesn't. `laya_predict_batch()` therefore sends one text per forward pass by default (`batch_size = 1`, or the `layaR.batch_size` option): ``` r one_by_one <- function(x) for (s in x) laya_predict(s, qs, engine = engine) batch <- function(...) laya_predict_batch(texts[1:64], qs, engine = engine, progress = FALSE, ...) batches <- bench::mark( `laya_predict() in a loop` = one_by_one(texts[1:64]), `batch_size = 1 (default)` = batch(), `batch_size = 4, sorted` = batch(batch_size = 4, sort_by_length = TRUE), `batch_size = 16, sorted` = batch(batch_size = 16, sort_by_length = TRUE), `whole chunk (NULL)` = batch(batch_size = NULL), check = FALSE, min_iterations = 3, filter_gc = FALSE ) data.frame(method = as.character(batches$expression), seconds = round(as.numeric(batches$median), 2), texts_per_second = round(64 / as.numeric(batches$median), 1)) #> method seconds texts_per_second #> 1 laya_predict() in a loop 15.08 4.2 #> 2 batch_size = 1 (default) 13.96 4.6 #> 3 batch_size = 4, sorted 13.99 4.6 #> 4 batch_size = 16, sorted 18.32 3.5 #> 5 whole chunk (NULL) 23.14 2.8 ``` The answers are the same whatever the batch size; only the speed changes. If you run on different hardware, try a few values of `batch_size` with `sort_by_length = TRUE` on your own texts, and set the `layaR.batch_size` option to the best one. Even with the default batch size, `laya_predict_batch()` is the better tool for many texts than a loop: it checks the questions once, shows a progress bar, and stops cleanly when interrupted. `chunk_size` only decides how often R gets control back for that; it barely changes throughput: ``` r chunks <- bench::mark( `chunk_size = 16` = laya_predict_batch(texts, qs, engine = engine, chunk_size = 16, progress = FALSE), `chunk_size = 64` = laya_predict_batch(texts, qs, engine = engine, chunk_size = 64, progress = FALSE), `chunk_size = 256` = laya_predict_batch(texts, qs, engine = engine, chunk_size = 256, progress = FALSE), check = FALSE, min_iterations = 2, filter_gc = FALSE ) data.frame(method = as.character(chunks$expression), texts_per_second = round(length(texts) / as.numeric(chunks$median), 1)) #> method texts_per_second #> 1 chunk_size = 16 4.0 #> 2 chunk_size = 64 4.3 #> 3 chunk_size = 256 4.3 laya_engine_close(engine) ``` ### Threads ``` r threads <- c(1, 2, 4, parallel::detectCores()) by_threads <- do.call(rbind, lapply(threads, function(n) { e <- laya_engine("multilingual", intra_op_threads = n) on.exit(laya_engine_close(e)) invisible(laya_predict(texts[1], qs, engine = e)) t <- bench::mark(laya_predict_batch(texts[1:32], qs, engine = e, progress = FALSE), min_iterations = 2, filter_gc = FALSE) data.frame(intra_op_threads = n, texts_per_second = round(32 / as.numeric(t$median), 1)) })) by_threads #> intra_op_threads texts_per_second #> 1 1 1.9 #> 2 2 3.5 #> 3 4 4.3 #> 4 8 5.3 ``` ### Asking several questions at once All the questions about a text are answered in one forward pass, but their instructions and options are part of the input, so each question still adds tokens and time. Asking them together is cheaper than asking them one by one: ``` r engine <- laya_engine("multilingual") invisible(laya_predict(texts[1], qs, engine = engine)) separately <- function() for (id in names(qs)) laya_predict(texts[1], qs[id], engine = engine) n_questions <- bench::mark( `1 question` = laya_predict(texts[1], qs["churn_risk"], engine = engine), `3 questions, one call each` = separately(), `3 questions, one call` = laya_predict(texts[1], qs, engine = engine), `triage preset (5 questions)` = laya_predict(list(message = texts[1]), laya_preset("triage"), engine = engine), check = FALSE, min_iterations = 20 ) n_questions[, c("expression", "median")] #> # A tibble: 4 × 2 #> expression median #> #> 1 1 question 81.9ms #> 2 3 questions, one call each 223.5ms #> 3 3 questions, one call 174.2ms #> 4 triage preset (5 questions) 513.4ms laya_engine_close(engine) ``` Long option descriptions cost more than short ones: the triage preset's five questions carry more text than the three above. ## Summary * Load an engine once (a few seconds, under a gigabyte of memory for the multilingual checkpoint) and reuse it; the default engine does this for you. * The R layer adds no measurable time; the forward pass is the cost. * On a CPU, one text per forward pass is fastest, and that is the default. Use `laya_predict_batch()` or `laya_annotate()` for many texts anyway, for the progress bar and clean interrupts. * Ask all your questions in one call, and keep option descriptions short. * Use processes, not forks, for parallel work, and only when one engine isn't enough.