State-of-the-art NLP fashions from R



The Transformers repository from “Hugging Face” comprises a whole lot of prepared to make use of, state-of-the-art fashions, that are easy to obtain and fine-tune with Tensorflow & Keras.

For this function the customers often have to get:

  • The mannequin itself (e.g. Bert, Albert, RoBerta, GPT-2 and and so on.)
  • The tokenizer object
  • The weights of the mannequin

On this publish, we’ll work on a traditional binary classification activity and practice our dataset on 3 fashions:

Nonetheless, readers ought to know that one can work with transformers on quite a lot of down-stream duties, akin to:

  1. characteristic extraction
  2. sentiment evaluation
  3. textual content classification
  4. query answering
  5. summarization
  6. translation and many extra.


Our first job is to put in the transformers bundle by way of reticulate.

reticulate::py_install('transformers', pip = TRUE)

Then, as typical, load normal ‘Keras’, ‘TensorFlow’ >= 2.0 and a few traditional libraries from R.

Observe that if operating TensorFlow on GPU one might specify the next parameters with a purpose to keep away from reminiscence points.

physical_devices = tf$config$list_physical_devices('GPU')



We already talked about that to coach an information on the precise mannequin, customers ought to obtain the mannequin, its tokenizer object and weights. For instance, to get a RoBERTa mannequin one has to do the next:

# get Tokenizer
transformer$RobertaTokenizer$from_pretrained('roberta-base', do_lower_case=TRUE)

# get Mannequin with weights

Information preparation

A dataset for binary classification is offered in text2vec bundle. Let’s load the dataset and take a pattern for quick mannequin coaching.

Break up our information into 2 components:

idx_train =*0.8)

practice = df[idx_train,]
take a look at = df[!idx_train,]

Information enter for Keras

Till now, we’ve simply lined information import and train-test break up. To feed enter to the community we’ve got to show our uncooked textual content into indices by way of the imported tokenizer. After which adapt the mannequin to do binary classification by including a dense layer with a single unit on the finish.

Nonetheless, we need to practice our information for 3 fashions GPT-2, RoBERTa, and Electra. We have to write a loop for that.

Observe: one mannequin typically requires 500-700 MB

# checklist of three fashions
ai_m = checklist(
  c('TFGPT2Model',       'GPT2Tokenizer',       'gpt2'),
   c('TFRobertaModel',    'RobertaTokenizer',    'roberta-base'),
   c('TFElectraModel',    'ElectraTokenizer',    'google/electra-small-generator')

# parameters
max_len = 50L
epochs = 2
batch_size = 10

# create a listing for mannequin outcomes
gather_history = checklist()

for (i in 1:size(ai_m)) {
  # tokenizer
  tokenizer = glue::glue("transformer${ai_m[[i]][2]}$from_pretrained('{ai_m[[i]][3]}',
                         do_lower_case=TRUE)") %>% 
    rlang::parse_expr() %>% eval()
  # mannequin
  model_ = glue::glue("transformer${ai_m[[i]][1]}$from_pretrained('{ai_m[[i]][3]}')") %>% 
    rlang::parse_expr() %>% eval()
  # inputs
  textual content = checklist()
  # outputs
  label = checklist()
  data_prep = perform(information) {
    for (i in 1:nrow(information)) {
      txt = tokenizer$encode(information[['comment_text']][i],max_length = max_len, 
                             truncation=T) %>% 
        t() %>% 
        as.matrix() %>% checklist()
      lbl = information[['target']][i] %>% t()
      textual content = textual content %>% append(txt)
      label = label %>% append(lbl)
    checklist(,textual content),,label))
  train_ = data_prep(practice)
  test_ = data_prep(take a look at)
  # slice dataset
  tf_train = tensor_slices_dataset(checklist(train_[[1]],train_[[2]])) %>% 
    dataset_batch(batch_size = batch_size, drop_remainder = TRUE) %>% 
    dataset_shuffle(128) %>% dataset_repeat(epochs) %>% 
  tf_test = tensor_slices_dataset(checklist(test_[[1]],test_[[2]])) %>% 
    dataset_batch(batch_size = batch_size)
  # create an enter layer
  enter = layer_input(form=c(max_len), dtype='int32')
  hidden_mean = tf$reduce_mean(model_(enter)[[1]], axis=1L) %>% 
    layer_dense(64,activation = 'relu')
  # create an output layer for binary classification
  output = hidden_mean %>% layer_dense(items=1, activation='sigmoid')
  mannequin = keras_model(inputs=enter, outputs = output)
  # compile with AUC rating
  mannequin %>% compile(optimizer= tf$keras$optimizers$Adam(learning_rate=3e-5, epsilon=1e-08, clipnorm=1.0),
                    loss = tf$losses$BinaryCrossentropy(from_logits=F),
                    metrics = tf$metrics$AUC())
  # practice the mannequin
  historical past = mannequin %>% keras::match(tf_train, epochs=epochs, #steps_per_epoch=len/batch_size,
  gather_history[[i]]<- historical past
  names(gather_history)[i] = ai_m[[i]][1]

Reproduce in a           Pocket book

Extract outcomes to see the benchmarks:

Each the RoBERTa and Electra fashions present some extra enhancements after 2 epochs of coaching, which can’t be stated of GPT-2. On this case, it’s clear that it may be sufficient to coach a state-of-the-art mannequin even for a single epoch.


On this publish, we confirmed find out how to use state-of-the-art NLP fashions from R. To know find out how to apply them to extra complicated duties, it’s extremely beneficial to assessment the transformers tutorial.

We encourage readers to check out these fashions and share their outcomes under within the feedback part!


When you see errors or need to recommend adjustments, please create a problem on the supply repository.


Textual content and figures are licensed beneath Artistic Commons Attribution CC BY 4.0. Supply code is on the market at, except in any other case famous. The figures which were reused from different sources do not fall beneath this license and might be acknowledged by a be aware of their caption: “Determine from …”.


For attribution, please cite this work as

Abdullayev (2020, July 30). RStudio AI Weblog: State-of-the-art NLP fashions from R. Retrieved from

BibTeX quotation

  writer = {Abdullayev, Turgut},
  title = {RStudio AI Weblog: State-of-the-art NLP fashions from R},
  url = {},
  yr = {2020}


Leave a Reply

Your email address will not be published. Required fields are marked *