A demo of Rails and the OpenAI Chat API

This post will kick off a blogpost series of typical features, that can be built with the OpenAI API and Rails. It’s way more powerful to interact with the API directly than with ChatGPT and a lot of interesting features can be built with the API.

The AI post feature

In this post I’ll start with a “simple” feature:

In a simple blogpost form with a title field and content textarea, there’s a button above the textarea that fills the textarea with a draft post based on the title. I will implement it with the OpenAI Chat API, StimulusJS and Server-Sent events.

Here’s a small video demonstration:

The basic setup

Let’s add the ruby-openai gem to our Gemfile to interact with the OpenAI API.

bundle add ruby-openai

Aftwards generate an API key here.

Next add the api key to our encrypted development credentials with rails credentials:edit --environment development. Paste your key with e.g. openai_api_key: <your-api-key> and close the editor.

Next I’m creating an initializer to configure the ruby-openai gem with the generated API key

OpenAI.configure do |config|
  config.access_token = Rails.application.credentials.openai_api_key!
end

Building a client to generate a post

Let’s start with the “Generate a text based on a title” part.

We make use of the Chat API to generate a blogpost. The chat endpoint expects us to send messages to it, and it will respond with a generated response. This is basically the API that ChatGPT uses. Whenever building things with the Chat API, I start by building a good set of prompts. I’ll start with a simple PORO that interacts with the API and can generate blogposts based on a given title.

class ContentWriter
  MODEL = 'gpt-3.5-turbo'

  def initialize
    @client = OpenAI::Client.new
  end

  def write_draft_post
    @client.chat(
      parameters: {
          model: MODEL,
          messages: [
            { role: "system", content: "You are a world class copywriter" },
            { role: "system", content: "Your output is always correctly formatted markdown" },
    )
  end
end

As a base model we’re using gpt-3.5-turbo and add two system messages to the messages array. The gpt-3.5-turbo provides cheap, fast and fairly accurate responses.

The messages array contains two system: prompts. You can think of system prompts as baseline configuration you want the model to have. For this demo, I basically only told it to be a copywriter and output markdown. I played around with this prompts and it worked fairly well for me.

We’re missing a user: prompt, which is basically the equivalent of the ChatGPT user chat message, so I’ll add a parameter to the write_draft_post method and add a third message to the messages array.

class ContentWriter
  MODEL = 'gpt-3.5-turbo'

  def initialize
    @client = OpenAI::Client.new
  end

  def write_draft_post(title)
    prompt = "Write a 1000 word blogpost about '#{title}'."
    @client.chat(
      parameters: {
          model: MODEL,
          messages: [
            { role: "system", content: "You are a world class copywriter" },
            { role: "system", content: "Your output is always correctly formatted markdown" },
            { role: "user", content: prompt },
    )
  end
end

I encourage you to try our ContentWriter in the rails console and see what the API generates.

$rails console

irb>ContentWriter.new.write_draft_post('10 reasons to eat pizza') 
irb>ContentWriter.new.write_draft_post('5 important cities in italy')

If you’ve played around with it, you’ll notice, that even though we’re using the turbo variant of the model, it still takes quite some time until you receive your generated article. The model generates your article in small chunks (also called tokens). With our current configuration, the ContentWriter will first generate the complete article and then respond with the full article in a single response.

Like you’ve seen in the demo video, instead of waiting for the full article, we want to send each token as soon as it is available and basically build a similar experience to ChatGPTs interface. We’ll continue to do so by adding the stream: parameter to the #chat call.

The OpenAI API actually uses Server-Sent-Events and sends us each token as soon as it is available. The ruby-openai gem supports the streaming mode by passing a proc to the stream parameter.

I’ll add the parameter and pass an explicit proc to it:

class ContentWriter
  MODEL = 'gpt-3.5-turbo'

  def initialize
    @client = OpenAI::Client.new
  end

  def write_draft_post(title, &block)
    prompt = "Write a 1000 word blogpost about '#{title}'."
    @client.chat(
      parameters: {
          model: MODEL,
          messages: [
            { role: "system", content: "You are a world class copywriter" },
            { role: "system", content: "Your output is always correctly formatted markdown" },
            { role: "user", content: prompt },
          ]
      },
      stream: block
    )
  end
end

Now we can pass a block to write_draft_post and it will call our block for each token the API sends us. Neat!

Our Server-Sent Events endpoint

In the next step, let’s build the UI and live streaming part to the client. Similar to the OpenAI Chat API itself, I’m going to use Server-Sent-Events to livestream tokens to the browser.

My basic plan is to create a livestream endpoint, that streams the tokens to the browser and we’re reading the SSE with the EventSource JS API in a stimulus controller. Let’s build this endpoint next.

rails generate controller ai show

To enable SSE for a controller, I include the ActionController::Live concern.

class AiController < ApplicationController
  include ActionController::Live

  def show
  end
end

There’s currently one caveat with the Rack versions required for Rails 7.0, that “break” the real streaming of responses. Rails versions that upgraded to Rack 3 (probably 7.1) will fix this issue, but for now an easy workaround is to remove the Rack ETag middleware.

This is not as bad as it sounds, since the ETags produced by this middleware have probably never matched anything anyways before, because of the various types of tokens in your markup.

# application.rb
config.middleware.delete Rack::ETag # Rack::ETag breaks ActionController::Live

Let’s combine our ContentWriter class and the controller action and stream our API response in the controller action.

def show
  response.headers['Content-Type'] = 'text/event-stream'

  ContentWriter.new.write_draft_post(params[:title]) do |chunk|
    if token = chunk.dig("choices", 0, "delta", "content")
      write_token_event(token)
    end
  end
ensure
  response.stream.close
end

private

def write_token_event(data)
  response.stream.write "event: token\n"
  response.stream.write "data: #{CGI.escape(data)}\n\n"
end

We need to explicitly set the mime type manually to text/event-stream in streamed actions.

Now we’re calling our ContentWriter#write_draft_post with a :title parameter, and for each API response chunk we write a specially formatted response string with event: and data: keys, so we can use the EventSource API in the user’s browser.

For this demo, we only care about the value in "choices, 0, delta, content", so we’re pulling the token out with dig.

As the documentation for ActionController::Live recommends, we’re adding an ensure where we close the stream response to avoid having dangling unclosed streams.

We also escape everything that we put in the data: section of our chunk, since we sometimes receive "\n" as a token from OpenAI, whenever the model wants to add a newline, which messes with our streaming format.

The streamed Chat API has one special case for the last chunk it sends: It will have a chunk.dig('choices', 0, 'finish_reason') == "stop" signalling to us, that we have received the last response chunk. Even though it’s the last chunk under normal circumstances, we’ll send a special token to our clients that signals, that the response is done with streaming. We also close the response stream, by breaking out of our write_draft_post block and therefore closing the response in our ensure block.

This is what our action looks like in the end. It’s not the cleanest code we’ve ever written, there are some error conditions we did not handle, but it get’s the job done for now:

def show
  response.headers['Content-Type'] = 'text/event-stream'

  ContentWriter.new.write_draft_post(params[:title]) do |chunk|
    if chunk.dig('choices', 0, 'finish_reason') == 'stop'
      write_token_event('[DONE]')
      break
    end

    if token = chunk.dig("choices", 0, "delta", "content")
      write_token_event(token)
    end
  end
ensure
  response.stream.close
end

We’ve got the two out of three parts done, the frontend part is still missing.

Implementing a Stimulus Controller for SSEs

Let’s add a simple scaffold for a post with:

$rails generate scaffold post title content:text

And let’s modify our _form.html.erb and add a button that calls our API. I stripped large parts of the template and only added some notable markup

<%= form_with(model: post, class: "w-full", data: { controller: 'ai' }) do |form| %>

  ...
  <%= form.text_field :title, class: 'w-full', data: { ai_target: "title" } %>
  ...
  <%= form.label :content, class: 'block font-semibold' %>
  <%= button_tag type: :button, data: { ai_target: 'button', action: "ai#generateWithAI" }, class: '... disabled:bg-gray-600' do %>
    <svg>some cool svg icon</svg>
    <span class="whitespace-pre-wrap" data-ai-target="text">Generate draft with AI</span>
  <% end %>
  <%= form.text_area :content, rows: 15, data: { ai_target: 'textarea' }, class: 'resize-y w-full disabled:bg-gray-200' %>
  ...

Important things I changed in the form markup:

I added an ai_controller.js stimulus controller to the form element
I added a stimulus target for the title input field, textarea and the ai button target
I added disabled:bg-gray-... tailwind classes to style disabled elements without js
An stimulus action generateWithAI, that get’s called when we click the ai button.

Let’s look at the stimulus controller next and how it implements the streaming feature.

import { Controller } from "@hotwired/stimulus"


export default class extends Controller {
  static targets = ["textarea", "button", "title"]

  generateWithAI() {
    this.textareaTarget.disabled = true
    this.buttonTarget.disabled = true
    const eventSource = new EventSource(`/ai?title=${this.titleTarget.value}`)

    eventSource.addEventListener('token', ({ data }) => {
      if(data === '[DONE]') {
        this.textareaTarget.disabled = false
        this.buttonTarget.disabled = false
        eventSource.close()
      } else {
        this.textareaTarget.value = this.textareaTarget.value + decodeURIComponent(data)
        this.textareaTarget.scrollTop = this.textareaTarget.scrollHeight
      }
    })
  }
}

When calling generateWithAI, we disable the textarea and the ai button. This avoids double clicking the button and also gives us the ability to add some disabled styling, indicating that our JS is working.

We then use the EventSource JS API and connect with our Rails endpoint with the the title parameter, that we pass to our ContentWriter via params[:title].

This opens a connection to our live streaming action and starts the token generation. We then add eventlisteners for the 'token' event, which we specified with type: token in our controller action.

This eventlistener calls our function for each chunk received, and we have access to our tokens in the { data } property. If we receive our special marker [DONE], we’re closing the SSE-stream in the browser, to avoid having a connection open for longer than necessary.

If we receive any other data than our marker [DONE], we unescape our sent token (e.g. for "\n") before appending it to the existing value in the textarea. To keep the textarea scrolled to the bottom and the appended tokens visible, we’re setting the scrollTop property to scrollHeight of the textarea.

That’s it, now the streamed API response will be directly streamed into the textarea 🔥.

A next step could be to use the redcarpet gem to transform the generated markdown content and render a nicely formatted blogpost.

Final thoughts

There are some things I omitted for this post, but you should probably be handled in your implementation:

Add Rack::Timeout to kill long running connections that never got closed. The heroku docs have a good resource on it (Request timeout)
Handle some common exceptions in the Rails controller, e.g. ActionController::Live::ClientDisconnected
Add a eventSource.addEventListener('error', fn) to handle errors on the client side
Write some tests 🙈

I’ll write a post about detailed error handling and testing the OpenAI APIs in the future.

This is the first post in a series of OpenAI integrations with Rails, please email me if you have any questions, found bugs or if you have ideas for more features I could showcase. The next post will be about the Transcription API and how to integrate it with Activestorage, so that uploaded Audiofiles are being transcribed automatically.

Stay tuned 🤗

The AI post feature#

The basic setup#

Building a client to generate a post#

Our Server-Sent Events endpoint#

Implementing a Stimulus Controller for SSEs#

Final thoughts#