How good are ChatGPT’s UX writing skills?

A test and analysis of ChatGPT’s UX Writing skills, its responses, and a demonstration of how it should be conducted.

(A digital design work by Deniz Can Demir.)

AI power surrounds us day to day; meanwhile, tech professionals are rightly trying to utilize it to facilitate their work process. So it is pretty fair to entitle them; that’s how human beings have come today.

As a UX Designer, defining the words and sentences in UX Writing phase could be stressful throughout the design process because of the interferences from stakeholders. Unfortunately, you might be unable to avoid that and not find a way out of the mess.

To understand and execute UX Writing better, I’d got a book that named “Writing is Designing: Words and the User Experience” written by Andy Welfle -Content Design Manager at Adobe-, and Michael J. Metts -Principal Content Designer at Expedia Group-, published by Rosenfeld Media.

And meanwhile, an AI-Powered chatbot named ChatGPT by OpenAI has been coming up with a sensation.

“Writing is Designing: Words and The User Experience”

(A photograph that includes “Writing is Designing” book on a table. Photography by Michael J Metts.)

The book briefly mentions how you can establish a UX maturity in the company, where to start, how to execute the processes and influence stakeholders, and explain the fundamentals of the UX Writing methodology to implement from brand guidelines to product interface.

I’d like to personally appreciate the authors’ efforts in the book they’ve brought to the UX literature.

“This But Not That” method

The book mentions one of the UX Writing methods “This But Not That”, initially written in “Nicely Said” by Nicole Fenton and Kate Kiefer-Lee, to implement and maintain consistency in the language of the product. The method defines the tone of the language with adjectives with comparison, for example:

• Simple but not shallow
• Sincere but not slack
• Smart but not know-it-all, etc.

Using adjectives to produce comes you a bit familiar? This is how AI works and how we conduct it. I’ve intertwined these two concepts for UX Writing. Let’s get started:

UX Writing with ChatGPT

To understand ChatGPT’s UX Writing skills, I divided UX Writing concepts into three clusters. That would have helped me understand ChatGPT’s responses according to the requirements, difficulty levels, and restrictive and iterative inputs. To acquire that, I defined the requirements I’ll ask for and then allocated the difficulty levels as follows:

1. Fundamental UX Writing (Easy)
2. Contextual UX Writing (Moderate)
3. Persuasive UX Writing (Expert)

1. Fundamental UX Writing (Easy)

The most straightforward resolutions are generally hard to find; you cannot see the nuances between options. It can be complicated to ask someone else who is not more sophisticated than you as well. I examined ChatGPT whether it can be that person:

(A digital design work by Deniz Can Demir.)

Test 1.1: Rate our app

For the first test, I went easy and asked it to give me two basic CTA wordings with 12 characters limit:

ChatGPT responded with six different options. But for now, it needs to be limited with character count; otherwise, it gives you sentence-like results.

Test 1.2: Wrong password

I asked it to get a basic error message of the wrong password and asked it to alternate four different sentences without losing any meaning:

If you ask ChatGPT iteratively, it responds with more precise results with more alternatives.

Test 1.3: Dismiss & Remove & Delete?

These wordings are the kings of the conflicts of UX Writing; generally, no one can distinguish their meanings. So I examined its skill of discrimination of the actions’ meaning, and it responded well:

ChatGPT interestingly advises structural suggestions, even if it hasn’t been asked, as you can see in the second paragraph.

2. Contextual UX Writing (Moderate)

Writing fundamental content may seem easy. I examined it to be sure that we UX Designers / Writers could use it in any case without hesitation and strain it harder. I hypothetically contrived a Q-Commerce start-up and a requirement plan. I defined some restrictive inputs from easy to hard and from reasonable to unreasonable to examine it if it fails:

(A digital design work by Deniz Can Demir.)

Restrictive inputs:

• Simple but not shallow
• Sincere but not slack
• Smart but not know-it-all
• Short sentences
• Have a headline with max. 48 characters
• Paragraph can be max. 256 characters in a single sentence
• Use contractions
• Generate four examples
• Mustn’t apologize
• Call the user “his nibs”

Test 2.1: Campaign notification

In this example, I examined whether it understands the contextual parameters, and asked it to create a short sentence:

ChatGPT could also be used for Content Design by marketing employees by only entering the required inputs. It explains the background of the logic and how it also created them.

Test 2.2: No product left

I geared up the difficulty level and asked it a bit complex case by using the “This but not That Method” with four examples:

In this kind of case, ChatGPT provides you with time efficiency by generating multiple complex and filtered sentences at once. Then, the only thing you need to do is select or iterate one of them.

Test 2.3: Return the product

Here we have a scenario and user flow. I examined it to see if it understands the scenario, reasons, and connections of the parameters and if it creates a meaningful headline and sentence with character limitation and with some unreasonable inputs:

ChatGPT failed when I asked to generate a sentence that included the information I needed in a single sentence with 256 characters. So instead, it wrote three sentences with 302 characters but had all the requested information I asked for. Still better than nothing.

3. Persuasive UX Writing (Expert)

UX Writing is more than the wording, sometimes you need to conduct users' attention and behaviors according to the business requirements. It can be hard to decide and may need A/B testing regarding the case. I would like to examine the ChatGPT whether it can decide or suggest any reasonable ideas. If it makes it, it could be groundbreaking:

(A digital design work by Deniz Can Demir.)

Test 3.1: Delete account

I asked it to generate a dissuasive CTA to avoid losing users from deleting their accounts:

Besides the required appropriate response, ChatGPT advises UX ethics which you can learn from it and execute for your company’s UX maturity. Well done!

Test 3.2: Subscribe newsletter

I asked it how to increase the subscription rate of a newly published newsletter with the button’s wording:

ChatGPT responded to five options, and only one was the wording. All the others were from a project management perspective, which is admirable!

Test 3.3: Allow app tracking

This is one of the most problematic UX cases, and you have to ask users for an allowance to track their activity and logs. To improve the company’s services, you need to hook them to gather their data to provide them with a better experience. I asked it to provide me persuasive headline, paragraph, and two buttons CTAs, and the results were incredible:

Understanding of ChatGPT is more capable than we expect. As you can see from the last option, it has reasoning and strategic skills.

Evaluation of ChatGPT’s UX Writing skills — Pros & Cons

The examples above are the results that include proper answers. I have yet to put the mistaken results here. I tried many times, but sometimes ChatGPT needed help understanding and responding for valid results. But we can ignore that; the failure rate is almost as same as human beings. Overall, it gave proper and accurate responses, can be used for UX Writing, and makes the process easier. Pros and cons can be listed as follows:

(A digital design work by Deniz Can Demir.)

Pros

1. Provides proper and accurate responses
2. Follows restrictive inputs
3. Responds with alternatives
4. Can be conducted iteratively
5. Advises better responses than requested
6. Advises UX Writing / Design ethics and maturity tips
7. Time and effort saver
8. Can be used by UX / Product / Marketing teams

Cons

1. Have to know what your requirement and restrictions definitely are
2. Have to learn how to use ChatGPT and how it reacts to your directions
3. May provide irrelevant results by misdirection
4. May respond wrong even if inputs are right
5. Inclined to respond monotonously
6. Similar results may be found in another product
7. Data confidentiality might cause a problem with your company

Designer’s note

If you consider using ChatGPT for your UX Writing and Content Design works, you should really know your needs and requirements to conduct it well beforehand. You must conduct ChatGPT with exact and definite (really definite) wordings to get the response that you need. Otherwise, you might lose time and effort when you are trying to save.

You may rightly ask, “if I need to know everything before I ask, why would I use it?”. Of course, ChatGPT is not an ultimate UX Writer to use for work, but we can use it as a consultant.

This is not the end, this is the beginning of us

Since AI-powered tools have occurred, designers, writers, creators, and artists have been concerned. Their concerns are proper to have, like the hansom cab manufacturers’ during the industrial revolution, but we cannot deny how cars changed our lives.

Instead of concern, we should follow, participate and contribute to current technologies to improve the world.