Beware Of Roles In AI Prompting

When writing a prompt, clarity is king. There are many techniques we can employ to shape the output of the LLM for our purposes, and one option we have in our prompt engineering toolbelt is the use of roles. Roles encourage the AI model to approach your question or problem by assuming a specific character and viewpoint. One would assume that roles would always enhance the answer and make it better, and that’s why I was surprised while reading a recent study that showed that role prompting can sometimes decrease the accuracy and usefulness of model output.

Confused by these findings, I decided to test the effect of roles in my own prompts with a simple experiment. I submitted two kinds of prompts to two different LLMs with and without roles in the prompt to look for differences in response.

For the first test, I used a math problem from the Schaum’s Outline of College Physics as the prompt. That problem was:

When a building is constructed at -10 degrees Celsius, a steel beam (with a cross-section area of 45 square cm) is put in place with its ends cemented in pillars. If the sealed ends cannot move, what will be the compressional force on the beam when the temperature is 25 degrees Celsius? For steel, Y=200,000 MPa.   (The correct answer is 378, 000 Newtons).

This is a straightforward mathematical problem with only one correct answer, and I expected all of the answers to be the same. Here were the results when submitted to two different models:

The models were correct for the most part, but using the specific role of Electrical Engineer for this prompt with Google’s Gemini made the model believe it was not the right resource to answer the question.

For the second test, I used a prompt that would generate answers that were more creative in nature, again first without a role, then with several roles. That base prompt was:

Create a subject line for a cold email aimed at acquiring new customers for a consultancy that uses AI for business automation.

This is a subjective question, and I expected a different answer with every new prompt and role. These were the results:

The use of roles changed the tone and salesly-ness of the response every time a new one was used in both ChatGPT and Gemini. Side note: with Gemini, the first pass at this experiment generated nine subject lines that were listed under the categories of Short and Sweet, Intriguing, and Benefit-Focused. With the base prompt and using all of the roles, the answers were always the same. When I further refined my prompt to instruct it to provide a single subject line, the above answers were provided.

The takeaway from this experiment is that roles are a valid prompt engineering technique, but assigning the wrong or not perfect role will confuse the model and negatively affect your results. If using roles, try your prompt without a role and then with several roles to see if that improves results.

If you are interested in exploring AI for business, please Contact Us to request a meeting. Don’t have an information architecture for AI that you trust yet? LRS can also help you collect, organize, and analyze your data so that it is business ready.