This month’s Frame: remixing meaning—multimodality as a lens for understanding online expression

A framework for re-conceptualising communication and expression in digital spaces—seeing people as “designers” of modal ensembles.

TikTok continues to dominate amongst younger users, and much has been written on the reasons for its popularity—from its laser sharp algorithm through to its value as a search engine for Gen Z. What exactly has TikTok tapped into that has made it so successful, especially for young people? 

While there’s no doubt that TikTok’s algorithm has been a large contributor to its success, it’s not just about the social app’s ability to tailor the experience of consumption. For younger users in particular, it is also about the tools it provides for multi-faceted expression. Eugene Wei describes it as “combinatorial evolution”—the way that an app like TikTok enables people to remix communication and meaning. 

Remixed content abounds on digital platforms, but especially on TikTok—for young people, it’s a language in and of itself. Memes, for example, involve adding a new layer of meaning to pre-existing content, and can only be understood if the audience grasps both the original reference point(s) and the commentary that’s been appended.

This edition of Frames uses Gunther Kress' work on multimodality to show how TikTok has tapped into rich forms of adaptive expression that go beyond “message sent -> message received” and has harnessed the power of people’s ability to design modal ensembles.

The framework

Kress establishes, first and foremost, that communication is anything but straightforward. Particularly in digital spaces, people must deal with complex, often contradictory demands: their own interests (aims and purposes); the meaning or message they want to convey about a particular topic, issue, or themselves; the characteristics of the audience; platform norms and algorithms; and their knowledge of how information is transmitted online. 

Kress’ work posits that to deal with these contradictory demands, people express themselves through modes—multiple modes—which include but extend beyond “language”. A mode is a socially and culturally shaped resource for making meaning. Examples of modes in digital spaces include text but also image, sounds, layout, aesthetics, music, colour, and even the use of app settings/configurations in a particular sequence. 

In communication, several of these modes are almost always used together—that is, in modal ensembles. In digital spaces, people can also re-use or re-mix existing messages (“content”) with other modes that they deem appropriate for their message, and thus perpetuate the creation of the next modal ensemble.

How do users choose their modes? Kress highlights the fact that each mode (and its ensembles) has certain affordances—both potentials and constraints. For example, a potential of music is that it allows someone to communicate with a lot of emotion, but a constraint is that the audience has to take the time to listen to it, whereas an image only takes a second to absorb. Music also leaves more room for interpretation and ambiguity, which could be viewed as either a potential or constraint by the user. 

Or consider app settings—this is a mode valued by younger people as it allows them to communicate discreetly and indirectly. For example, on Snapchat, users can publish content to an audience of one, without them knowing that they are the only viewer. 

Users are constantly looking at a variety of modes, assessing their potentials and constraints, combining them into ensembles, and then remixing them with other people’s modal ensembles.

Modal ensembles let people aptly convey their messages to particular audiences in particular digital environments for particular aims and purposes. Even a small tweak in the ensemble (which often occurs in the case of remixed content) allows someone to completely change the meaning of their message for their intended audience.

Consequently, we should think of people as “designers” of complex messages using modal ensembles. Modal ensembles are powerful for users because they can meet multiple demands. They are like the “Rosetta Stone” of online language—a key that unlocks new interpretations of meaning and compounding forms of expression.

Using the framework

For platforms who want to ensure that younger users’ communication needs are met while engaging with their product, it’s important to see people as designers of messages. Digital platforms need to make available multiple modes for expression and communication that provide users with a variety of affordances. Platforms should support users in designing modal ensembles, and in particular, make it easy to remix ensembles through small tweaks to modes. 

Returning to TikTok, we can see how it is an app that accomplishes this effectively, and how it also helps users design modal ensembles efficiently. As Eugene Wei puts it, “they explicitly lower the barrier to the literal remixing of everyone else's content. In their app, they have a wealth of features that make it dead simple to grab any element from another TikTok and incorporate it into a new TikTok.” It’s a simple process, but make no mistake, as a modal ensemble, remixed content is a sophisticated form of expression that’s vital to younger users’ communication.

TikTok provides people with tools for meaning making and lets its users decide how they want to use them. And, as Wei points out, even the seemingly weird ones are powerful in the hands of human agency: “[The] myriad of ways TikTok users put [the Warp Scan filter] to use just shows what happens when you throw random tools to the masses and allow for emergent creativity.” Random or not, it’s these tools that are allowing young people to engage with the world and represent their worlds on their own terms, in their own distinctive ways.

News

• We are looking for a bright, organised and enthusiastic Junior Research Consultant to help coordinate our multi-country research projects.
  If you are curious about culture, social science, technology and business, then follow this link to apply.

• We recently shared the “the new research playbook” in which we outline 3 shifts executives need to facilitate to set research up for success in the future.

  The piece was put together following fruitful conversations with fellow practitioners about the underlying challenges research is facing. We’d love to hear what resonates and what doesn’t, join the conversation on Linkedin and tell us what you think.

Frames is a monthly newsletter that sheds light on the most important issues concerning business and technology.

Stripe Partners helps technology-led businesses invent better futures. To discuss how we can help you, get in touch.