Some thoughts on mutli-modal (signing / speech) languages and communication.

One thing I've been curious about is : Why aren't more natural
languages signed? In particular what are the advantages and
disadvantages of signing for communication? This is a question I'm
working out to put into a conlang / conculture, and which I think
might interest the list. A lot of this came out of a conversation I
had with Alex Fink last night, and some personal ruminations on the
subject I've had over the years. Alex prodded me into putting it to
the List with his curiosity, and I'd like to say that this essay is
100% Research Free. It's only a bit of imaginative speculation.

All the languages I know of are (simultaneously) signed to some degree
or another, and a little reflection shows that there is a syntax of
sign underlying the attested gesticulations we see in conversation.
For example, if while asking a stranger on the street "What time is
it?" I make a breast stroke as though swimming, this will be
incongruous, but if I say this while pointing at my wrist with my
dominant index finger, it will seem natural. Also - there are cultural
and language dependent signs that are generally associated with voiced
utterances, for example telling someone to go away while gesticulating
a V-shape isn't (commonly) used in Ontario, but it was an attested way
to tell someone to bugger off, or show defiance in England.  I tend to
find people who do not gesticulate at all while speaking somewhat
unnerving. So - There's something going on here.

So, why aren't more natural languages uni-modally signed, or at least,
why is so little of the "formal grammar" of spoken natural languages
encoded in sign (transmitted mutli-modally)? A related question is,
why did humans opt for speaking over signing? Every time I ask these
questions there are a couple responses that come up, what I hear most
often I'm going to call the Distance Problem, the Attention Problem,
and Sofa problem. I don't think these are the most serious problems,
but I'll start off with them.

The Distance Problem
"Human languages aren't signed because it is more pragmatic to be able
to communicate over long distances". Well - Two things are at play
here, a cultural thesis, and a physical fact. I think former is pretty
dubious. As things are, I can say that only a miniscule fraction of my
communication happens over distances where I can't see the recipient's
fine articulation (barring lighting, and sight line obstructions). I'd
guess that most communication happens in close proximity, so the
cultural idea that people tend to talk over large distances is at
least, suspect. Now, "it's better to have a tool and not need it, than
to need a tool and not have it." This is probably the way things are,
I rarely need to shout, but when I do, I really need to. But - Why
couldn't there be a course grained signed mode akin to semaphore, that
communicates a minimum amount of information? I tend not to shout
elaborate utterances to people, and generally I say a word or two
"HELLO!", "HELP!", "YOU THERE?", etc. if it's not urgent (most of the
time), I just wave!  The physical fact is that verbal communication
over long distances is hugely lossy; that's why I usually say so
little. A lot of phonemes just don't project well, in fact, I when in
a large auditorium, I can often see the speaker much better than I can
hear them!  The flip side of the Distance problem is the Racket
problem, which I will get to.

The sight line problem is pretty severe though, a lot of domestic
communication happens when you can't see the other person, e.g. when
they are in another room it is generally useful to be able to talk to
people who are near to you, but whom you can't see ie. when they are
following you, shingling a roof with you, etc. Special cases like
lighting aren't a problem, since electric lights are so ubiquitous,
but a lot still remains. At first glance, I think it is something that
would work itself out culturally, that is, even more emphasis could be
placed on face to face communication than is already, and maintaining
lines of sight could bear great cultural weight. More on this later.

The Attention Problem
"Human languages aren't signed because it is so much easier to get
someone's attention vocally." In American Deaf culture, there are a
variety of ways to get someone's attention, including waving, if there
is a sight line open, tapping on the shoulder, if there isn't and
you're sufficiently close. If someone else is around you can wave to
them, and they'll try to get attention directed to you. If all else
fails it's even possible to stamp a little. These all work if the
person whose attention you're seeking is profoundly Deaf. If they're
hearing, of course all of the above still apply, but you could also
say their name, or make a noise (being louder than anything else going
on might be a problem). It doesn't seem like there is anything in the
way of getting someone's attention by exclusively multi-modal means
especially if both parties are hearing.

Alex points out that getting someone's attention is much easier than
keeping it, and the real problem lies in maintaining signed
communication, while doing other things. I agree with this, and it is
pretty limiting to have to divert all of one's visual attention to one
person while they talk to you. Also receiving simultaneous messages,
and spontaneous messages might be problematic. Receiving two streams
of information several times over could be hard, and if someone
doesn't have your visual attention, but they catch your auditory
attention, then you might not catch the whole message before they get
it out. The Deaf seem to manage the former well enough though, and
perhaps social stigmas would grow around hogging peoples' attention.

The Sofa Problem
"Human languages aren't signed because humans do a lot of stuff with
their hands while communicating (ie. moving sofas, washing dishes),
and hence their hands are generally occupied." I'm not sure how much
of a problem this is. I think that a negligible fraction of
communication occurs while hands are absolutely occupied, and in
situations where they're not absolutely occupied a signed human
language should be robust enough to function. Compare talking with
your mouthful, it's lossy, and you can't articulate everything, but
you can get your point across. I'm going to take this point up in a
bit, under another guise though.

Now I'd like to consider some less obvious problems both for and
against multi-modally transmitted human languages used by hearing
humans.

The Racket Problem
This is an instance of "needing a tool and not having it" though it is
a kind of exceptional need. It is generally very hard to verbally
communicate in a noisy environment, and in my experience I am in a lot
of noisy environments e.g. busses, student spaces, and concerts. There
have been a couple times I really wish my fellow concert goers spoke
ASL, so I could tell them how much I like the act, or that I won't
take a bottle of water if it costs more than $5. Similarly with
busses, often I can barely hear someone speaking two feet away from me
on the other side of the bus.

The Racket problem does have an interesting consequence though, there
is a sort of security through noise, which is why eavesdropping can be
such a challenge! As I said above, sight lines are often clearer than
one's hearing, and hence eavesdropping might be more practicable in a
signed context.  A sketchy cultural thesis, is that over time the
Distance problem and the Racket problem have switched roles. Where our
agrarian ancestors would have wanted to communicate more over long
distances, and almost never had to communicate under heavy noise
burdens, we almost never need to communicate over great distances and
often need to under heavy noise. So in this sense, signing might make
more sense now.

The Contorted Problem
Again, a somewhat special case, but a lot of communication happens
when we are physically contorted into chairs, or couches, though
conveniently, these two leave our arms mostly unimpinged. Problems
occur when we are lying down, or not upright, but it's clear that
chatting before bed is pretty negligible. In the worst wing back
chairs, this problem is only really as bad as the Sofa Problem, so
it's not too worrying. In a hypothetical signing community, presumably
these problems would be eliminated by design, and difficult yoga
positions would be explained well in advance.


The Sound Problem
One possible reason for adopting a signed language is a dislike of
sound. This could be the product of a cultural trend, or environmental
factor. If quietness is considered desirable, then perhaps at over a
period of time people would just give up talking, and a whole society
would take up signing instead of voicing as some monasteries have
done. Another socio-genesis of signed language could come from a deaf
elite, or a large portion of hereditary deafness such as occurred in
Martha's Vineyard. Pragmatically, a signed language could occur in a
hunting intensive society, where scaring off prey could be mean very
harsh consequences for the tribe as a whole.

The Apparatus Problem
One major blow against signed language in general is that, at the
bottom of it, it is much more mechanically difficult to produce. As we
get older,  the joints in our hands and fingers wear out much faster
than our mouths. Our hands, though incredibly tough, are also very
easily damaged compared to our mouths. At a guess, I think that
people's vision also tends to wear out faster than their hearing but
I'm not sure. The idea being that if people sign more, then their
ability to communicate is going to be tied to their ability to use
their hands, and if their hands wear out, they're out of luck. If a
language is sufficiently mutli-modal, then your ability to communicate
relies on being able to do two very separate mechanically intricate
things well, and by Occam's razor, this is sub optimal. Now, by the
same token, your ability to communicate is spread over two media, and
if one medium starts to go, perhaps you'll still have the redundancy
of the other. There is a kind of guarantee of robustness, something in
slight opposition to Occam's razor anyway but very well attested in
natural languages.

Another problem related to the mechanism of transmission, is that we
tend to obscure our hands with things like gloves or mitts that make
fine finger movement hard to distinguish. This is somewhat like the
Sofa and the Contorted problem, but I think it is more contingent on
the way we tend to use our hands. We're never really forced to wear
mitts, or anything that seriously obscures our fingers save casts and
such, probably if more people signed, they'd wear gloves instead of
mitts (and figure out other workarounds). Although, as anyone who has
known a winter, or been in a meat freezer knows, the ability to
gesticulate is substantially decreased by coldness. Similarly by
drunkenness (fine gesticulation!), or tiredness. Of course those last
two are also contingent on the culture of the speakers.

The Technology Problem
A lot of our existent communication technology relies on how it is
relatively easy to transmit sound. That is why we had gramophones
before film, the phone before the video phone, radio before TV. This
relative facility provides a technological bottleneck on transmitting
mutli-modally communicated languages, and I'm not sure how to solve
it. At a guess, technology would be developed differently in a society
that uses these kinds of languages, or certain "spoken only" forms
which presumably maps signed components to sounds would develop for
radio transmission and such. As things are right now, with the advent
of broadband internet, it is incredibly easy to get rich video and
audio easily, so the problem isn't as serious as it could be but any
culture that uses a mutli-modal language will have to solve it.

If you've read this far, you're either interested, or just skipped to
the end. In either case, I hope these reflections have sparked some
ideas in you. If anything in this arm chair speculation has been
really erroneous, or you think you can add something, I'd really like
to hear it. I've tried to be somewhat thorough in this reflection on
the possibilities of signing, and in particular of creating a
multi-modally communicated signed/spoken language. The language
towards which I strive would be usable by a population of "average
modern humans" living in a contemporary developed civilization. That
is, humans who hear well, see well, and live in an environment not
unlike my own. I think signing is an incredibly rich medium, and is
utterly fascinating. There are some things I think it is much better
suited to express, and it really is an area ripe for conlang
exploration. Hopefully I'll have some more material for the List soon.

Replies