Conlang evaluation

[Future readers: please see
http://community.livejournal.com/conlangs/339595.html for the most
current version of this essay / system - and possibly a better
formatted one.]

Conlang evaluation: a framework for criticism
By Sai Emrys - conlangs@saizai.com /  http://saizai.com
16 December 2006

BACKGROUND

David Peterson posted an entry in January 2002 later entitled the
Conlanger's Manifesto, an apologia in defense of conlanging as an art
to be taken seriously [1]. In March 2002, Jesse Bangs wrote an entry
[2] on the CONLANG mailing list entitled "Lighting Some Flames:
Towards conlang artistry", and later known as the Artlanger's Rant. It
went one step further, advocating for a new body of conlang criticism.
This part was relatively well received, except that he also advocated
for "schools" of conlanging – his being the Naturalist school – and
that triggered a rather… heated debate.

Most of the problems people had with the idea of conlanging schools
came under two basic categories:
1.	Concern that doing so would split up what is already a relatively
small subculture (approximately 1000 members worldwide, of whom
probably 100 are publicly active), and set the schools in opposition
to each other.
2.	Belief that conlang criticism could not be done because the
intention of the creator can vary so widely – and that, e.g., applying
Jesse's Naturalist criteria to languages that don't hold the same
values as his would be inapplicable at best or ideologically arrogant
at worst.

When I posted questions asking about readers' *own* priorities in prep
for this essay, the response was by contrast completely positive and
non-flamewar-like. Perhaps this is from the community having learned
from the previous experience (and avoiding those subjects), but I
believe it is in what exactly is asked, and what the assumptions
behind the questions are.


INTENT OF THIS ESSAY

My intent here is not to reignite that 2002 controversy, but to offer
an alternative – one that serves the purposes of Jesse Bangs in giving
a framework for and encouraging substantive conlang critique; is
value-neutral and takes into account the varying goals of language
creators; and is fun and easy to use yet comprehensive.

In my opinion – and from my surveys – there is no doubt that there are
numerous criteria that can be applied to examine a given language
(whether natural or created). For example, one can ask how well
documented it is; how closely its features fit in the bell-curve of
natural language 'universals'; how aesthetically pleasing it is from
certain perspectives; how regular its grammar is; how easy it is to
learn.

Some of these questions can be evaluated objectively. To rate
'universality' for example (see below), take each of its features
(e.g. split ergativity) for which there is a known occurrence
frequency in natural languages, and compute an average number using
those frequencies to say "how normal is this language, compared to all
known natural languages".

Others can only be evaluated subjectively. For example, phonaesthetics
will vary widely depending on the reader's personal preferences,
cultural background, etc. At best for this one can try to define some
types of aesthetics that are relatively consistent and agreeable (such
as "mellifluousness"), and rate the language for that rather than for
general subjective "aesthetic" in the "sounds good" or "sounds bad"
sense – because tastes can vary so widely.

I must emphasize strongly that none of these criteria have any
*inherent* value to them. Some creators may prefer to have a highly
regular language; others may prefer one that is just irregular enough
to add 'spice'; still others may actually prefer a language that is so
irregular as to be cryptographically baffling.

As such, though each criterion can be evaluated, there is no inherent
"value" rating to the criteria – i.e. no concept of "good" or "bad",
merely specific value-neutral qualitative questions. They should be
treated solely at face value for each item.


CRITERIA - META

The criteria I describe here can be used as a sort of Conlang Code, à
la Geek Code et al. To distinguish from the Conlang*er* Code v1.1, I
dub this the Conlang Code v2. They can be used in parallel and mostly
do not overlap. [3]

For each element, ratings are listed. They can be used alone for
classification and rating purposes, but for a real review it would of
course be more helpful to include a per-criterion breakdown where
relevant explaining the reasons for evaluating each item as such, or
with suggestions for how to improve it (toward the author's desired
state). They can also be used to state what your goals are for your
language(s).

Each criterion is rated on a scale from ---- to ++++, with some other
options (using !) for when the answer falls outside that spectrum.
Alternatively, criteria can be listed in numeric format for concision.
E.g. nat-(+++) = nat-1(3).

Variables work much like the Conlanger Code:
! placed after: not enough data to evaluate (review)
* placed after: item does not apply (review); indifference, neutrality (goals)
[item omitted]: item was not evaluated (review); indifference /
neutrality (goals)
() placed after: value varies (e.g. foo++(+++) = foo varies from ++ to +++)
! placed before: tertium quid or other special variation (see descriptions)

Code abbreviations are the capital letters of their heading. (E.g. nat
for NATuralness). They are listed in lowercase ("nat") in the code or
as a rating ("nat++"), but in uppercase when referring to the
criterion itself ("NAT is a criterion that….").

Some items have sub-criteria listed under 'extras'. These are optional.

Most of these criteria can be approached both objectively and
subjectively. In a detailed review, one should expect to find for each
point the rating followed by specific reasons (possibly empirical
ones, possibly not) why that rating was given.

All reviews are *only* expected to be accurate of what is publicly
documented as of the date of the review.

The review should normally be of the *primary* mode of the language.
In most cases, this will be the spoken one. (See MOD.)


Prefixes:

Part 1: CLCv2 (for "ConLang Code version 2" – this part is invariant)

Part 2: Date of writing this code (mandatory!), in DMonYY format

Part 3: Code usage type
R	Conlang Review (by author or others)
G	Conlang Goals (permissible by author only)

Part 4: Language(s) covered by this code (author may have separate
Goal codes for separate languages, or cover multiple ones in one code
if they're similar enough in goals)

Part 5: Priority (goals version only)
++++	I work on this nearly all the time.
++	This is my main current project.
--	Enh, someday I'll edit it.
----	No longer under development and not likely to be any time soon.

If priority differs, list appended to each language name; otherwise
just append to the last one.

E.g. a review might start with: CLCv2 6Dec06 R Foolang …
Or a goals code might start with: CLCv2 23May05 G Erisian-, Discordian++ …


CRITERIA PROPER

In deference to Jesse, I will start with his "naturalist" school's
primary criteria.

NATuralness: Appears to be a normal human Earth language.
nat++++	At the apex of normality: exactly 2.3 children, half a
divorce, etc. (Within 0.5 s.d. of mean.)
nat++		Medium normal. No linguist would blink an eye at finding a
description of it in Languages of the World. (Within 2 s.d.s of mean.)
nat		Within the bounds of normal, but perhaps somewhat quirky. (Within
3 s.d.s of mean.)
nat--		Not really normal at all, but maybe if you squint *really*
hard… (Within 4 s.d.s of mean.)
nat----		Just plain bizarre. 10 clicks, split ergativity, and a
built-in encryption algorithm? (What mean?)
!nat++		This language is fairly normal… for the world/culture in which
it's set ("Well, if there *were* a blind space faring avian race, this
probably *would* be a pretty normal language for their physiology…)

E.g. languages: French: nat++ Esperanto: nat; All Noun: nat---
Goals: Jesse Bangs: nat++(+++); ye generic loglanger: nat*; ye generic
sci-fi author: !nat+

ComPLeteness: Can be used to talk about anything.
cpl++++	Shakespeare, astrophysicists, and politicians can all use it.
cpl+++		Can be used to translate Hamlet.
cpl+		Can be used to translate the Babel Text
cpl		Enough for tourists to get directions…
cpl--		Well, I'm kinda still working on orthography…
cpl----		I only just named it!
!cpl++		It can be used to talk about most subjects… within the limited
scope it's intended for. (e.g. "Battle-Command-ese")

E.g. languages: English cpl++++; Toki Pona cpl+(+++); Quenya: cpl+; ye
generic toylang: cpl—
Goals: ye generic auxlanger: cpl++++; ye generic sci-fi author: cpl

ComPleXity: Number of rules, exceptions to those rules, and various
interacting contradictory levels of grammar.
cpx++++	Might require a doctorate and supercomputer to be able to use
in real time.
cpx++		Wait, I thought you said I comes before E and sounds like /i/…
cpx		At least it's no harder than my Spanish class in high school.
cpx--		So there're a couple small exceptions, but otherwise it's
pretty simple and regular…
cpx----		There's only a dozen rules? No exceptions? Everything else is free?

E.g. languages: Ithkuil cpx++++; Sanskrit cpx+(+++); Toki Pona cpx----
Goals: ye generic auxlanger: cpx--(---); ye generic artlanger:
cpx+(++); ye generic loglanger: cpx*

Note: complexity of rules does not necessarily equate to complexity of
output. Viz. John Conway's /Life/, or the Turing machine.

Personal INnovation: Breaks away from the author's preconceptions
about how language works (i.e. their L1s).
pin++++	You, a non-linguistically-trained native English-only speaker,
 came up with split ergativity by yourself?
pin++		Playing with bits not present in your L1 language family…
pin----		Another relex, eh.
!pin		Sure it's a euroclone and the author's L1s are English and
French, but zie is 100% aware of all the places zie copied zir native
grammar and explicitly decided to do so.
!!pin		The author seems to be rejecting their L1 grammar without real thought!

E.g. languages: almost everybody's first conlang (be honest now): pin---
Goals: my students' for their final project: pin++

Global INnovation: Something never done before in human language.
gin++++	[insert doctoral thesis and Nobel science award here]
gin++ 		You have a language… with no nouns!?
gin		The language has a unique "feel".
gin----		Yes, people have made European auxiliary languages before…
!gin+		That's a pretty unusual language concept… for a sketch / fiction story.

E.g. languages: ye generic IAL: gin----; Heptapod B: !gin+++
Goals: ye generic loglanger: gin+++(++++); ye generic artlanger: gin*()

CoHeRence: Overall fit, consistency of 'feel', etc.
chr++++	Every singly aspect of the language is brilliantly tailored to
the same goal.
chr++		Everything fits together well, with a very definite "feel".
chr--		Somebody's been learning new things in Linguistics class again…
chr----		Design by committee using dice rolls, a random word
generator, and every grammatical feature ever mentioned on CONLANG.

E.g. languages: Toki Pona chr+++; my conlang class languages: chr--(---)
Goals: ye generic loglanger, artlanger, and auxlanger: chr++++

CuLTural expressiveness: Shows that it is attached to a specific culture.
clt++++	A sociolinguist could create a complete analysis of the mores,
technological level, and societal structure of the culture that uses
this language just from looking at the language.
clt++		The language is clearly tied to a strong cultural
clt----		You can't even tell that such a thing as sociolinguistics
exists judging from this language.

E.g. languages: Quenya clt++
Goals: ye generic auxlanger: clt----; ye generic artlang: clt++++

LIBeralness of culture expressed: In the generic American sense.
lib++++	We love everyone! No biases! Completely gender-equal! Sex positive!
lib----		Only the righteous deserve respect.
!lib		Does not sufficiently express a culture in the first place.

E.g. languages: Laadan lib++; Newspeak lib---
Goals: ye generic auxlanger: !lib; ye generic feminist conlanger: lib+++

MeLliFluousness: Smooth, pretty, poetic in its phonaesthetic.
mlf++++	The most vile curses sound like lullabies.
mlf----		Spoken by an angry Klingon with a bad cold.

E.g. languages: Quenya mlf++(++++), Klingon mlf---(----)

SaPir-Whorf: Encourages (or forces) a certain worldview through its structure.
spw+++++	Grants new powers – genius, psychic ability, etc. – or
permanently changes the speaker's psychology.
spw++++	Makes it impossible to even think about something against its design.
spw++		Strongly encourages a specific way of looking at the world.
spw----		Exactly the same as speaking any other generic L1.
!spw+++	Does not *require* it, but does *allow* one to think about
things nearly inexpressible or inconceivable in a generic L1.

E.g. languages: Heptapod B spw+++++; Toki Pona spw+++(++++); Laadan !spw++
Goals: ye generic cognitive psycholinguist: spw+++;

EASe of learning: Ease with which this language would be learned (as an L2)
eas++++	Anybody could be fluent within days.
eas----		Would require a PhD and 20 years of exclusive study just to
ask offhand how to find the bathroom.
!eas++[eu]	Qutie easy to learn… if you already know a eurolang. (See
below for the two-letter abbreviations.)

Extras: L1 acquisition ease (format: eas[L2 ease]:[L1 ease])

E.g. languages: ye generic African click language eas---:+; Toki Pona
eas++; Ithkuil eas---
Goals: ye generic auxlanger eas+++(++++); ye generic euro-auxlanger
!eas+++(++++)[eu]; ye generic a priori loglanger eas*:++

DOCumentation: Thoroughness of the *publicly available* documentation.
doc++++	Dictionaries, linguistic analysis, primers – the works. And
all easily understandable.
doc++		Thorough description available, enough to learn the language if you look.
doc		Enough to at least get a pretty good feel for what it's all about
doc--		More of a sketch than an actual documentation. Missing some
important bits.
doc----		It's all in the author's head… or so zie claims.

Extras: list 2-letter codes in which the documentation is available.
E.g. doc++[en,sp,fr]
Extras v2: for each code or codes, separately rate thoroughness. E.g.
doc---[en]:+++[jp]

If not specified in extras, the default is English.

E.g. languages: Laadan doc++++; NLF2DWS doc--
Goals: ye generic hobbyist: doc*; ye generic cryptolinguist: doc----;
ye generic auxlanger: doc++++

CoRPus: Variety and volume of in-language material available.
crp++++	Thousands of novels, audio/video recordings, poetry,
scientific documents, etc.
crp		There's a Babel text…
crp----		You mean I'm supposed to *use* this language?

E.g. languages: English crp++++; ye generic sketch conlang crp--
Goals: ye generic loglanger crp*; ye generic auxlanger crp++++

FINishedness: Stablity & completion of the grammar and vocabulary.
fin++++	Essentially not changing at all.
fin+++		Not changing *intentionally*, but is changing through usual
natlang social mechanisms.
fin++		Quasi stable. Still in progress but no major revisions expected.
fin----		Might completely change without warning tomorrow.

E.g. languages: NLF2DWS fin--; Lojban fin++(++++)
Goals: ye generic artlanger fin*; ye generic auxlanger fin++++

FIDelity: How much the map looks like the territory.
fid++++	Everything possible is a near perfect *matopaeic match.
fid++		Most things are at least "translucent", i.e. they make sense as
representations once you know what they mean.
fid----		Everything is completely arbitrary. You would never guess
what something means without being told.

E.g. languages: ASL: fid++(+++)
Goals: ye generic non-engelanger: fid*; NLF2DWS: fid++(+++)

EFFort: How hard it is to produce / understand sentences (includes
both physical and mental effort).  Also includes length of time
required.
eff++++	Takes hours to figure out how to say it, and then leaves you
exhausted to actually try.
eff----		Very easy and fast to speak and comprehend.

Extra: Separate difficulty from time. (Format: eff[difficulty]:[time])

E.g. languages: Speedwords eff--:----; Ithkuil eff+++(++++)
Goals: ye generic auxlang: eff----; ye generic artlang or engelang: eff*

DeNSity: How much information is transmitted in a certain amount of
space (or time).
dns++++	Could write the whole plot of Hamlet on one piece of paper.
dns++		One page of English text translates to 20 seconds of speech.
dns----		One page of English text translates to *20* pages…

E.g. languages: Ithkuil dns++; formal Japanese dns--
Goals: ye generic engelang: dns++(++++); ye generic artlang: dns*

CLaRity: Ease of saying exactly what one wants, no more and no less
clr++++	Can be both completely ambiguous about whatever one wants and
retain absolute precision about the rest.
clr		About the normal amount of ambiguity or forced specification for natlangs.
clr----		Forced to over-specify or under-specify greatly.

Extra: Distinguish "allowed" from "forced" clarity. (Format:
clr[allowed]:[forced] E.g. clr+++:--- = can be very specific, but
language requires specifying e.g. everybody's gender, sexual
orientation, and hair color when referring to them.)

E.g. languages: Lojban clr++; ye generic gender-neutral artlang clr:++
(= clr0:2)
Goals: ye generic loglang: clr++++; ye generic artlang clr*

NoiSE resistance: How well utterances are understood when in noisy
situations (shouting, bombs, static, bad TV reception, speech
impediments, etc)
nse++++	You could use it in the middle of a heavy artillery barrage
and still be able to discuss the finer points of Kant as applied to
astrophysics without being misunderstood.
nse		Fairly hard to get the message right if there's a lot of noise;
might be misunderstood if there's a mild amount.
nse----		Anything less than perfect clarity means you're unlikely to
get the message even close to correct.

Extra: Semantic difference type, aka the Hungarian phrasbook index
(Format: nse[resistance]:[diff type]
nse:++++	Small difference in form à vastly different or completely
unrelated meaning (e.g. mother à Chomskian meta-pragmatics or mother à
"I cannot permit your ugly face to continue existing, prepare to
die"). Aka "tourists likely to get killed".
nse:----		 Small difference in form à similar or less specific meaning
(e.g. mother à brother or mother à relative). Aka "graceful
degradation".

E.g. languages: ye generic triconsonantal-root language (for vowels
only): nse:--(---)
Goals: ye generic engelang: nse++++:----

Form / Concept Complexity matching: How much complex ideas have
complex forms, and simple ideas simple forms.
fcc++++	"Now" = 1 phone. Postnuclearantidisestablishmentarianism = 20.
fcc		About even distribution
fcc----		Postnuclearantidisestablishmentarianism = 1 phone. "Male" =  20.
!fcc		Random relationship

FAMily: Extent to which this conlang is part of a greater family of languages
fam++++	Part of an extended family of languages, with multiple
intersecting lines, intermixing, protolanguages, etc.
fam		Part of one line of languages (e.g. "Old Conlish" vs "New
Conlish" vs "modern slang Conlish")
fam----		What family? This language stands on its own.
!fam++	This con-creole is plausible… if Malaysia ever *did* invade
Morocco... (Rating = how well it fits in its adopted
real-world-natlang family.)


MODalities: Diversity and robustness of language's modes
mod++++	Can be spoken, written, signed, or maybe even smelt; utilizes
each mode's medium uniquely; each mode conveys full spectrum of
meaning (e.g. emotion, undertones, etc)
mod		Has a unique writing system
mod----	Only exists in one mode.

Note: Most conlangs are oral with incidental written codes. Some add a
more robust writing system, e.g. ideographic. More extensively
developed languages may require a separate evaluation for each mode
(i.e. treat the oral, sign, and written languages as separate
languages in the same family).

DireCTness: How straightforward the language's idioms are
dct++++	Wanna fuck?
dct		Some of both…
dct----		I see a book I might read.  Do you have it? *
!dct		The language doesn't have idioms as such, because it isn't
strongly tied to any particular culture.

* from Sally Caves' Teonaht ("Kyam rykken ellepma wemry hain. Ain fy dhar ha?")


TiLT: Overall rating of all factors not covered above (Reviews only)
tlt++++	This is the best language I have ever seen.
tlt++		Unusually good.
tlt 		About average.
tlt----		This is the worst language I have ever seen.

Extra: Separately specify tilt for aesthetics vs technique. (Format:
tlt[aesthetics]:[technique].)

E.g. tlt++ = excellent overall. tlt++:- = very pretty, but not
particularly technically adept.
Goals: ye generic artlang: tlt+++:*; ye generic engelang: tlt*:+++


CRITERIA - WRAPUP

AMBition of design goals: Difficulty of implementing the author's design goals
amb++++	Might well require a lifetime's hard work by a brilliant mind.
amb		Would require at least a few months to accomplish.
amb----	A newbie could do that over a weekend.

Note: the amb score should take into account the number of design
goals listed, how difficult each one would be, how difficult it would
be to fulfill competing goals, etc.

E.g. languages: ??
Goals: ye brilliant artlang: amb++(+++); English relex: amb----

SUCcess: Success at fulfilling the stated goals of the language. (Reviews only)
suc++++	Scores 0 = perfect.
suc		Scores 5 = average / needs work.
suc----		Scores 9 = managed to do the exact opposite of what was intended.
!suc		Goals are not available.

To calculate the SUC score, for each item – except AMB – for which
there is a goal rating available (i.e. not * or unlisted), take the
absolute value of the goal rating minus the review rating. Average all
these numbers and round down.

E.g. goals: eff4 mlf4 cpx-2 (= eff++++ mlf++++ cpx--)
Review: eff4 mlf-1 cpx0 (= eff++++ mlf- cpx)
SUC: avg(abs(4-4), abs(4-[-1]), abs(-2-0)) = avg(0, 5, 2) = 7/3 = 2.3 = suc++


CONLANG SCHOOLS

In all of the foregoing, I have tried to be meticulous in emphasizing
that these criteria are completely value-neutral… of themselves. No
criterion is inherently more or less 'important' or 'valuable' than
any other.

However, it is nonetheless true that people both have goals for their
own languages, and beliefs or values as to what makes for a better or
worse language (or at least, one that they personally like more or
less). This is reflected in using this framework to declare one's
goals.

I believe the most relevant evaluation, then, is one where many people
rate a language on the specific criteria above, and this is compared
against the author's goals to determine how well they met those goals.
This score is admittedly subjective in many of its parts, but at least
cannot be criticized as applying an inapplicable standard, as it is
the author zirself who decides what criteria are important.

This also allows for an easy way to define conlang "schools" as fuzzy
categories. Using various statistical and neural net methods – with
people desiring similar things generally grouping together – one can
distinguish certain broad or specific categories of goal-sets. Keeping
with fuzzy categories avoids the too-rigid and perhaps polemic concept
of fixed 'schools' as such. However, one would expect to see a
grouping of e.g. artlangers and loglangers separate out if you only
look for two groups, as the two tend to value very different things.

An "ideal" or truly "successful" language is, thus, one that reaches
all its design goals perfectly.

Some design goals will be more ambitious than others of course, as
reflected in AMB. I think we can agree that a language that is highly
successful *and* ambitious is more deserving of praise than one that
is only one or the other.


WHAT NEXT?

I believe it to be undeniably obvious that conlanging is a *craft*,
like being an architect of a living system. All architects must
incorporate some degree of both aesthetics and technical mastery, but
which aspects are more relevant will depend greatly on the project.

1. Figure out what you are really aiming for – both in the context of
the ideas mentioned here, and ones not covered – and publish your
goals along with your language. This framework should help, as a
starting point, to clarify for yourself what sort of language you are
trying to create. Invite people to review your language.

2. Review both your own conlangs and others'. Use this framework as a
beginning from which to review a language – to give the review from
structure and "get to the point" with ratings. Supplement it with more
detailed descriptions for those points that are interesting, need
work, or are done particularly well.

Please note, this system intentionally does *not* cover linguistic
typology or linguistic features. There are simply far too many to
cover; doing so would require another, rather more cumbersome, system.
More importantly, though, linguistic features are not *inherently*
purposeful, and only indirectly related to the 'feel' of a language or
to whether one is likely to think more highly of it. This system is
meant to rate those more core, meta elements of how a language is
perceived.

I have tried to ensure that this system is maximally comprehensive for
95% of people, and has a minimal amount of overlap between its
criteria. It is not intended to be able to be 100% comprehensive; that
is not possible in any finite rating system. It should however cover
the goals of all types of languages; most languages will probably find
many of these ratings irrelevant, and that is fine. The point is,
different languages will find different things to be relevant.

However, I may have overlooked something; if so, please let me know,
and we can make version 2.1. ;-)

Enjoy,

- Sai


[1] http://archives.conlang.info/ce/jusae/theilvepian.html
[2] http://archives.conlang.info/fo/jinbua/vhavheintian.html
[1, 2] these entries are archived, with retrospective annotation by
Jesse Bangs, here:
http://web.archive.org/web/20041022065450/http://students.washington.edu/jaspax/conlang.htm
[3] http://taliesin.nvg.org/conlang/conlang-code-1.1.html

This can be used in conjunction with the conlang code v1.1 above. To
do so, use the format:
[conlang code v1.1 or later minus 'name of conlang' item]
CLCv2 [date] G language(s)[priority] [insert CLCv2 code  here] [linebreak]
G language(s)[priority] [insert CLCv2 code here] [etc as relevant]