Solvable Mysteries
One of the major surprises in writing Harry Potter and the Methods of Rationality was that I way, way, way underestimated the Transparency Illusion.
Transparency illusion is illustrated by experiments where, e.g., someone uses their fingers to tap out the rhythm of a melody—-no, not Twinkle Twinkle Little Star, people will recognize that because it’s the first rhythm that comes to everyone’s mind. People pick some other melody, then estimate the probability that the other person will recognize the melody just from hearing the rhythm tapped out. If the person estimates a probability of 50% of the melody being recognized, the actual frequency will be more like 2%. Similarly, people think that the difference between their sincerely intoned ‘Yes’ and their sarcastically intoned ‘Yes’ is much more distinguishable on a phone message than the listener is actually able to distinguish.
The Transparency Illusion played out in my writing of HPMOR as follows:
Certain things which I meant to be completely obvious, which I did not intend for one moment to be mysterious, which I foolishly assumed would be transparent to the readers and which I assumed would be part of the average reader’s interpretation of the story, became hotly debated issues on the /r/HPMOR subreddit.
Certain clues which I meant to be slightly less obvious and to come as a moderate surprise when the reader finally deduced them were much more obscurely debated issues on the /r/HPMOR subreddit. People would suggest the idea much less often, or the idea would only become commonly mentioned on /r/HPMOR after the meme had time to spread, after all the evidence had been carefully assembled to convince the average judge.
The things I meant as subtle hints that ten percent of the readers would get on their second chapter read-through, would be picked up by one out of several hundred reviewers or just completely missed until eventually one brilliant reader would suggest them on /r/HPMOR several years later only to be greeted by a chorus of skepticism.
Lest I sound like I’m blaming my readers for being insufficiently smart, I rush to remark that often /r/hpmor would offer alternative hypotheses for the story’s observations that I myself had never considered. Professor Quirrell is a time-traveling version of Harry and that’s why the two of them must never touch, as the author clearly foreshadowed by discussing how time-reversed matter looks just like antimatter and annihilates explosively in Ch. 14, just before introducing the sense of doom in Ch. 16? At that point in the story it was a legitimate attempted decoding, but one that had never occurred to me! I was blinded by my own certain knowledge of the story’s real facts; I knew what was going on inside the story, and so I didn’t experience the sense of curiosity and uncertainty that would have helped me see equally valid alternative hypotheses. (An important general life lesson!)
This leads me to present you with my New Improved Recipe for how to build mysteries, clues, and hints into your story.
First, know some really important and plot-determining background facts that your viewpoint characters won’t know and that the text is not going to state explicitly until the end of the story.
Then make absolutely no effort at all to conceal these facts. Absolutely do not leave false trails for the reader, besides any misdirection that antagonists may realistically employ to deceive protagonists. Don’t try to obscure hints and clues that you think are too explicit. Don’t worry about the reader figuring things out too early. Just let the facts cast whatever blatant huge shadows they want, so long as the story doesn’t literally, explicitly blurt out the actual truth right there in the text.
Thanks to the illusion of transparency, these background facts will be far harder to deduce than you are thinking, and they will become the mysteries that your more attentive readers ponder in the course of reading the story.
And even then you cannot assume that most readers will notice the mysteries at all before the end of the story - that they will even think of the question - unless viewpoint characters explicitly think about those questions and their clues. If the mystery is important to the plot, you had better have a character ponder it (and not solve it, presumably, due to genuinely insufficient data or an intelligent mistake) if you want the readers to know that mystery exists.
This is not because readers are stupider than you. It is because reading a story goes by much faster than writing it. If you have not called upon your readers explicitly to halt and pay attention, they are already reading the next sentence. Even if you do explicitly ask them to pay attention, they are already reading the next sentence. If you have your character think, “Hm… there’s something funny about that story, I should stop and think about that?” guess what your reader does next? That’s right, your reader goes on to read the next sentence immediately, to see what the character thinks about it.
I can now do hints somewhat subtler than yelling at the top of my lungs, and have them become visible to more than one reader, but that’s only because I have the luxury of there being a Reddit community with 6,800 readers that analyzes HPMOR. I was actually quite impressed by how much the existence of a reddit community improved the collective clue-reading power of the HPMOR readership. The dominant theories improved a lot, the subtle-clue pickup rate improved a lot, compared to the days of individual readers leaving fanfiction.net reviews on individual chapters. The advent of /r/HPMOR is the only time I’ve ever really seen the power of collective intelligence proven, because it’s the only occasion where I had all the correct answers myself but kept them secret and watched other people try to find them over a multi-year period using two different forms of communal organization, allowing me to compare their actual truth-finding efficacy.
Of which the real moral is that if there isn’t a large organized online community collaboratively analyzing your work, don’t be subtle if you want any readers to reliably pick up on something, especially during their first read-through.