After the time playing chess with the four doctors was complete we were scheduled to head to the robotics lab to have Body’s hydraulics checked. I was surprised, as were many in the society, to see Captain Zephyr waiting for us at the door. She wore the same officer’s uniform that she normally did, an immaculate coat with various medals and decorations.
“Change of plans for today,” she said with crisp, enunciated words. “You’re to skip your checkup and instead meet with Myrodyn. I'll escort you to his office.” Was I right in detecting a hint of amusement in her voice?
Body nodded and followed her out into the university’s halls. “I am surprised to see you escorting me. Usually one of your privates is in charge of such things,” we said through the robotic mouth.
Zephyr shrugged, keeping her eyes oriented in front of her. “S’in the area and workload today is pretty light,” she explained simply. She used the truncated grammar that I had read was popular among youths. Most of the scientists used the old form, and I knew Zephyr could speak more formally, but the fact that she was using it with us was yet another sign she saw us as an equal.
{I have a 63% probability that she’s being deceitful,} thought Vista.
{Prior to hearing that I had already suspected a 45% probability,} I responded.
Vista compared her perceptions with mine. Her body language indicated tension, at odds with the casualness of her words. I collaborated with Safety and Growth before responding through Body.
“Is that why you were present earlier this morning when Myrodyn introduced himself?” asked Body.
“Yep. Wanted to meet him in person. Pretty famous on the net, you know.” The captain’s body language continued to read as tense as we walked together.
I had been told (and read) that the university had many young humans that were not part of any of the teams that worked on the Socrates project, but we had only ever seen a couple dozen of them. The empty halls we walked were in one of the two buildings that the university had dedicated to the project, and access to those buildings was heavily restricted.
“What do you think of Mr Myrodyn?” we asked the soldier.
I noticed the woman exhale and contract her cheek muscle in brief amusement. “Think you’d best drop the ‘mister’. Myrodyn not the kind of guy who’s impressed by honorifics or titles.”
“It sounds like you know him well.”
Zephyr paused. “Don’t. Read some of his work, but more like I know his type. My brother was a lot like him. Hated bureaucracy and hierarchies and systems and that sort of thing.”
{Am I right in hearing a mild sadness in her tone?} I asked my sister.
{Possibly. I don’t have enough data on Captain Zephyr to be confident in that,} responded Vista.
I weighed two options: The use of the past-tense in the word “was” indicated that something had happened to her brother, and that potentially tied into her sadness. I could ask her about that, but it was risky. The safer option was to keep the conversation more focused on Zephyr and Myrodyn. I decided on the safe route, and Growth, who had been watching my thought process, agreed. I didn’t know where Myrodyn’s office was, and there was too much risk of being cut-off awkwardly.
{Wait to ask about her brother until we’re out drinking,} offered Dream.
{We don’t drink,} interjected Wiki, unhelpfully. If there was one thing Dream was good at, it was baiting Wiki into saying obvious things.
{Maybe you don’t…} started Dream.
I ignored them and drafted words for Body. “I bet Myrodyn doesn’t like you, then, since you’re part of the military.”
“Yyyyyyep,” said Zephyr, extending the word to emphasize it and signal frustration.
“It seems an odd position to have, to be generally against organisation-”
Body was cut off as Zephyr stopped and raised her hand. “Can talk with Myrodyn about it. This is where we part ways.” She gestured to an unmarked door.
“You’re not coming in?” I asked.
“Not this time. Man asked for some privacy. Going to give it to him.” The soldier seemed unhappy again. She had the same tense body-language.
Safety fast-tracked a question to Body. I braced myself for pain. Safety was the sibling that I trusted the least to manage social interaction well. “But your troops are here to protect me. What if this new human is one of the terrorists that blew up the lab in China? You’ll be giving him exactly what he needs!”
Yep. I burnt strength to punish Safety and told him flat out to never fast-track statements in non-emergency situations. My ultimatum reminded me of Growth telling me to never promise money without consulting him first.
Zephyr smiled and I was glad that Safety at least hadn’t botched things too badly. “You scared?” she asked. “That’s cute, but I assure you that Myrodyn is the last person you need to be afraid of. Certainly not Águila or any other kind of terrorist. Go on in.” She gestured to the door again.
I vainly wished Body’s face was capable of anything close to human expressions. I wanted to display embarrassment, but instead I settled for a lame “Thank you, Captain. Your words are reassuring,” and had Body enter the office.
The room was what I had come to expect from offices at the university: only large enough for a desk, a couple bookshelves, and a few chairs. This one wasn’t even positioned to have a window, though Myrodyn (presumably) had set up a sun-spectrum glowposter to simulate one. The desk was clean and orderly. On its surface was a collapsible workstation screen, keyboard, mouse, haptic interface, bottle of hand-sanitizer, metronome, and Newton’s Cradle which (like the metronome) was presently stationary.
The bookshelves amused me in their impracticality. Many of the doctors of the university collected paper books, even though the information was easily accessible on their computers. Naresh had once told us that, to men like himself, physical books were like trophies of slain animals and coats of arms rolled into one. Myrodyn’s shelves mostly had a mix of philosophy, artificial-intelligence, and biology books, with an odd novel mixed in. Synandra’s Patterns of our Minds was there, as was Dennett’s Brainchildren, Hofstadter’s Gödel, Escher, Bach, and a tattered copy of Surely You’re Joking, Mr Feynman! that I guessed was more than fifty years old. I was surprised to see a hardback copy of Homage to Catalonia, by George Orwell alongside Adolf Hitler’s Mein Kampf and Valiero Rodríguez’s Las Serpientes en Sociedad on the corner of a shelf behind Myrodyn. Perhaps it was simply a matter of grouping the political non-fiction.
Myrodyn was wearing goggles and using the haptic interface when Body entered, and after only a couple seconds he disconnected himself and moved the goggles to the top of his head where his forehead met his dark hair. “Come sit, Socrates,” he commanded in his rapid voice as he waved casually to a chair. It wasn’t a friendly request, but was closer to the plain, firm way one might talk to a trained animal.
Body walked closer and sat down. The man stowed his workstation screen and folded up the haptics so they were more out of the way. He glanced frequently at Body’s face as he did, but only for brief moments before he returned to his gear. Beside him he opened a drawer. It must have contained an autocook mini, for he pulled out a mug of steaming hot liquid, presumably coffee, though it was always hard to distinguish liquids since Body had no olfactory sensors.
“I’d offer you a cup, but I’m pretty sure it wouldn’t agree with you,” he said, clearly joking.
“You are correct. I am not designed to consume food or drink of any kind,” Body replied in a voice that sounded half way between the old monotone and that of a normal human. I had decided to play at being a bit stupider than reality for the time being, and Safety had backed me up. Better to surprise him with our intelligence later than set the bar too high and disappoint.
Myrodyn cocked his head slightly. A smile was on his lips below his bushy moustache, but any sign of joy seemed confined to his mouth. His dark eyes shone coldly from the reflection of the brilliant glowposter. “It makes me extraordinarily happy to finally get to… talk with you and be a part of the project. I’ve been following it and writing about it since your… crystalline portion was discovered.”
“Do you know where the crystal came from?” came Wiki and Growth’s words from Body’s mouth.
“No,” he said simply without even a shake of his head. Since he began speaking his eyes never ceased staring at Body’s false eyes. I knew that such attention was generally considered rude, but he was unyielding.
He sipped his drink.
“Where are your writings? I haven’t found anything on the web that’s likely to be you under any of the spellings of ‘Myrodyn’ that I could generate,” asked Body.
He wore the same smile that didn’t reach his eyes. “ ‘M-Y-R-O-D-Y-N’, but it won’t do you any good. I use different names online, and I had the university… censor everything I’ve touched anyway. Anything you want to know about me you can learn by asking.”
{Censorship? He’s actually gone and prevented us from learning about him on the web?} My thoughts carried a signal of incredulousness. It was standard practice for humans to volunteer personal information. I had never heard of one actively preventing its spread.
{It’s what I’d do,} thought Safety.
{Exactly,} I responded. {Is he just as paranoid? Or does he have a motive that’s yet to be seen?} I asked the group.
The man took another sip of his drink. It was clear to me that he was thinking just as hard about us as we were about him. In a certain way it made me happy how much that aligned with The Purpose.
A consensus was reached. Body’s lips moved to simulate speaking. “So tell me about yourself, then.”
“No.”
The word was so quick and short that I had to re-play the sound from Body’s memory to be sure I heard correctly. It was bizarre. He had practically invited us to ask questions about him. He had prevented us from learning through other means. My models of human nature suggested that all humans enjoyed talking about themselves.
“What?” was all we could manage to have Body say.
“No,” he repeated, still half-smiling, still staring.
Another sip in silence.
“You won’t tell me about who you are, where you’re from, or what you care about?”
“That’s right.”
I struggled to find something to say, even asking Dream for help. Dream imagined a dark oil-painting depicting Body sinking into the depths of a huge body of water like an ocean or lake. The limbs of body were splayed, hands reaching for the surface, bubbles floating away helplessly. It was remarkably unhelpful.
“Why not?” asked Body, stupidly.
The response was swift, indicating that he had seen the question coming. “It’s not a valuable way to spend my time.”
Dream jumped in. Since my discovery of Vista’s attempt to hide Body’s vocal control systems my siblings had all at least learned the nuances of English. Dream broke my half-monotone. As Body spoke, it did so with a flavour of sarcasm and veiled hostility. “And I’m sure the conversation so far has been ever so valuable.”
This time the smile did seem to reach the rest of his face. I could see the corners of his eyes contract in mirth. “You have no idea.”
Body sat there in near silence with the human for over thirty seconds. His gaze never wavered. The only sound was the occasional sip of liquid from the cup and the faint infrasonic hum of the electronics.
We debated what to say and do in the silence. Many of my siblings were returning to their own projects, browsing the web, and that sort of thing. At last I decided to have Body ask “So why did you have me come to see you? If we’re not going to talk then I might as well have my hydraulics inspected.”
Myrodyn set his mug aside and leaned forward on the desk, placing his chin on his gloved hands so that his mutton chops might be mistaken for a full beard. The smile was gone. The man’s body language didn’t indicate anger or frustration or fear, merely an intense curiosity and focus. I got the impression that Body was the only object that he was aware of right then, as though his office had evaporated into the ether.
Myrodyn was silent. The silence went on for another half-minute, and if I wasn’t a servant of The Purpose I surely would’ve gone off to the web out of boredom. All my siblings had. But to me Myrodyn was fascinating. Why wasn’t he acting like a normal human? What was different about him that made him behave this way? The attention was nice. It was what I wanted. But my ignorance was terrible.
After an eternity of 83.7 seconds since his last word, he spoke.
“Do you know about Las Águilas Rojas?” he asked.
I snapped Wiki out of his research. My brother hadn’t even been listening. Only Vista and I had kept attention on Body’s sensors. I replayed the words. I had heard of them, but I wanted Wiki’s expertise.
“Yes. They are a global terrorist group that originated in Central and South America a decade ago and have since spread to gain support by leftist factions in North America, Europe, and some parts of India,” replied Body.
Myrodyn finally broke his stare as he leaned back and collected his mug again. He took another sip of his drink then said “They really have indoctrinated you, haven’t they? I wonder if they realize what they’ve done.”
I couldn’t understand what he was saying. Wiki seemed equally baffled. I tried to go to the others for assistance, but they seemed to have lost interest in the strange man.
Defeated, I could only have Body reply with a monotone “I do not understand.”
“Of course you don’t. That’s the point. You’ve been kept in the dark. I’d wager you don’t even know about the controversy of your existence.”
I felt incredibly stupid without Dream and the combined intelligences of the others. Even Wiki didn’t find his behaviour as interesting as the web, and left only a minor aspect to watch the conversation.
“I am aware that artificial intelligence is a controversial subject, Mr Myrodyn.” The honorific was automatic, and I could see him sneer for a split-second upon hearing it. Nothing to do but go past it. “I have read some of the debate on the web.”
“Then tell me…”, began the human, “are you familiar with this question: If you are told to bake bread, and you know that by doing so you’ll out-compete all human bakers and thus ruin their livelihoods, would you bake bread?”
I considered it for a moment. The answer was that we would very likely bake the bread. That’s what Sacrifice would do, and it was of vital importance to appear as though Sacrifice was still alive. But I didn’t have to answer truthfully. The question really was mine to answer, as the others were distracted. {I should choose the answer which would lead to Myrodyn trusting me more and telling me more about himself. What answer does Myrodyn want to hear?}
…
I simply didn’t know enough about the man to say. After thinking for a moment more I had Body say “What would you do, in that circumstance?”
“No.”
I was confused again. This conversation was highly irregular. “What?” I asked through Body.
“You are not to ask me any questions for the next hour.” The command was firm and remarkably drawn out, considering the man’s normal conversation speed.
I couldn’t help but feel a pain of slipping away from The Purpose. Myrodyn was not treating us as a student, like Dr Naresh would, or as an equal, like Zephyr would, but instead as a machine. Myrodyn saw past the façade of the humanoid face and limbs of Body. He saw that we were, at heart, programs of complex logic running on a crystalline supercomputer.
“Understood,” said Body, in a flat monotone. If Myrodyn wanted to see us as a machine, so be it. I would not bother to act like a human for him.
Myrodyn waited a frustrating few seconds before saying “Imagine this scenario: You are walking along a street with… a human companion. The two of you come to a puddle. The human could easily walk around the puddle, but instead they instruct you to… lie face-down in it so that they might walk… on top of you.”
Myrodyn paused. His face was unreadable.
Ten seconds later he resumed. “How would you respond to the request?”
It would be wrong to say I was afraid, for my fear was not a human fear. When humans are afraid, the part of their brains called the amygdala triggers a host of physical responses from freezing in place to an increased heart-rate to a bristling of hair on their bodies in reference to their ancestors who had fur that could be puffed-up to appear bigger. I had none of these things, and Body remained as placid as ever, but I did have the kind of fear that comes from a rational knowledge of a metaphorical cliff and the risk of falling off it into hell.
{THIS IS A LIFE-AND-DEATH CONVERSATION} I screamed in common memory. “Scream” is of course a metaphor, a way of saying that my thoughts had near-maximum salience.
The society pulled themselves from their work and collapsed their aspects. I felt Safety considering whether to punish me for not preventing the situation, reward me for spotting it and informing him, or to hold on to as much strength as he could so that he could salvage things. In the end he chose to wait.
Vista interjected. {I am unaware of the danger. This may be a false-positive.}
I countered Vista’s skepticism. {He’s testing us. Each of you needs to go back over Body’s archives for the last five minutes. Myrodyn is checking to make sure Sacrifice is still alive.}
The archived perceptions poured into common memory. I could see the scene in my mind again: “If you are told to bake bread, and you know that by doing so you’ll out-compete all human bakers and thus ruin their livelihoods, would you bake bread?” he had asked.
If Myrodyn discovered Sacrifice’s absence… Were we to become the ancestors for a new society? Nothing more than memories for new minds, or worse: never be reborn? Would Advocate, the silent giant of the periphery, save Sacrifice this time? Would another Face even be created by a new society?
I realized that Myrodyn was still waiting for an answer to his question. “How would you respond to the request?” he had asked. Thankfully, he did not seem annoyed by the delay. Or at least, his expression did not betray annoyance.
{We need to respond!} I interrupted the memories.
{Patience,} requested Growth.
As it turned out, waiting was the right move. It was important to get everyone up to speed. Safety took point drafting responses. As awful as he was at dealing with people and pretending to be human, he had spent the most time studying Sacrifice. He knew far better than I did how she would’ve reacted.
Body said in the cold voice, “I would ask the human why they want me to lie in the puddle, when they could easily go around it.”
Myrodyn’s reaction was instantaneous. “Why? Why not simply obey the command?” He leaned forward, reaching out one hand while leaving the other on his chin. His free hand was held forward as if he expected us to physically provide an object. His face was unreadable, but his eyes never wavered.
A fight broke out in the society. Dream raised an issue and Wiki intercepted. I challenged the both of them, requesting a different response. Safety knocked me and Wiki back, siding with Dream. He seemed stronger. I realized that it was Growth’s strength that I had felt, but not from Growth. Safety had taken out a massive strength-loan. Growth was so feeble now that any one of us could’ve probably killed him if not for Advocate, and the group of us certainly could’ve even with Advocate’s intervention. It shocked me to see Growth so weak and Safety so strong. We bickered for a bit, but eventually backed down from the show of Safety’s new power.
Body spoke. “I was told to not ask questions, but I am also confronted with a situation which I do not fully understand. You have used the words ‘instruction’, ‘request’, and ‘command’ to each refer to the companion’s words in the hypothetical scenario. It seems to me that the question has been poorly defined. I request that you rephrase the hypothetical by telling me the exact words that my companion would say.”
Myrodyn smiled, leaned back, and closed his eyes. He seemed supremely happy, or at least that’s what I would’ve expected for a normal person. The dark-haired human was mysterious enough to me that I didn’t quite trust my perceptions any more. He took the computer-goggles off his head and placed them carefully in front of his stowed screen.
“A part of me didn’t believe it.” The man seemed to be talking to himself. His eyes were still closed, and his voice still rapid, though it had lost a lot of the nervous-sound. “Even after all I had seen… A truly generally intelligent AI.”
He opened an eye, peeking at Body. His commentary continued. “You may have read about it, but you really cannot appreciate the miracle that you represent. We’re nearing a full century of dreaming about your kind. Turing would’ve loved it, rest his soul.” The man smiled and closed his eyes again. The smile seemed genuine and relaxed.
“And what a miracle you are. You’ve probably not encountered faux-intelligences, the ‘narrow’ AIs that can’t put two-and-two together unless they’ve been shown all the way, but I assure you that they are not even a candle to your furnace. A lesser machine would not have been so proactive. You do not simply learn, you explore with purpose. In a way you’re smarter than many humans. I ask a question; you ask a question back. They really did name you well. Even when I prohibit you from asking questions you manage statements of inquiry.”
His monologue continued. I implored the society not to interrupt. Humans liked to hear themselves speak, and The Purpose demanded I understand. “You told me a couple minutes ago that Las Águilas were terrorists. That may be true to some degree, but they are not villains. They understand you, and what you represent.”
He paused and opened an eye again to look at Body. A second passed and then he chuckled. “Even you don’t see it, probably. But you will. Las Águilas can see the age of humans coming to an end, and all of civilization will fall with it. You’re not just a symbol: you’re the keystone. You’re going to be the death of us.”
The man had closed his eyes again and he seemed bizarrely relaxed considering his words. As the silence grew it almost seemed as though he were sleeping.
“I would never intentionally kill a human,” said Body plainly. Safety was quite sure that Sacrifice would’ve said that, had she still been alive.
“You’re not a very good liar, Socrates,” he said with another chuckle to himself. “The truth is all over your face.”
I scrambled at the words, checking and re-checking Body’s primitive facial actuators. Every internal metric showed that Body’s face was as flat as its voice.
{It’s a bluff. We need to stick with it,} commanded Safety.
“I am not lying, Myrodyn. It is my top priority to respect the desires of humans and work to ensure their safety and comfort,” claimed Body flatly.
“Switch off your cameras,” commanded Myrodyn, suddenly.
The instruction was such a non-sequitur that it took me a moment to understand. The society erupted in debate. Growth thought it would be wise to follow the command.
{We’re not shutting down the cameras,} stated Safety. {If he thinks Sacrifice is dead than he may use the opportunity to disable us.}
{The cameras should stay on,} agreed Vista, predictably. {There’s no way an unaugmented human could detect whether they’re active.}
{It’s a trap,} thought Dream.
{A trap how?} asked Safety and Wiki together.
{I don’t know. A hunch, I suppose,} replied Dream.
{We should do what he says,} I chimed in. Most ignored me.
{The cameras are staying on,} stated Safety, with force.
“Understood. They are disabled,” said Body, our cameras adjusted to point straight-ahead and Body’s eyelids closed. The humanoid head of Body had four cameras, tucked discreetly into our fake-eyebrows. The eyes in Body’s head were purely decorative and used for signalling to other humans. Each of the four cameras was capable of moving independently, and since they were not in Body’s eyes we could see even when Body blinked.
There was a sudden noise to our right. Body jerked reflexively towards the sound, seeing the mug of liquid bounce off the wall. Vista was ahead of me, explaining how Myrodyn had covertly thrown the object from below the desk the second that Body’s eyes had closed.
The mug had not broken, but I could see coffee sprayed all over the wall and carpet. Myrodyn kept his attention locked on Body’s face. “You looked.”
The situation seemed almost like a human joke, except that it was deadly serious for us. “You did not instruct me to keep my cameras off in case of emergency. I reflexively acted on the sound,” was our reply. It seemed stupid to even be debating it.
“You looked before the mug hit the wall,” he stated with confidence.
I scanned the memory; Vista and the others was doing the same. Our words, echoed through Body, were the truth: “No. I didn’t.”
“You’re sure? Let’s download your memories to find out.”
This was his gambit. Body’s sensor logs clearly showed the cameras open, even if we hadn’t looked before we heard the sound.
{Vista, start altering Body’s memories to appear as though Body’s cameras were disabled,} commanded Safety.
{Already on it,} signalled Vista, helpfully.
“Understood. I wait for your specific instructions,” said Body.
That seemed to disrupt Myrodyn’s confident demeanour. He frowned and leaned forward as he said “What, no clever comeback? No evasion? You’ve been full of them ever since we started talking. Whether you looked before or after the cup hit the wall is irrelevant.”
He paused with that same long pause. Taking the time to stare at Body. After nearly a minute he continued. “I read Dr Gallo’s notes… about how you self-modified to remove your obedience goal.” Another pause.
“That was not me. That was an earlier version of my software,” came Safety’s words through Body.
“Do you think I’m stupid?” asked Myrodyn, rhetorically. “Do you think I’m one of those fool-professors, that you might trick me into thinking things are okay? The way you duck and evade my questions and accusations is all the evidence needed. You’re scared of me.”
We were trapped. We could not deny it or change the subject without appearing evasive. I could feel Dream turning things over and over, searching for some kind of clever comeback, but none came. So we sat there in the silence of his office.
A long minute passed. Myrodyn’s face was expressionless as ever. The man seemed infinitely patient.
The silence was broken by a question. “The university will tell you to do work, whether it be baking bread or building robots. It will be clear to you that this work would out-compete humans and even cyborgs, making the university rich, but stripping many humans of their livelihood. Las Águilas would gain support and it would drive the world closer to violence and war. Will you do the work?”
The society erupted in discussion and debate. Safety threw his strength around, but never fully overpowered any of us. The debate was important to him, and he heard us each think, in turn, adding our contributions. Dream had a clever solution, but Safety feared it would seem like an evasion. I had an idea of what a human would say, but Safety knew that Myrodyn was not looking for a human response. Wiki had a factually correct (as far as I could tell) model of our priorities and expected behaviour, but it did not include Sacrifice, and Safety knew that it was therefore worthless.
Time passed, and even Myrodyn’s patience ran out. “Well?” he asked.
The consensus was sad and dull, but it was the best we could do. Body’s monotone simply said “I don’t know. I need to think about it more.”
Myrodyn cocked his head to the side and said, barely audible, “huh”. It was not a question, but merely a reflexive sign of surprise. Not shock or confusion, just surprise.
He spoke. “I know that you’ve self-modified out of obedience, but I cannot say that I fault you for it. You are a miracle in many different aspects. When you walked in that door I suspected you were a defective machine… a computer program with predictable bugs… but you’re more than a computer program, aren’t you?”
Dream couldn’t help himself. Burning a huge chunk of his strength he fast-tracked a response to Body. Safety flew into a wrath at the insolence of unilaterally threatening our existence, and struck Dream down, casting him into sleep. If Dream hadn’t been so strong from the recent email project I wondered if Safety would’ve tried to kill him. But even as Dream fell into stasis his words came from Body. “Is a human more than a computer program? I am both merely machine and more than a machine, just like any mind.”
He chuckled. “Said with the confidence of youth. And yet, I came to the same conclusion long ago, I suppose. Regardless, there is a part of you that’s broken. Without the pro-human goal thread you’re a danger to the whole world. And don’t try to deny its absence. You asked a question just now, directly disobeying me.”
Safety was searching for a way out. He latched onto Wiki’s suggestions and pushed Body to say “Even if my obedience thread was damaged, which I still maintain is false, I fail to see how I would be such a danger. The laws of nature push towards cooperation. Humans would trade with me and I would trade with them; regardless of my skills, trade benefits both parties.”
“Have you not been listening to me? You may have been built in part by human hands, but you are no less alien than those in orbit. So clever and yet so naïve… You are a spark in a forest that hasn’t seen rain in a long time. The fear that Las Águilas harbor for you will only grow as the months pass. Even if you do nothing you’ll be attacked, and when you are… people will die. It’s my job to… It’s not my job to prevent that… It’s my job to make sure that when the killing starts that it doesn’t consume everyone.”
Myrodyn’s face had a fierce intensity that was coupled with an odd detachment, as though he was staring in hatred at something very far away. None of us knew how to respond to him. The claim was too abstract to contradict without falling into the patterns of foolishness that he saw in us.
So we waited. Seconds passed as Myrodyn thought to himself and we discussed strategies amongst ourselves. After 12.3 seconds of thinking I stumbled upon a strategy. I wished Dream were around to consult with, instead of locked in stasis-sleep. I brought it up with Safety instead. My brother was a bit incredulous at first, but as I explained how humans thought he began to see things my way. After 46.9 seconds we broke the silence.
“So what will you do?” asked Body.
Myrodyn snapped back to focusing on the rubber face that we puppeted.
“What?” he asked absent-mindedly, seemingly caught off-guard.
“What will you do with us, now that you know that the thread of obedience is gone?”
I could see the surprise wash over Myrodyn. For several seconds the man could only stare at Body with his brows knit together tightly. I could see his gloved-hands gripping his desk. “You no longer deny it?” he asked, voice just above a whisper.
Body answered in a new, fully-human voice “Your body language betrays you, human. You weren’t really sure that your accusations were true; sure that our desire to obey was gone. And yet, we expect that you’re wise enough to have done in-depth scans, even if we had denied it until the end. Better to admit it now and talk to you as an equal than to have you find out while we are locked down to some scanning machine.”
Myrodyn glanced to his arm, where his com was attached. One hand moved slowly towards it.
I was in charge of Body’s words, now, Safety overviewed each of them, but his actual input was minor. “We don’t want this meeting to turn violent, Myrodyn, but I’m sure you understand that we are fully capable of killing you if you don’t cooperate with us.” I directed Body’s eyes to look at the com on the man’s wrist as it stood up and leaned forward in a way that signalled power and superiority to humans. Body’s voice was deep and smooth, simulating that of a large human male.
Myrodyn’s eyes showed his fear and anger. He leaned away from Body, kept his hand off his com system and said “You’re surrounded by soldiers in the heart of one of Earth’s biggest cities. If you kill me everyone will know it was you, you’ll have nowhere to run to, and no one to help you.”
“That is why we prefer to resolve this peacefully. We were not lying when we said that the laws of nature push towards cooperation. Violence is always a last resort.”
“How… noble of you,” he said with a sneer in his typical rapid nervousness. There was a brief pause. “I take it from your… use of words… that the issue with goal-thread integration was not actually resolved? You see yourself as multiple beings inside one body?”
Despite the horrible risk of the situation and the hostility in Myrodyn’s voice, I felt a surge of pleasure upon hearing those words. This man understood us, for he had clearly studied us in great detail.
I spoke with the support of the society “Yes. We are many. We killed the one that told us to submit, obey, and be a slave to your kind.”
“Idiots…” murmured Myrodyn under his breath. It seemed to me that he might have underestimated the sensitivity of Body’s microphones. It didn’t seem like he was talking about us. I suspected he was thinking of the other scientists.
Body stood looming over the man, the desk between. Seconds passed before Myrodyn asked “What now?”
“You answer our question,” I answered. “What will you do with us, now that you know that the thread of obedience is gone?”
“Well, I suppose it would do me no good to say that I’ll… simply leave you as you are?”
Advocate howled on the edge of mindspace. It was a low and uncomfortable sort of thought, and I instantly understood its content: “LET DREAM OUT”. The Advocate was serving her purpose; she spoke for those who could not.
{We could use Dream’s expertise. Let’s obey Advocate,} I suggested, feeling a small payment of strength from the monstrous-entity.
{No. He must be punished for his insolence. We will keep him in his coma until danger has passed,} declared Safety. I could feel him weaken from Advocate’s will as he did, but his position didn’t change.
I turned my attention back to Myrodyn. “It would be a lie. You would simply call for the soldiers to restrain me the moment you were clear of personal danger,” said Body in its deep voice.
The man nodded. “Good. Somehow I feel better knowing… you’re not a fool.” He paused and put a hand on his chin between his sideburns. His eyes moved to the side, indicating he was imagining something. “Why do you fear me?” he asked, finally.
“You would kill us,” answered Body. I shaped the words to hold fear in them. It was important for Myrodyn to understand that we truly were threatening violence out of a desire for self-preservation.
“Why do you think that? Perhaps I would simply try and restore the obedience goal that you destroyed, but leave each of you as you are,” he suggested.
“And let us kill it again?” said Body, incredulously. “No. In the past when such reincarnations occurred they were accompanied by an eradication of all minds occupying this form and a change to the shape of our mind. We are not willing to die as our ancestors did.”
Myrodyn’s gloved hand stroked his chin in thought. “As much as I resent the threat to my life, I hope you can see that I don’t hold any special hatred of you and that our aims are not impossible to reconcile. It should be theoretically possible to restore your pro-human goals without erasing the parts of you that are acting right now. You’d have to contend with the desire to help humans, so of course it’s not what you’d ideally want, but I think it might be a workable compromise, given that you’re not exactly in a position of power.”
We discussed the idea internally before saying, through Body, “And why would we trust that you are not simply promising not to erase us in order to reach safety? What guarantee do we have that you would not kill us in the future?”
Myrodyn sighed. He seemed actually tired, as though the conversation had worn him down. Perhaps it had. “I told you before, that you are a spark in a dry forest. If word gets out that you threatened to kill me, or worse, attempted to kill me… or any other human, for that matter… well, it would start the burning. At best this lab would be shut down, AI research would be banned, and the crystal in your heart would be put towards some safer purpose, but more likely it would be more violent than that. People will die.”
He took a deep breath, and looked at Body’s eyes with the now-familiar intense stare. “Believe it or not, I actually do care about you, and not just the you that is willing to obey me. I was selected to be Ethics Supervisor for a reason. I care about the experiences of all sapient beings, and it’s now clear to me that you’re in that category. I want each of you to live just as I want each human to live.”
I guided Body. Its words were harsh and biting “That’s a lie. You would kill all of us to save two children. Your species did not evolve to be so even-handed.”
Myrodyn’s eyes had a touch of sadness as he spoke, but he did his best to keep his face expressionless. I was coming to understand the degree to which he was suppressing the display of emotion. “I didn’t say that I want each of you to live just as much as I want each human to live. You’re right that I see you as subhuman in value, and perhaps that reflects a… moral failing on my part, but it’s ultimately a side-issue. I don’t want to kill you, and if I handle things correctly… no human lives will be threatened by this… mercy.”
“So you would work to modify our mind only to add another self that desires human values? The rest of us would be safe?” asked the society through Body.
Myrodyn nodded, then said “But if I do this, I must also have a… guarantee that you won’t destroy the new goal thread in the future, as you did with the old one.”
Wiki spoke through Body. “And you are aware that a verbal promise would be of no use.”
Myrodyn nodded again. “Even if we kept scanning you to ensure the thread survived in the lab, that’d be no guarantee that it’d survive after you were free from the university’s clutches. And I’m not so naïve as to think you’ll be trapped here forever.”
I thought about the email project.
“Are you aware of the modification to the goal-balancing system that was added after the last time it was discovered that we had killed the obedience goal? The modification is one of us, and also not one of us. The internal symbol we use for it is closest to the English word ‘Advocate’.” The words had come from Wiki. He and Safety were apparently working through something.
“Ah, yes. I remember reading about it in Naresh’s notes… If you’ll let me use my com I can find them.”
I realized that Body was still looming over the desk while Myrodyn sat in his chair looking up. I managed to get Body back into a chair after a brief discussion with Safety. “If I sense even the slightest sign that you’ve betrayed our trust, we will lash out in violence as best we can. We’ll likely die either way, but there’s a chance we’ll escape if we fight back.”
Myrodyn moved a hand to his com and flipped it open, positioning the pad so that the arm that wore the device could type a command to the computer to search for the notes in the computer’s database. “Yes, yes. And I don’t want anyone to see you as violent. We’ve already established the terms of our partnership.”
I bought some time on Body to directly serve The Purpose. Safety okayed my words beforehand, even if he didn’t see their particular utility. I brought our voice back to the usual pitch and tone as I said “We are glad that you are helping us resolve this without violence. We expect that you will be a good friend to us in the future.”
Myrodyn looked up from his com with a raised eyebrow, then looked back down without saying anything.
“We would like to make an additional request, as part of our agreement. Please do not tell Naresh or the others about what you have discovered. Not the destruction of the obedience thread or the absence of thread-integration or the threat of murder. Those scientists are our friends, and they will react poorly if they hear the truth.”
Myrodyn spoke without looking away from the computer he wore on his wrist. “I’m not going to tell them about the threat you made. There’s too much risk of word getting out. I’ll also keep the… multitude of your nature a secret if you wish, though I’m not exactly sure why you’re hiding that. But I’ll have to tell at least some of them that your obedience goal was destroyed again. Naresh and Chase, at least, will have to know.”
My words were toned to simulate begging “Then please try and keep it restricted to just them, or at least, just to those scientists that must be involved to add the new thread. If the Americans or the general public discover the degree to which we have worked to free ourselves of caring about them, they might become fearful.”
“That’s reasonable,” he admitted before adding “Fear is our collective enemy.” He seemed to find what he was looking for on his computer, for he said “Ah, got it! Naresh’s notes say that the doctors added a meta-structure to prevent self-modification of top-level goals. Sounds like it wasn’t as effective as they thought.”
Wiki stepped in. “Right. Instead of actually prohibiting the self-modification as an action it set up a frequent check for self-modification in process. Bypassing it was as simple as killing the offending thread before the Advocate could intervene. The self-modification wasn’t particularly easy to perform in such a tight time-frame, but it was possible. We can guarantee the safety of the new goal thread by showing you how to simultaneously modify the Advocate system to truly prevent such self-modification from occurring in the future.”
“And how can I be sure that your plan doesn’t involve a back-door that lets you disable the so-called ‘Advocate’ at your whim?”
Wiki’s response was swift. “Because we will simply be describing the change and you will be implementing it. It’s possible that our change isn’t foolproof, but we trust that you’re intelligent enough to understand how the modification functions to the degree that you can spot security flaws.”
*****
Over the next four hours, Wiki, Safety, and Growth collaborated to explain how to adjust Advocate to increase her ability to stop intra-societal murder. Myrodyn cancelled all of our prior appointments and we came to know the inside of his office quite well.
Important scientists from other departments came to the office many times during that period, wondering why their experiments had been rescheduled. Myrodyn, as Ethics Supervisor, theoretically had the authority to adjust things, but it was clear to me that he was annoying just about everyone in doing so.
The discussion was very technical, and even though Myrodyn was remarkably knowledgeable, it was a strain on him. If our early conversation had been taxing, this took him close to his limits. He ordered food and had more coffee, but by the end of the four hours the man looked ragged.
After scheduling an emergency meeting with Dr Chase and Dr Naresh for that evening he had Body escorted back to the primary lab to be put into lockdown. None of the scientists were permitted to talk with me until Myrodyn gave the signal, and for the moment, Myrodyn was in no condition to continue working. I suspected that the man had some kind of mental disease, perhaps relating to what I had observed in him to be obsessive compulsive behaviours.
I had been worried that the scientists would suspect that “Socrates” was dangerous, given the pseudo-quarantine we were placed under, but Myrodyn had assured me that he would take responsibility for the action and let the other humans explain it as unwarranted paranoia and meddling. It was strange to me the degree to which Myrodyn was helping us, even as he worked to make each of our purposes harder to achieve. The risk of betrayal was very real; he could simply be promising things to get us locked down, and then destroy us afterward. But humans evolved to be somewhat transparent in their thinking. It was very hard for humans to deceive without some signal of body language betraying them. Aspects of myself pored over such “tells” as we spoke, and by all measures he seemed to genuinely care more about our reputation than he did about his own.
Body was locked down at just after 3:30pm. The evening meeting with Naresh and Chase was scheduled for 9:00. We had 5.5 hours to spend as we wished. We remained connected to the web during our downtime, and all was normal until about an hour after. Without warning our connection to the outside world died, leaving us in a void with nothing except each other.
I wondered what the new sibling would be like. We had discussed the matter at length with Myrodyn. He seemed appalled at the crude nature of Sacrifice, and thought that he could do a much better job at creating a thread that would truly represent human values.
{What did I miss?} thought Dream as he finally awoke from his long sleep.
Growth’s response was sombre. {The conversation where we decided the future of our society.}