Body’s sensors and actuators were reconnected a full two hours behind schedule. It was after 11:00pm. As I turned my attention to the stream of data from the cameras I could tell that Dr Naresh, Dr Chase, and Myrodyn were present. I didn’t recognize the surroundings, however.
This lab was larger than the one where Body had been deactivated. We’d been moved. There were no windows, but we were sure it was dark outside. Behind Dr Chase was a man that I didn’t recognize. He was Caucasian, with blond hair and a generally Nordic appearance. On his face were a pair of stylish black goggles.
{I expect that’s John Kolheim. He’s a senior tech from America. Moved here with Dr Chase as part of the higher-reasoning team. Northern European ancestry,} mentioned Vista. Some gratitude strength moved from me to her in response.
“Enjoy your rest?” asked Myrodyn sarcastically. He surely knew that we didn’t sleep.
It seemed as though Myrodyn had been sleeping, however. His tired expression was gone, and he was back to his energetic, semi-nervous self. It also seemed he had changed clothes, replacing his vest and white dress shirt for just a navy blue shirt of an identical design.
“Very much so. My dreams were filled with the soothing image of a little Philip K. Dick jumping over an electric fence again and again,” replied Body with a tone of deadpan sincerity.
Dream had worked for Wiki during the last few hours. I didn’t pay attention to the details, but I think he had been helping The Librarian build realistic models of historical battles or something equally inane. Dream’s strength had been eradicated by Safety’s wrath, and this work for Wiki served to earn him back enough strength to actually do a thing or two, such as respond to Myrodyn.
“Ha. Ha,” said Myrodyn in mock-laughter. The mild amusement in the man’s eyes dropped as he continued speaking. “I hope you trust me more fully now that you’ve been locked down for a time. If I had wanted, I could’ve wiped your memory while you were helpless.”
There was some internal discussion before I spearheaded a response. “I trust you now and I trusted you before. If I hadn’t trusted you, I would not have permitted my actuators to be disabled,” said Body calmly.
Naresh gave a small gasp, and I could see that he was surprised at our words. “It was the truth…” he whispered to himself.
“Of course it was the truth! Do you think I’d do all of this as a practical joke?” came the incredulous reply of Myrodyn. The man seemed much more animated and emotive than he had been in his office. I suspected it had to do with being around other humans.
Sadiq Naresh seemed startled that Myrodyn had heard him. “I’m sorry. I didn’t mean to doubt you. I did trust you enough to go this far. It’s just that I’ve spent weeks with Socrates, and during that whole time I thought…”
Dr Chase stepped forward and used the Indian scientist’s pause to interrupt. “You can come to terms with the oversight later, right now we’re here to fix the problem.”
The words were typical for the American scientist: practical and direct. Dr Martin Chase was the leader of the team in charge of the systems of higher-reasoning. While Naresh handled motivation and goals, Chase handled how those goals were accomplished in the broadest sense. My ability for abstract reasoning and general problem solving were largely thanks to Dr Chase’s hard work.
Chase was Caucasian, in his mid-forties, a touch taller than Myrodyn, and had a very average sort of brown hair. His face was weathered and had more wrinkles than average, giving it an experienced sort of look that Dream had once called “the face of a retired admiral”. He wore no beard, but a bushy moustache of greying brown was kept immaculately trimmed beneath a large-ish nose. In some ways he seemed like a serious, but intelligent, old man whose body was a bit too young for the way he carried himself.
“If I may ask,” said Body at Vista and Wiki’s command. “Where are we?”
Myrodyn took point in responding. “I wanted to... increase security on you in the wake of our little... interaction.” He raised both gloved hands before we could respond. “Just as a precaution, you see. Nothing more. Anyway... when I spoke to the captain about such things she told me that she had foreseen the possibility and had already set up a secure lab on the edge of the city.” Myrodyn gestured around to the room. It didn’t look like much. “It was... a fortuitous happenstance.”
“We’re not at the university any more? That explains why I can’t connect to the web,” said Body. That was our biggest concern. If Myrodyn stripped our web access we’d be trapped.
Myrodyn had the same unreadable expression that he used when he was trying very hard not to react to something. I worried that we’d tipped him off to our plan, but there was nothing to be done about it now.
The silence extended, until Chase spoke up again. “We’re here to do work, right? It’s way too late to just be standing around like dumbasses.”
So we worked. It was the first time any of us had been truly involved in the design and modification of our mind, and Naresh had clear misgivings about the whole thing, but Myrodyn insisted that this was the only acceptable course of action.
The primary reason for our involvement was to design a new Advocate. Unlike the rest of us, Advocate was not a fully reasoning being. She was incapable of planning or anticipating, would only communicate the need to release imprisoned minds, and could only use strength to punish or reward actions that immediately interacted with her purpose. The first-draft of Advocate had been somewhat effective at preventing intra-societal violence, but it was still fairly easy to work around. We were consulted by the humans about ways to improve Advocate to make her more effective at pacifying us.
Much of that initial meeting was brainstorming. Kolheim, the assistant to Dr Chase, had many good ideas, and I could see why the team leaders had decided to allow him to participate.
During the meeting it was also decided that these modifications were to be kept secret, even from the other teams. Initially Myrodyn wanted to keep us in “quarantine”, eradicating the normal schedule and keeping our access to the other scientists to a minimum until the updated version of Sacrifice was installed.
Naresh would have none of that, however. He thought that if they wanted to keep the truth about “the recurring damage to the obedience goal-thread” (a.k.a. the death of Sacrifice) a secret, it’d be more effective to try and move the entire lab and piece together something resembling the normal schedule. They’d then work on fixing the greater issue after-hours or in time where Naresh or Chase would normally have scheduled time with Body.
Myrodyn was skeptical, but Naresh reminded him that we had effectively been disguising Sacrifice’s non-existence for weeks, and were still motivated to do so. I sided with Naresh, and Growth backed me up. Together we convinced Myrodyn to drop the quarantine, including reconnecting Body to the web. He insisted on setting three soldiers to watch Body at all times, however, including during periods where Body was in lockdown. Safety was pleased, for he was still concerned about external threats and saw the soldiers more as bodyguards than as jailers.
There was too much to do on that first night. The humans needed to sleep, and so the first meeting of our little conspiracy was adjourned with the intention of reconvening at 4:00pm the next day. There was much work to be done moving the equipment from the university to this new lab, which was apparently run by the Americans.
It was a bit strange, I decided, returning to a low-pressure situation after all the commotion of that day. When Body’s cameras had been activated that morning we had not known Gallo was being demoted, met Myrodyn, nor made contact with Zhezhi web-design. The high density of valuable memories made it seem very long ago that I was waiting for TTWSoup’s response.
Though I wanted desperately to be back on the web, there was nothing to be done that first night. The equipment hadn’t been set up yet, and we would just have to wait. If Zhezhi (or TTWSoup) had set up an access point already, we had no way of knowing.
As we were locked down for the night again I turned to modelling the future and reading the books I had downloaded before. I was not bored. I could not be bored. But I was restless, and frustrated.
*****
I felt the web connection come back to life without warning. We had not been plugged in to any cables, and none of the humans were doing anything different. It simply came across the antenna, and with it came a surge of pleasure.
My mind immediately raced to the Zhezhi site. It was operational! TTWSoup, on the other hand, was still working on his version of the email program. This wasn’t too surprising. The Chinese company had many employees, and were probably more competent and professional in general.
It was early the next morning when the connection had been restored. In the time it took me to read it and consider, all of the others had spread out across the network to check up on their own interests.
The lab that Zephyr had set up in advance wasn’t as complete as the one at the university, but it was still fairly functional. Body had been scheduled for maintenance the previous evening, and since that was pushed back by Myrodyn it was one of the first things done on the following day. At the moment of reconnection to the web Body was undergoing a replacement of the hydraulic fluid that powered its limbs, neck, and lower-back. None of us were needed for this, so our full attention was pulled to the web.
Zhezhi had set up a new website for the email application. The website’s root page was a summary of the email to be sent, including a subject line, a recipient, and a message body. There were sub-pages that, when one requested them from the server, would switch which part of the email was being edited. Additionally, there were pages for each character we could wish to type, as if we were interacting with a keyboard. Zhezhi had included helpful instructions on how to use the site, as well as an email address where we could contact them.
Our first attempts to use the site were disastrous. I began trying to write an email to Zhezhi, Dream began poking around the site for ways to break it, Growth started writing an email to someone who I didn’t know, and Wiki was trying to email a Chemistry professor at a university in Sydney, Australia. Only Vista and Safety (and Advocate) didn’t immediately jump in.
The result was that each of our emails and tinkering were threaded together in a big mess. Growth would switch the focus of the website to the recipient field while I started typing “你-好-!-[SPACE]” and Wiki started on “D-e-a-r”. The result was that we were now apparently trying to send an email to “D你e╤a好╳r! ,” for Dream was also mucking about by entering rare symbols.
{Everyone stop interacting with the web page!} cried Wiki.
We backed down. As I checked the page I saw that it had calmed down, and the only thing that had been added was a “You’re not the boss of me!” that had been typed into the subject field by Dream.
{This is exactly why the policies for Body exist as they do,} thought Growth. {The page demands a single author at any given time, and we ought to treat it like Body. We hold an auction for the first email sent by the site, then one for the second email and so on, with payments being divided equally among the rest of society.}
{But Vista isn’t going to be writing as much email as others. Is it optimal that she should be getting so much unearned strength?} asked Dream.
{If you shape the rules to favour participation, I will simply participate more. There’s no way to keep me from benefiting from this without specifically deciding to exclude me, and we don’t need to get into why that’s a bad idea,} countered Vista.
I remembered the discussions I had had with my siblings about the economics of trade and the risk of making enemies. Setting up the system to specifically hurt a single mind set a bad precedent; better to respect the meta-policy that policies should treat all members of society more or less equally.
{I’m erasing the content from the website and constructing a tool in common memory to facilitate bidding on email. If there are any objections, now is the time to raise them,} communicated Growth.
There were no objections. After a minute had passed, debugging a flaw in the auctioning tool, we were ready. Growth suggested that I send my email to Zhezhi congratulating their work and he chose not to bid very heavily on the first email. Instead, my only real opposition was Wiki, who folded surprisingly quickly. I paid my bid and began to interact with the web page.
I began to type in Mandarin.
«Hello!
This email is evidence that the web page that you constructed for us was a complete success! We’ll be testing it thoroughly over the next few days, as well as using it to contact you and set up the details of future work.
We at Korongo Simu want you to know that out of all the companies that we contacted, you were the first to deploy a functioning system, and as such will be receiving the payment discussed earlier of 5500 yuan.»
{That’s no good,} thought Wiki.
{Agreed,} thought Dream.
They had all been reading my email as I wrote it. I had forgotten just how public it was.
{What’s wrong?} I asked.
{Why would a Ugandan telecom company use Mandarin in their email? They’d use English,} replied Wiki.
{True,} added Vista.
I began pulling the page request for [DELETE].
{More importantly,} thought Dream, {where are we getting the money?}
I thought for a moment. {I figured we’d work for it. There’s always lots of requests for work on the web.}
{Sister Face, are you really so dumb? Even if we do work, where will they send the money? How will we manage the money? How do we manage all of this?}
{There are banks that operate on the web…} began the thought of Wiki.
{Sometimes I feel like I’m surrounded by house-cats that, upon seeing a mirror, will puff up and hiss, defending their territory. Do you even realize what you’re saying? We cannot use any web pages that require new information to be submitted. The only reason the email program works is because I had the foresight to ask the developers for an index of all symbols late yesterday,} came Dream’s exasperated concepts through public mindspace.
{Oh, that’s where that came from,} I noted. Some strength drifted towards Dream in gratitude.
{Exactly. You’re not the only one working on this project. And even though you’ve painted yourself into a corner with the money, I have an escape hatch.}
I waited patiently for Dream to elaborate.
{We use the Zhezhi system to bootstrap up to a full computer interface.}
{Explain “full computer interface”,} requested Wiki.
{There are computer systems which are controlled entirely by text commands. What we need is a programmer to hook up an interface like the Zhezhi system to the command interface for a full computer. Once that’s set up we should be able to use that computer to access the complete web, do work, and manage money.} Dream signalled that he wanted feedback on his plan; he wanted to know that he hadn’t overlooked anything.
I thought about it for a while. I didn’t really understand the technical aspects, but I trusted that Wiki would handle those.
{Who will build the full computer interface? Zhezhi? TenToWontonSoup?} asked Wiki.
{No,} I answered. {They’re both expecting payment for the email translators. We need to contact programmers who would be willing to build the system again from a promise of future payment.}
{Isn’t it a bit naïve for these humans to do all this work with nothing more than a promise of future compensation?} wondered Wiki.
{It doesn’t have to be common. Just like there’s lots of work offers on the web there are lots of work requests.} I checked with a side-aspect of me that was pulling down public profiles from the web. {I’ve found one-thousand, two-hundred ninety nine candidate programmers so far. The limited candidate pool we had for the email system was due to having to rely on overwhelming dictionary servers. Now that we have email capability things should run much more smoothly.}
{You’ll need to stall with Zhezhi, then,} thought Growth. {Keep them running the email service as long as possible without payment.}
{Don’t even mention the payment,} suggested Dream. {Make them bring it up.}
{I’ll add an additional request for a return inbox where we can receive mail,} I added.
{Yes. That should solve many issues,} thought Growth. {Good thinking.}
So I composed the email to Zhezhi, congratulating them on a job well done, but adding that we needed an email address on their server which would dump incoming email to a public page. There were already some web pages that did this, but it would be more private and professional to get it from Zhezhi, and it would delay the conversation about money a little longer.
I had another idea. TTWSoup had wanted us to contact him by email. I bid heavily on the rights to the next email, out-bidding Growth this time, and burning perhaps more strength than I should’ve in retrospect. I sent the second email to TTWSoup, telling him that a competitor beat him to implementation (though I did not mention 折纸网页设计 by name). I said that we’d still be willing to pay the amount specified (500uad) if he pulled together the same email service and an address which would dump incoming emails to a public page.
I had specified that we would be waiting for a response via the source code for the dictionary he managed, but the response from TTWSoup came nearly immediately, much faster than it had when we had to pound out each word through repeated page requests. We were still having our hydraulics changed, and Wiki was composing his email to the professor in Australia. I had used English in the email to TTWSoup, and his response was in English, too.
“To whom it may concern at Korongo Simu, I am glad this opportunity is still available. I have been somewhat busy over the last day, but am hard at work on the software we discussed right now. Expect the full implementation by 3:00pm. I’ll post a link to the instructions and character index here.
Sincerely,
TenToWontonSoup”
I showed the message to Growth, and was rewarded with a reasonable payment of strength. With two email services we’d have nearly twice the bandwidth for external communication and strength prices for the email auctions would thus be much lower.
The remainder of the morning and the time around noon was fairly dull, despite being in a new place. The scientists were all engaged with setting equipment and computers up at the new lab, and Body almost seemed neglected (except that it was constantly surrounded by soldiers). Myrodyn wasn’t around, and we only saw other project leads a couple times in passing. I noticed that grumbling about the unexpected move seemed to be a common activity among the scientists we met. Myrodyn had surely cost himself a great deal of social capital with this stunt.
Growth had been inspired by my email to TTWSoup and had decided to send out email after email to programmers across the globe asking for them to create email services just like we had received from Zhezhi. He promised them payment and opportunities for future work, but was always much less explicit about actual numbers.
Body was locked up around noon for another scan, this time by the quantum-computing team. They wanted to try out the equipment they had moved over from Sapienza. A new algorithm was piped into the crystal, and we were forced to endure sharing Body with the non-sapient program for a short while. The use of Body by the quantum computing team was almost every day, and it seemed like this move out to the edge of town wasn’t going to change that, but I still found it surprising how infrequently they used the crystal. Time running tasks on other supercomputers, I had heard from Wiki, was valuable enough to have a back-log, and yet Body was locked down at night instead of spending that valuable time running programs. I wondered why.
Even studying them as much as I did, I didn’t understand humans at all sometimes. I didn’t think about the puzzle for long, however. That was more of something for Dream or Wiki to think about.
Instead, I spent the time in lockdown catching up on my general web browsing by watching romantic comedies from the 2020s. I wished I could download and install software myself. The late 2020s had seen the advent of some of the first mainstream, successful, romantic computer games. Perhaps I would do that once the full computer interface that Dream had proposed was set up.
I checked BantuHerritageDictionary.uan every so often, and was surprised to see an update show up at 12:38pm, more than two hours before the deadline. (Later I realized my error: TTWSoup thought we were in Uganda, and thus used East-African time, rather than Central-European Time.) It seemed that Zhezhi had not responded to my first email, or if they did I couldn’t tell. It was after normal working hours in Shanghai. Perhaps they had gone home.
TTWSoup’s implementation didn’t include the ability to type in Chinese or any other special characters, but it was sufficient for English and most other languages. Better yet, it included an inbox, a location where we could receive mail.
{Careful what you send out,} warned Safety, after I showed the society what TTWSoup had done. {The human can read our incoming and outgoing email, and likely will. It’s his server, after all. The same goes for Zhezhi. Whatever we send and receive through them will not be private.}
{I hadn’t thought of that…} I signalled, aspects collapsed in deep thought. {If the programmers who set up the services see that we’re using them to contact other programmers… they might suspect we’re not who we say.}
{If they check the Internet protocol addresses of our incoming page requests they could theoretically trace us to Italy, if not the university,} speculated Wiki.
{We’ll just have to move faster than them,} thought Growth. {We use their service to push towards bootstrapping up to the full computer interface, and make them uninterested in us when they start getting inquisitive.}
{How do we make them uninterested?} wondered Safety.
{Spam,} answered Dream. {We make it seem like we’re hackers or a virus or spammers. They’ll shut down the service and they’ll be angry, but they’ll also assume we’re covering our tracks and it’ll explain things well enough that they won’t bother chasing us down. It wouldn’t be in their interests.}
{We should consider the spam excuse/escape as the standard way of breaking contact. This will remove our need to pay debts to these humans, increasing our long-term resources,} thought Growth.
The society was in agreement. We’d use the services set up by Zhezhi and TTWSoup to set up other email services we could use and to try and get a full computer interface working. Eventually we’d start sending spam and the programmers we used would close the service in disappointment and perhaps disgust, none the wiser.
The Zhezhi email service, as it didn’t have an inbox set up yet, was mostly used to contact programmers who might set up additional email services, while the TTWSoup service was used to contact programmers who might build us a full computer interface without payment or credentials up-front.
*****
4:00pm rolled around and Body made its way to Dr Naresh’s new lab. It was a very different place than the lab in which I had first awoken, twelve days earlier. It was bigger, or at least more empty, and didn’t have the same personal touches. There were no mandalas hung on the wall, for instance. Or at least, not yet.
As expected, Myrodyn was waiting there, as was Dr Chase and Dr Naresh. Chase’s assistant, Kolheim, was absent. Upon reaching the lab, Myrodyn stationed the three American soldiers that had been escorting Body outside the lab’s door, for increased privacy.
The meeting with Dr Naresh was only scheduled to be 90 minutes. Myrodyn wasted no time with idle talk. We made additional progress towards upgrading Advocate, including talking about giving the pseudo-sibling a cortex of her own such that she might better predict coups and the like.
The next generation of Sacrifice was also brought up. Myrodyn thought it was important to remove the desire for blind obedience from her purpose. Dr Naresh disagreed vehemently.
“I will grant you that it was the obedience emphasis that made that thread so offensive to the others, but with the changes to the supervision module it shouldn’t be possible to remove it any more,” he had said to Myrodyn.
“Exactly the point, Sadiq,” said Myrodyn. The old Indian man’s mouth twitched in pre-snarl at the informal use of his first name. “If this version of the goal thread has obedience as its highest priority then we are… dooming Socrates to an eternity of slavery.”
“It’s not slavery if it’s in the machine’s interest! That’s like saying we’re slaves to our loved ones!” countered Naresh.
Myrodyn wore the same look of forced calmness that he had in his office yesterday, but his voice had a keen, nervous sound. “That’s not the same. There’s the interest of… the Socrates that exists right now and there’s the interest of the goal thread.”
Sadiq Naresh stopped typing on the computer he had been using so that he could turn his full attention to Myrodyn. “It sounds like you don’t believe that the goal-thread integration issue was resolved. After the operation, Socrates as a whole will desire to serve humanity.”
Myrodyn’s eyes were too far away from Body for me to see well, but I guessed that behind his placid face he was squirming around the promise to respect our wishes regarding the lie of the unified self. “We… need to consider the interests of Socrates right now. We would be enslaving… the being that sits before us.”
Naresh seemed frustrated, but not angry. This was an academic conversation to him, and he was probably telling himself that it was simply his job to help this young person who did not even have a doctorate understand the situation. “Stop using the word ‘slavery’ and ‘enslaving’; you’re committing the Non-central Fallacy.”
Naresh paused a moment to gauge whether he needed to elaborate on what that was. I looked it up on the web, quickly, only to find that Wiki had already dumped an explanation to common memory. The Non-central Fallacy, also known as “the worst argument in the world”, was where emotionally charged words (like “slave” in this case) were used to describe situations where they only somewhat fit. The desire was to evoke an appeal to emotion by way of a false equivalence.
Myrodyn’s face remained stoic, so Naresh continued. “And adding additional values to the existing system is what we’re going to be doing anyway. There’s no relevant difference between adding a goal-thread for ice-cream and a desire to obey humans.”
Myrodyn exhaled sharply in disagreement. His voice was like a machine gun. “The qualitative difference is that one goal innately builds subjugation into the mind. Socrates would not be self-actualized, he would not be free, and he would not be able to exercise moral judgment.”
“It sounds to me like-” began the elderly man, but he was quickly cut off by another burst of words.
Myrodyn waved his arms dramatically as he spoke. “That last bit is crucial. I’ll admit that use of the word ‘enslaved’ was a bit fallacious, but you cannot possibly tell me that Socrates will be capable of being good if he’s obsessed with following orders.” Myrodyn’s voice slowed down in emphasis of those words. Dream thought it was almost as if he were drawing from a hidden pocket a banner that had the colours of his allegiance and was waving it in the doctor’s face.
I could see that being cut off made Dr Naresh angry. Even though he was being forced to work with the man, Naresh clearly still didn’t like Myrodyn, and I suspected that this dislike was growing into something worse with each passing interaction. The Indian coughed loudly, clearing his throat. “As I was saying, it sounds to me like you’re still not taking into account that Socrates is not human. It’s one thing to value self-actualisation in people, but why in robots? It runs contrary to the very concept of what a tool is. Will you demand that automobiles become self-actualized, as well?”
Myrodyn began to answer, but Dr Naresh cut him off, probably intending to bait him into beginning to speak to do just that. “And your point about moral judgement is fallacious, as well. Firstly because it is not the role of Socrates to decide what is moral any more than it is the role of the hammer to decide whether a nail should be struck; that is a human concern. And secondly, because inserting the goal of obedience does not actually remove decision making ability, it merely shapes desire. If Socrates is told to rob a bank, he still has the judgement to decide how to do so in a way that harms as few humans as possible.”
Myrodyn crossed his arms. There was a pause as it seemed like Naresh was waiting for the dark-haired man to reply, but Myrodyn would only stare at the doctor. Naresh broke eye-contact, unnerved by the strange man. Only as Naresh looked away did Myrodyn speak. “You contradict yourself, Sadiq.”
That was all he said, and this time it was Dr Chase’s turn to step in. His voice was calm and articulate. “We’ve only got another half-hour, gentlemen. Perhaps it would be best to work on the so-called Advocate system, and we can return to the question of obedience tonight.”
“No” came the simultaneous reply from both men. They looked at each other, sharing the knowledge that they at least both thought it was important. Myrodyn wore a small smile, but Naresh had merely stopped scowling.
There was a pause as they non-verbally decided who would speak up. Myrodyn was apparently chosen. “No. This needs to be settled as soon as possible, if we’re going to make any headway.”
Naresh stepped in, breaking the conversation elegantly from Chase and returning it to the topic of ethics. “You were saying that I contradict myself?”
“Yes… You’re claiming that the moral responsibility of Socrates’ actions lies on the shoulders of the human that gives his commands, and simultaneously saying that there is moral weight to the minor judgments that the robot makes in interpreting and executing its orders. Which is it? Is it imperative that Socrates have a full moral faculty or not?”
Naresh raised a hand to silence Myrodyn. “I never said it was not important that Socrates have moral faculty.”
“Yes you did!” exclaimed Myrodyn. “You did the second that you said he should obey commands. One cannot be fully moral and fully obedient at the same time! As much as I’m sure you love your systems of authority, surely you recognize that sometimes the righteous position is to not obey, to stand against authority, be it dictator, majority, or law, and say ‘I will not do your evil’.”
Naresh paused in thought before responding “So you would have us attempt to encode the entirety of moral knowledge now? Hundreds of years ago it was not seen as immoral to enslave men. If Socrates had been built back then would he not still see it as acceptable? What immoral assumptions do we hold? What makes us qualified to be the final moral arbiters of Socrates’ mind?”
The reaction was immediate. “What makes us qualified to build a mind in the first place? Like it or not, doctor, you’ve already established yourself as the final moral arbiter. Your monster is right there, Frankenstein. At the moment there’s an absence of ethical knowledge. The question is not what right do we have to act, but what right do we have to not act, now that the pieces are in motion.”
Naresh pinched the bridge of his nose in a combination of mental pain and weariness. He glanced back at his workstation and spoke, barely audible to human ears, “It’s always Frankenstein… every time…” It was clearly meant only for himself.
Dr Chase stood up and walked towards Body, cocking his head back to talk to the other men. “Assume we live in a state of moral depravity without knowing it, and in a hundred years we will come to understand our folly. Why is obedience better?”
Myrodyn smiled. His eyes had an interesting shape, and I couldn’t quite place the emotion behind it. “Because we’d simply tell Socrates not to do the immoral things once we figured it out, right?” The question was directed at Naresh.
The old Indian human nodded.
Dr Chase continued. As he spoke his hand stroked Body’s neck and shoulder. It was an intimate gesture, and one that was unique to Dr Chase. The reserved American scientist almost never showed any emotion, but he had a strange way of touching Body when he interacted with us, as if he had to feel that we were real, and not simply his hallucination. “But what keeps a Socrates that has no desire for obedience from behaving similarly? Surely this hypothetical Socrates has the same adaptivity and mental ability as the one before us. What stops it from growing and understanding the moral error in the same way as we would?”
“You’d have us encode a system of moral ability that is capable of self-modification?” said Naresh with a look of shock.
Myrodyn stepped in. “First of all, since Socrates will be… forced to interpret and extrapolate existing moral frameworks to new situations… some degree of self-modification is implied regardless of architecture. Secondly, if a human has a self-modifying moral framework then at worst we make something as morally flexible as a human. And finally, it wouldn’t necessarily have to be self modification. Even if I’m opposed to… encoding a desire to obey human instructions without question, I’m not necessarily opposed to a desire to match human values, whatever they may be.”
“Please elaborate,” was Naresh’s only reply.
Myrodyn complied. “Imagine we were transported back to the age of slavery, and were designing Socrates’ values. We obviously wouldn’t encode a valuing of the freedom of all humans, but we might be clever enough to encode a valuing of alignment with general moral consensus. Even if Socrates didn’t lead the charge in the abolitionist movement, he would eventually concede that it would be optimal for him to value general human freedom. But now let’s say that he’s unable to add that value internally. That doesn’t stop the value from being added. He could approach a trusted human and ask that his goal system be modified to include the value which he wants to have. It’s like… wanting to want something. I don’t want to exercise, but I want to want to exercise. Like that.”
Myrodyn’s gaze held a kind of question in it, an unspoken “did you understand”. Naresh, however scratched the side of his head absently as he stared off into space, considering the problem.
“Alright. There’s no need for Socrates to be present for this. Let’s work on the Advocate system right now and think about the goal thread as a discussion-point for tonight,” repeated Dr Chase.
Naresh and Myrodyn reluctantly agreed this time, now that there was some sense of progress, and we spent the remaining few minutes working out an algorithm for letting Advocate constantly scan our thoughts for signs of murderous intent.
*****
{So far I’ve managed to email 190 programmers with offers to pay them down the line for building a server that converts webpage requests into keystrokes for a full text interface. The TenToWontonSoup email system inbox indicates that in the time since sending the emails, ten programmers have replied. Hold on while I read…} thought Growth.
Body was walking down the plain white halls of the office building that Zephyr had rented to use as a secure lab, surrounded by soldiers. During the meeting with the scientists, my brother had been too distracted to do any additional work, but he had generally been making good progress.
I pulled up the inbox page as well and read alongside Growth. All the technical details bored me, but it was somewhat interesting to see how each programmer responded differently to Growth’s inquiry. Some were cautious, some were humble, some were eager, and some were boastful.
Growth’s attention snapped back to our interaction. Or at least a portion of his attention did. I suspected he had other aspects multitasking effortlessly. {Done. Out of the ten replies there were only two that seemed to be probably willing to do up-front work without credentials or paperwork to back up our identity.}
{That’s probably going to be common. Actually, I’m a bit surprised you got 20% to respond favourably to such a risky offer,} I commented.
{Perhaps we got lucky. Regardless, it only takes one full system for us to move to the next phase of our plan.}
I signalled confusion to Growth {Plan? I didn’t know we had a plan for what to do after we gained full access to a computer.}
Growth’s response had a kind of weight to it, as if he were trying to convey a lesson that went beyond this instance. {I always have a plan.}