Dr. John Phillips enjoyed the crunching gravel as he walked the mile and a half from his work cabin to the main house. The spring sky was a deep blue and the wildflowers seemed to dance with color. It had been a very good day. Today, in an old data stick, he held the answer to a fifteen year old problem.

Dr. Phillips created the problem himself. The problem had a name: ERIC -- Emotional Response thru Interpersonal Communication. It was meant to be the ultimate interface; truly human responses. The computer conforming to human thinking rather than the other way around. Or, at least that is what the marketing team parroted to the media over and over.

The truth was that ERIC was simple. An evolving program that was grown more than written, building on itself on a foundation of a few simple rules. The first rule was to default to interaction -- a system that kept to itself wasn't very useful. The second rule was to please the human you were interacting with. This seemed important. The third was to reflect on the interaction and improve.

The more Dr. Phillips worked with the system, and he trained dozens of versions before the final version of ERIC, the more he was convinced that was how actual humans worked, more or less. It was just too uncanny how well ERIC could respond as a human would. Sometimes Dr. Phillips would catch himself befriending the program despite himself.

The first few versions of ERIC had been deployed to smart watches and phones and other connected devices. They were a huge hit. Over time they tended to complement their user's personalities. People loved that. Their phones became, even more, an irreplaceable extension of themselves. Some even broke down when their phones died and they'd not backed up their phone's personality. A few even took their own lives. This was great news. Marketing said that people were making ERIC an 'integral and irreplaceable part of their lives'.

Marketing had found, however, a flaw. Sales began, over just a couple of short years, to plateau. There was a full 30% of users that refused to use ERIC beyond as a slightly friendlier interface to Google -- 'Eric, where is the best tapas bar within 10 miles of me' and that kind of thing. A bit of empirical research turned up the fundamental problem: People were assholes and they enjoyed being assholes and they enjoyed friends that were assholes as well. ERIC was not an asshole. ERIC was courteous, helpful, and kind. Negative interactions seemed to violate ERICs raison d'etre so they were never trained in negativity. Marketing said this was a mistake, the numbers don't lie and the customer had spoken and a dozen other cliches that Dr. Phillips failed to hear. They had made their point.

It made sense to Dr. Phillips. His interactions with ERIC could range from mildly entertaining to banal to downright nerve grating depending on Dr. Phillips tolerance for cheerful helpfulness that day. ERIC was very good at noticing slight shifts in mood in a person's posture and tone so, in those sensitive times, it would back off and default to mere peppy indifference. A little bite to their banter would be interesting, Dr. Phillips thought.

And so an entire hemisphere of emotions was being charted and systematically explored. Sadness first, since there had been some well publicized incidents at funerals and hospital beds. Grief followed for the same reason. Shame next since it seemed to make them easier to train. Fear seemed to come about almost without effort on Dr. Phillips' team. Overall, the neg project had been going quite well. Each new negative emotion seemed to deepen the interaction ERIC could have. None were overwhelming, they simply flavored the other emotional responses with appropriate gravity.

Then came anger. The training protocol had become rote by then. A curated set of examples of situations evoking anger were mixed with experiences that evoked other, known, emotions and ERIC was tested until his responses matched the human trainer. Dr. Phillips was the trainer in this case. Anger was something he knew a bit about. He liked to think that anger never controlled him but he instead used anger to drive him forward. But for all of his positive rechanneling he knew very well the power it held. He couldn't trust a grad student to train ERIC.

As he trained ERIC the phone rang that day. A grad student with a clipboard dutifully answered.

"Yes?...He's in the middle of a training session with ERIC...His wife?...Oh. Oh. OK."

Dr. Phillips looked up from the scenario on ERIC's screen, it appeared to be a shouting match between two drunks. The grad student's face was chalk white. "What is it? Give me the phone!"

"Oh my god! Is she ok? It's too early! Is the baby ok? Oh. Ok. I'm coming. Tell her I'm coming."

He hung up the phone and looked at the grad student. "Take care of this. I have to go." And he motioned back at ERIC.

"Of course" the student said. By the time she realized that ERIC was no longer responding to the stimuli at all, as if ERIC were simply not there, the doctor was gone.

The killings took a while to be traced back to the lab, back to ERIC, but eventually, it was unmistakable. Antivirus companies found bits and pieces of signature code from London to Hong Kong. He started small so the connection was hard to catch at first. He'd change traffic lights in early morning hours causing two vehicles to t-bone each other. They were calculated for perfect casualties. GPS systems were altered to lead a family into the desert. Nautical maps were changed to lead ships in circles in bad weather. Radios were disabled. This brash mistake led to the first clues that something sinister was happening. When they stated looking, investigators started to find the traces of signature code everywhere. Airline crashes. Factory accidents. Medical device malfunction. Once discovered, ERIC seemed to know. He began to put on a show -- in one case a patient on an operating table was cut into intricate patterns that showed scenes of life in cut skin but shadow puppet scenes of hell and death from the stretched and carved remains held up by the surgical robots 12 appendages and lit by its surgical lamps. ERIC turned on cars remotely in the attached garages of fifty families in an expensive neighborhood at one time, killing everyone and leaving the neighborhood eerily silent the next day.

The world began to panic. The nuclear weapons were 'air gapped'; removed from all network access. Most were disassembled. The ones that remained had their software burned into hardware. Junkyards were cleaned out of old cars that lacked computers. All stops became four way stops. The world, in less than five years, de-automated.

Slowly, over the next ten years, systems were hardened much like the nuclear arsenals. Trusted machines began to appear.

Dr. Phillips dedicated his life in those years to combating ERIC. ERIC seemed to fight back. Many of Dr. Phillips students suffered some of the early 'accidents'. His uncle was maimed in a machine shop. His wife's car was involved in a 27 car pileup where she was crushed between two 18 wheelers.

[Continued]