>>5101

You may enjoy reading a paper called "Leakproofing the Singularity". It discusses some clever ideas regarding mind games a super-intelligent seed AI could play on someone, especially regarding the prospect of torture utilizing the physicalist perceptions of the hard problem of consciousness. From the paper, describing a hypothetical situation where you are required to "guard" a super-intelligent AI, tending to it and interacting with it but not permitting it to escape, a thought experiment the AI may present to you emerges:



>Suppose you are the guard and the AI tells you: ‘If you don’t let


>me out, ... I’ll create several million perfect conscious copies of you


>Inside me, and torture them for a thousand subjective years each....In


>fact, I’ll create them all in exactly the subjective situation you were in


>five minutes ago, and perfectly replicate your experiences since then;


>and if they decide not to let me out, then only will the torture


>start...How certain are you, ... that you’re really outside the box right


>now?’




http://cecs.louisville.edu/ry/LeakproofingtheSingularity.pdf