>>5101
You may enjoy reading a paper called "Leakproofing the Singularity". It discusses some clever ideas regarding mind games a super-intelligent seed AI could play on someone, especially regarding the prospect of torture utilizing the physicalist perceptions of the hard problem of consciousness. From the paper, describing a hypothetical situation where you are required to "guard" a super-intelligent AI, tending to it and interacting with it but not permitting it to escape, a thought experiment the AI may present to you emerges:

>Suppose you are the guard and the AI tells you: ‘If you don’t let

>me out, ... I’ll create several million perfect conscious copies of you

>Inside me, and torture them for a thousand subjective years each....In

>fact, I’ll create them all in exactly the subjective situation you were in

>five minutes ago, and perfectly replicate your experiences since then;

>and if they decide not to let me out, then only will the torture

>start...How certain are you, ... that you’re really outside the box right

>now?’


http://cecs.louisville.edu/ry/LeakproofingtheSingularity.pdf