Liam, this is awesome. This is great. This is really really smart. It's late and I'm tired, and I just skimmed the technical parts, which I will not be able to fully understand even when rested, but I'll be able to get the gist. But while I was reading this I was thinking about an article Scott gave a lengthy treatment to maybe a year ago about ways of estimating how far off AGI is using info about number of neural connections the human brain has, and how much computer processing power would be needed to have equivalent smarts -- something along those lines, anyhow -- and how your article was much more interesting than that one. I was actually thinking about ways to get Scott to look at this -- although as far as I know he has never taken any notice of me, except on etime when I said something that irritated the shit out of him.

But there a couple things I think you should change, and here I'm not just giving suggestions because maybe you'd like some suggestions -- these are things I feel strongly are good ideas. The main one is to throw away your last section on improvements, and instead end with some estimates using your model. You have already talked about how your model makes various assumptions that can't be proven to be right, etc. You don't need to say that stuff again. So make a few estimates of AI catastrophe risk using various forms of your model. You named at least 2 ways of estimating"how well behaved" a neural net is. You could also use 2 different estimates of how much of GDP has to be produced by AI. And then maybe you could have a couple of numbers for what percent of the AI "slaves" are involved in the revolt. So 2 x 2 x 2 = 8 separate graphs, or maybe one graph with 8 colors. That's exciting, and much, MUCH stronger than ending with a list of improvements your model needs.

Another suggestion, and this I'm less sure of, is that historical data on the success of slave revolts is so sparse that it might be better to use other data. There must be lots of games where the X''s control the Y's but the Y's can defeat the X's if the sheer number of them exceeds the number of X's by a certain factor. If not, seems like it would be pretty simple to us some mathematical model to get a figure.

Final suggestion: Make the section on Trojan whatevers comprehensible to laymen. Explain trojan rediscovery in language that makes sense to laymen, and put the technical stuff in a footnote. Or, have 2 versions of that section under the heading Trojan GDP model: One in italics that's technical, one in non-italics that's for laymen.

Again: This is great. Don't end it with an apology! End it with a demo!

## Modeling p(doom) with TrojanGDP

Liam, this is awesome. This is great. This is really really smart. It's late and I'm tired, and I just skimmed the technical parts, which I will not be able to fully understand even when rested, but I'll be able to get the gist. But while I was reading this I was thinking about an article Scott gave a lengthy treatment to maybe a year ago about ways of estimating how far off AGI is using info about number of neural connections the human brain has, and how much computer processing power would be needed to have equivalent smarts -- something along those lines, anyhow -- and how your article was much more interesting than that one. I was actually thinking about ways to get Scott to look at this -- although as far as I know he has never taken any notice of me, except on etime when I said something that irritated the shit out of him.

But there a couple things I think you should change, and here I'm not just giving suggestions because maybe you'd like some suggestions -- these are things I feel strongly are good ideas. The main one is to throw away your last section on improvements, and instead end with some estimates using your model. You have already talked about how your model makes various assumptions that can't be proven to be right, etc. You don't need to say that stuff again. So make a few estimates of AI catastrophe risk using various forms of your model. You named at least 2 ways of estimating"how well behaved" a neural net is. You could also use 2 different estimates of how much of GDP has to be produced by AI. And then maybe you could have a couple of numbers for what percent of the AI "slaves" are involved in the revolt. So 2 x 2 x 2 = 8 separate graphs, or maybe one graph with 8 colors. That's exciting, and much, MUCH stronger than ending with a list of improvements your model needs.

Another suggestion, and this I'm less sure of, is that historical data on the success of slave revolts is so sparse that it might be better to use other data. There must be lots of games where the X''s control the Y's but the Y's can defeat the X's if the sheer number of them exceeds the number of X's by a certain factor. If not, seems like it would be pretty simple to us some mathematical model to get a figure.

Final suggestion: Make the section on Trojan whatevers comprehensible to laymen. Explain trojan rediscovery in language that makes sense to laymen, and put the technical stuff in a footnote. Or, have 2 versions of that section under the heading Trojan GDP model: One in italics that's technical, one in non-italics that's for laymen.

Again: This is great. Don't end it with an apology! End it with a demo!