Do these agents evolve to communicate with each other (e.g. by wiggling)?

With OpenAI5, it would have been cool to restrict inter-agent communication to an audio channel. Would a discrete language evolve?

'language' as we know is quite ambiguous and an inefficient way of communication. my guess is that what they will probably evolve is some kind of an audio encoding format for the exact information they want to convey to the other entity, like literally sending it control inputs over audio.

What is the difference between what you are describing and a language?

Human language has a second or more of latency.

Shouting "he's behind you!!", together with the other person hearing it, understanding it, and doing something about is is a very slow process.

AI would probably come up with signals which have less latency.

Given that the AIs were playing with humans, perhaps using speech to text/text to speech as an interface in order to be "human compatible" would overcome some of that.

That is a very cool question. Now I am interested to see if anything like that has been done.

Like bees?

This feels a lot like OpenAI 5, but with a few key differences - for one, it seems to handle randomly generated maps and other unexpected situations, and each agent does not have full knowledge of the others on its team. Really cool!

And it would be hilarious if this ends up giving us video game enemies that don't suck. (Until, of course, someone accidentally creates an RTS AI that manages to 'break out' and conquer the world or something)

evolving agents like those are very good at finding bugs to abuse. so, if they'd find a zero day to break out of the quake-sandbox, what would they do? i guess, for the longest time they'd just use the DMA into the sandbox to increase their score (i.e. cheating by developing and using external tools), as their fitness function still weights CTF wins only.

to (accidentally) change their fitness function to change it to "conquer the world" they'd need to modify a lot of different processes at once.

IMO the chance is negligible for now.

I think the (very sci-fy) fear there isn't "we gave this agent a goal of 'score xyz points' and it escaped the sandbox to increment the points counter" but instead "we gave this agent a goal of 'conquer the world' and made it think that the game it was in was the world, and then it discovered otherwise"

I wonder who from Google, Facebook and OpenAI will be the first to productize a "CEO as a service" AI.

I think it's an interesting question when this will start to show up - not in terms of managing everything including taxes but in terms of making automated business decisions.

In niches where business decisions are data driven, connecting a supply in some market to a demand in another, the choice of what to buy and sell could conceivably be automated.

Like gauging demand for mug/thermos/branded item du jour on amazon, checking whether similar items exist on alibaba and then setting up an amazon market place that resells your slightly rebranded but identical item from alibaba. Of course this all hinges on the data - an algorithm is prone to fall into traps that some humans fall into (i.e. the company you buy from only sells the box and not the product).

>> The agents are never told anything about the rules of the game, yet learn about fundamental game concepts and effectively develop an intuition for CTF.

Here we go again- throwing around big words, "intuition", and marring an otherwise interesting piece of work.

From a cursory glance at the relevant figure, "A look into how our agents represent the game world" - it looks like what is being learned is very much a goood old behaviour tree. For instance, the figure suggests that agents learned to react to situations like "Agent flag at base & opponent flag at base & not respawning & agent in home base". So basically, the condition part of an IF-THEN-ELSE rule.

Why is this called an "intuition" rather than a "rule"? From my reading, the only reason is that it was learned by a deep neural net by reinforcement learning without explicit supervision, i.e. without anyone telling the agent "learn this rule".

That's a very narrow, procrustean, definition of intuition. Is it really what most people would mean by "intuition"? Is it even close? Who knows- nobody can tell what most people would mean by "intuition". There's a dictionary definition, but chances are most people would not know it by heart. So it's very hard to even say "that's not what intuition means"- one would just be begging the question.

So why use such a vague term in an article like this, that is already pretty impressive stuff? What do such meaningless claims add to the result? I don't get it.

I'm not up to date on the relevant research, but isn't most learned or innate intuition just a set of fuzzy rules based on observation (be it visual or other sensory perception; or pre-defined behaviour in the case of innate intuition)?

If I play tennis for a while, I will get better at predicting where the ball will go after my opponent hits it merely because I have seen many combinations of shots and subsequent ball trajectories.

I think it would be weird to not call it an intuition just because it was learned by a deep neural net in this case.

In my experience, in the context of artificial and human intelligence, any statement of the form "X is just Y" or, equivalently, "Isn't X just Y" is neither sound nor complete and X and Y are not well defined.

In your example, for instance- I have no idea what are "fuzzy rules" and how they apply to tennis playing, or how they relate to predicting the position of the ball etc.

So in summary you are saying they used "intuition" wrongly, while subsequently saying that it's really hard to say what intuition is objectively, and when someone pointed out that their understanding of intuition does at least partially match what is insinuated in the paper you continue saying "yeah but when talking about AI all of this is really complicated, please be more concrete"

I really have no idea what this is adding to the discussion, except for talking down to the authors which, in your opinion, have misused a word that you yourself find hard to define..

Edit: typo

What I said was that it's pointless to use terms like "intuition" or "fuzzy rules" that are not well defined.

That's because then anyone can claim anything they like in relation with those terms, and nobody learns anything form the discussion, which is a complete waste of time.

Let's try, then.

My understanding of "intuition" is having some subjective feeling of knowledge about aspects of a situation or being able to predict what is likely to happen in the near future automatically, i.e. without consciously trying to work it out. At first glance this seems to match with common dictionary definitions of what intuition is.

What I meant by "fuzzy rules" as applied to predicting where the ball will go after my tennis opponent hit it was that I do not measure exact speeds, angles, etc. and then deliberately calculate an expected trajectory, but I see roughly where the ball is coming from, how fast and at what angle my opponent is hitting it and where it starts to go. From this, after enough practice, I have a "feeling", i.e. an intuition, about where it will roughly end up hitting the court on my side again. This merely serves as an example.

What I am trying to get at is that as far as my knowledge goes, a lot of what we humans call "intuition" in our own behaviour by any widely agreed upon definition is just the same as what a trained AI is doing - applying a heuristic learned over time to make a decision based on an input.

No, just because the trained AI can play a game well does not mean it will outdo humans at any task. But a lot of human behavior is not that exceptionally special either, I think.

Thanks for clarifying.

I don't know that "a trained AI" is "applying a heuristic", like you say. Usually, "a trained AI" refers to a machine learning model, which is an approximation of some function. A heuristic is not the same as an approximation. A heuristic is a shortcut, or a rule of thumb, that is often guaranteed to work under certain conditions but not others. An approximation is a function that is correct up to some margin of error with respect to some other, true function.

Heuristics are more common in hand-crafted systems, rather than machine learning systems. For example, the A* algorithm is a search algorithm that uses a heuristic to estimate the cost of a path to agoal.

In short, no, I don't think you're right to say that "trained AI" is applying a heuristic, etc.

As to whether what us humans do is applying heuristics- well, maybe. It's plausible, but how can we know for sure before we actually, well, know for sure? We don't yet understand human intelligence.

Edit: Actually, it's obvious that many things we learn to do are not heuristics. For example, learning a foreign language as an adult (with instruction). First you painstakingly learn the rules and then at some point you can use the language without explicitly thinking about the rules. Did you suddendly develop a heuristic to replace the rules you learned with such effort, or did something else happen?

>> No, just because the trained AI can play a game well does not mean it will outdo humans at any task. But a lot of human behavior is not that exceptionally special either, I think.

That's a strange thing to say. Yeah, what we do is pretty exceptional. Otherwise we'd have been able to reproduce it with our machines. And even if AI like AlphaZero or the Quake agents in the article can outperform humans- it doesn't mean that they're doing what humans are doing.

For example- a calculator is better than me at arithmetic. However, we know how a calculator calculates and it's not how I, or most other humans, calculate.

I’m having a similar “here we go again” feeling, only it’s the completely expected reaction trying to move the goalpost as to what “intelligence”, or, in this case, “intuition” are supposed to mean.

The fact is that certain behavior appears, from the outside, to be fascinatingly human-like. It doesn’t matter that it can be explained by peeking behind the curtain, because it’s almost certain that all behavior, including human, just comes down to “statistics”. Unless you’re a proponent of some sort of body/mind dualism, which would put you squarely outside the scientific mainstream since ca. 1865.

>> It doesn’t matter that it can be explained by peeking behind the curtain, because it’s almost certain that all behavior, including human, just comes down to “statistics”.

"Statistics"? You're saying that human minds calculate means and standard deviations and the like? I find that very hard to believe. Just think of the amount of time it takes very powerful computers that are much better than humans at calculations to do statistics.

I also don't see how the "behaviour is just statistics" is the polar opposite of the body/mind dualism. I would think the opposite of "behaviour is just statistics" is "behaviour is not just statistics". Which we can't really verify until we figure out how human behaviour comes about.

Basically, human intelligence is not going to be "just X" anything. We've tried all the "just X" we could think of and nothing really worked.

> "Statistics"? You're saying that human minds calculate means and standard deviations and the like?

Clearly the brain is capable of some very powerful, yet unconscious math, otherwise catching a ball out of mid-air or simply walking would be impossible. It might not be able to solve them analytically or with extreme precision, but definitely approximately and extremely fast.

Yes, I'm aware of the concept. I don't think it is clear at all. I agree that it might be possible to model some of our behaviours using different kinds of maths. But that doesn't mean that those processes are actually carried out by a computation of the maths that we can use to model them.

By analogy- I don't think any phycisist would say that "the universe is capable of maths". Maybe our physics uses maths to model physical processes. But that doesn't mean that there is something that actually computes those processes, like a computer would.

Of course, intelligence is probably not a force of nature, like gravity. Then again, we don't know what intelligence is.

And just to kick this a bit more- why would this great computing power be "subconscious"? Why would it be the case that my brain can do maths better than I can? Does that even make sense? Who am I if not my brain? Is it really very likely that my brain is born with the innate ability to compute Pearson's correlation coefficient but I had to be taught what 1 + 1 is from scratch? And what about all the organisms that are perfectly capable of complex behaviours (spiders weaving webs, bees dancing, ants swarming etc) but have tiny little neural clumps rather than fully formed brains? On what substrate do they run all the statistics?

Super interesting!