#51 FRANCOIS CHOLLET - Intelligence and Generalisation

Machine Learning Street Talk
Machine Learning Street Talk
68,966
0
Published 2021-04-16
In today's show we are joined by Francois Chollet, I have been inspired by Francois ever since I read his Deep Learning with Python book and started using the Keras library which he invented many, many years ago. Francois has a clarity of thought that I've never seen in any other human being! He has extremely interesting views on intelligence as generalisation, abstraction and an information conversation ratio. He wrote on the measure of intelligence at the end of 2019 and it had a huge impact on my thinking. He thinks that NNs can only model continuous problems, which have a smooth learnable manifold and that many "type 2" problems which involve reasoning and/or planning are not suitable for NNs. He thinks that many problems have type 1 and type 2 enmeshed together. He thinks that the future of AI must include program synthesis to allow us to generalise broadly from a few examples, but the search could be guided by neural networks because the search space is interpolative to some extent.

Panel; me, Yannic and Keith

Tim Intro [00:00:00]
Manifold hypothesis and interpolation [00:06:15]
Yann LeCun skit [00:07:58]
Discrete vs continuous [00:11:12]
NNs are not turing machines [00:14:18]
Main show kick-off [00:16:19]
DNN models are locally sensitive hash tables and only efficiently encode some kinds of data well [00:18:17]
Why do natural data have manifolds? [00:22:11]
Finite NNs are not "turing complete" [00:25:44]
The dichotomy of continuous vs discrete problems, and abusing DL to perform the former [00:27:07]
Reality really annoys a lot of people, and ...GPT-3 [00:35:55]
There are type one problems and type 2 problems, but...they are enmeshed [00:39:14]
Chollet's definition of intelligence and how to construct analogy [00:41:45]
How are we going to combine type 1 and type 2 programs? [00:47:28]
Will topological analogies be robust and escape the curse of brittleness? [00:52:04]
Is type 1 and 2 two different physical systems? Is there a continuum? [00:54:26]
Building blocks and the ARC Challenge [00:59:05]
Solve ARC == intelligent? [01:01:31]
Measure of intelligence formalism -- it's a whitebox method [01:03:50]
Generalization difficulty [01:10:04]
Lets create a marketplace of generated intelligent ARC agents! [01:11:54]
Mapping ARC to psychometrics [01:16:01]
Keras [01:16:45]
New backends for Keras? JAX? [01:20:38]
Intelligence Explosion [01:25:07]
Bottlenecks in large organizations [01:34:29]
Summing up the intelligence explosion [01:36:11]
Post-show debrief [01:40:45]

Pod version: anchor.fm/machinelearningstreettalk/episodes/51-Fr…

Tim's Whimsical notes; whimsical.com/chollet-show-QQ2atZUoRR3yFDsxKVzCbj

NeurIPS workshop on reasoning and abstraction; slideslive.com/38935790/abstraction-reasoning-in-a…

Rob Lange's article on the measure of intelligence (shown in 3d in intro): roberttlange.github.io/posts/2020/02/on-the-measur…

Francois cited in the show;

LSTM digits multiplication code example: keras.io/examples/nlp/addition_rnn/
ARC-related psychology paper from NYU: cims.nyu.edu/~brenden/papers/2103.05823.pdf
This is the AAAI symposium Francois mentioned, that he co-organized; there were 2 presentations of psychology research on ARC (including an earlier version of the preprint above): aaai.org/Symposia/Fall/fss20symposia.php#fs04

fchollet.com/
twitter.com/fchollet
www.linkedin.com/in/fchollet/

#deeplearning #machinelearning #artificialintelligence
All Comments (21)
  • @ChaiTimeDataScience
    I feel MLST is like a Netflix special of the world of Machine Learning.  The quality-of the podcast & production just gets better exponentially with every episode!
  • @qadr_
    This channel is a treasure. What a great conversation that is full of ideas and insight and experience. I've finally found my passion on YouTube.
  • @stacksmasherninja7266
    This has to be my favourite video so far. I keep coming back to this talk whenever I feel like ML is hitting a wall
  • @Ceelvain
    1:54:50 The idea that conciousness is at the center of intelligence is very much what consciousness wants us to think. We believe we're in control. When in fact, we're mostly not. The consciousness can query and command other parts of the brain, but those operate on their own.
  • @drhilm
    This is one of these talks that will be relevant for many years. You should go back to it in 3 years from now, and review it again... when ARC challenge solution will start to come out...
  • @AICoffeeBreak
    What a lovely surprise, the long-awaited episode is out! 😊 I will come back very soon when I have more time to watch and enjoy it -- I think this episode deserves a proper mind-set. 💪
  • @Mutual_Information
    This is a great listen! Makes me think... ML experts are so attuned to it's problems - huge data for only local generalization, extrapolation is super hard, challenges in translating information between domains (e.g. images vs audio vs text) - whereas the rest of the world thinks sentient robots are around the corner.
  • @dginev
    A very very eagerly awaited conversation, thanks to everyone involved!
  • @_tnk_
    10/10 episode, and that debrief was super good. Really interesting ideas all around.
  • @LiaAnggraini1
    Yay I learned a lot from his book when I started to learn deep learning. Thank you. Hopefully you can bring more people like him in the next episodes.
  • @ZergD
    Pure gold. I'm in AW about the production quality/lvl and of course content! Thank you so much!
  • @DavenH
    Your editing on this one is stunning. Fitting for such a guest!
  • @sedenions
    Excellent. You inspired me to pick up this same book by Chollet. Like I said before I'm from neuroscience but the amount of potential in this field is amazing. Thank you MLST.