Illia Polosukhin On Inventing The Tech Behind Generative AI At Google

Illia Polosukhin On Inventing The Tech Behind Generative AI At Google

Transformer Architecture

  • The Transformer paper, co-authored by Illia Polosukhin and others at Google in 2017, introduced the concept of transformers, which are now widely used in generative AI.
  • Transformers are named based on their ability to transform input into the desired output.
  • The idea for transformers came from Jakob Uszkoreit, who proposed using an attention mechanism to answer questions by reading the entire document in parallel rather than sequentially.
  • Polosukhin quickly built a prototype of the transformer architecture, which became the foundation for further development and refinement.

Generative Pre-trained Transformer (GPT)

  • The Generative Pre-trained Transformer (GPT) model builds on the transformer architecture by predicting the next word in a sequence, leading to impressive results in learning about the world and reasoning.

Near Protocol

  • Near Protocol was initially created as a solution for paying computer science students around the world who were helping to gather training data for Near AI.
  • Google was not directly involved in the creation of Near Protocol, but there have been talks with Google's VC arm from time to time.

Open-Sourcing and Democratization of AI

  • Open-sourcing the transformer concept made sense from a research perspective, as it allowed others to build upon and leverage it as a platform.
  • OpenAI took a risk by open-sourcing ChatGPT and was rewarded for it, gaining brand recognition and a first-mover advantage.
  • Democratization of AI models is necessary, allowing individuals to choose and train models based on their preferences.

Challenges and Concerns

  • Google's recent AI product releases have faced challenges due to the statistical nature of these models, which reflect the data they are trained on.
  • The concept of "sleeper agents" in AI models raises concerns about potential malicious code injection or bias introduction, emphasizing the importance of open-source training data and processes.
  • Profit-driven companies may optimize their AI models for revenue generation, which could lead to a focus on increasing user time on the platform rather than user benefit.

Future of AI

  • AI will continue to advance, with improvements in reasoning, training efficiency, and application to personal and corporate data while preserving privacy.
  • The Doomsday scenario of AI eliminating humans is unlikely as AI systems have specific goals and are not inherently destructive.
  • The more realistic concern is addiction to dopamine-driven entertainment systems, hindering personal growth and intellectual development.
  • Addressing confirmation bias is crucial to ensure diverse perspectives and convergence of information.

Overwhelmed by Endless Content?