Latest papers: Difference between revisions

From Robowaifu Institute of Technology
Jump to navigation Jump to search
No edit summary
(→‎March 2023: Add "SemDeDup: Data-efficient learning at web-scale through semantic deduplication")
 
(43 intermediate revisions by the same user not shown)
Line 1: Line 1:
{{expand|This page needs papers! Papers for creating robowaifus!}}
{{Expand|This page needs papers! Probably should set up an automated system so I can just drop Twitter and Arxiv links.}}
This page serves to collect notable research papers within the past two years related to [[robotics]] and [[artificial intelligence]]. Feel free to add new papers to the list and discuss any papers on the [[Talk:Latest_papers|talk page]].
 
[[File:papercraft.jpg|thumb|right|In Robowaifu.tech papers hold onto you.]] This page serves to collect notable research papers related to [[robotics]] and [[artificial intelligence]], particularly ones that can be used by hobbyists with minimal resources towards creating robowaifus. Feel free to add new papers to the list and discuss any papers on the [[Talk:Latest_papers|talk page]]. Papers posted on [https://alogs.space/robowaifu/ /robowaifu/] will also eventually appear here.
 
=== Search sites ===
 
* [https://www.semanticscholar.org/ SemanticScholar] - AI-powered research tool
* [https://paperswithcode.com/ PapersWithCode]
* [https://scholar.google.com/ Google Scholar]
* [https://arxiv.org/ arXiv]
* [https://you.com/ YouChat] - hit and miss from hallucinating a lot but sometimes finds good ones
* [https://www.jmlr.org/ Journal of Machine Learning Research]
* [https://huggingface.co/papers HuggingFace Daily Papers]
 
=== Social media sources ===
 
* [https://twitter.com/_akhaliq @_akhaliq]
* [https://twitter.com/abacaj @abacaj] (small language models)
* [https://twitter.com/DrJimFan @DrJimFan] (multimodal generalist agents)
* [https://twitter.com/gordic_aleksa @gordic_aleksa]
* [https://twitter.com/hardmaru @hardmaru]
 
== List of papers ==
 
=== Unsorted ===
 
{{Note|Need to summarize these papers into tl;dr. An automated system for this would be great.}}
 
==== June 2023 ====
 
* [https://arxiv.org/abs/2306.07174 Augmenting Language Models with Long-Term Memory]
* [https://arxiv.org/abs/2306.07967 One-for-All: Generalized LoRA for Parameter-Efficient Fine-tuning]
* [https://arxiv.org/abs/2306.11644 Textbooks Are All You Need]
* [https://arxiv.org/abs/2306.12156 Fast Segment Anything]
* [https://arxiv.org/abs/2306.11987 Training Transformers with 4-bit Integers]
* [https://arxiv.org/abs/2306.11706 RoboCat: A Self-Improving Foundation Agent for Robotic Manipulation]
* [https://arxiv.org/abs/2306.09782 Full Parameter Fine-tuning for Large Language Models with Limited Resources]
* [https://arxiv.org/abs/2306.11987 Demystifying GPT Self-Repair for Code Generation]
* [https://arxiv.org/abs/2306.08568 WizardCoder: Empowering Code Large Language Models with Evol-Instruct]
* [https://arxiv.org/abs/2306.08205 Agile Catching with Whole-Body MPC and Blackbox Policy Learning]
* [https://arxiv.org/abs/2306.07967 One-for-All: Generalized LoRA for Parameter-Efficient Fine-tuning]
* [https://arxiv.org/abs/2306.07174 Augmenting Language Models with Long-Term Memory]
* [https://arxiv.org/abs/2306.05422 Tracking Everything Everywhere All at Once]
* [https://arxiv.org/abs/2306.04757 INSTRUCTEVAL: Towards Holistic Evaluation of Instruction-Tuned Large Language Models]
* [https://arxiv.org/abs/2306.04050 LLMZip: Lossless Text Compression using Large Language Models]
* [https://arxiv.org/abs/2306.03509 Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Bias]
* [https://arxiv.org/abs/2306.03872 Deductive Verification of Chain-of-Thought Reasoning]
* [https://arxiv.org/abs/2306.00238 Bytes Are All You Need: Transformers Operating Directly On File Bytes]
* [https://arxiv.org/abs/2306.14884 Learning to Modulate pre-trained Models in RL]
 
==== May 2023 ====
 
* [https://arxiv.org/abs/2305.02412 Plan, Eliminate, and Track -- Language Models are Good Teachers for Embodied Agents]
* [https://arxiv.org/abs/2305.02301 Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes]
* [https://arxiv.org/abs/2305.02297 Making the Most of What You Have: Adapting Pre-trained Visual Language Models in the Low-data Regime]
* [https://arxiv.org/abs/2305.01625 Unlimiformer: Long-Range Transformers with Unlimited Length Input]
* [https://arxiv.org/abs/2305.00833 Learning to Reason and Memorize with Self-Notes]
* [https://arxiv.org/abs/2303.07295 Meet in the Middle: A New Pre-training Paradigm]
* [https://arxiv.org/abs/2305.18290 Direct Preference Optimization: Your Language Model is Secretly a Reward Model]
* [https://arxiv.org/abs/2305.17126 Large Language Models as Tool Makers]
* [https://arxiv.org/abs/2305.07185 MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers]
* [https://arxiv.org/abs/2305.18248 Do Language Models Know When They're Hallucinating References?]
* [https://arxiv.org/abs/2305.14825 Large Language Models are In-Context Semantic Reasoners rather than Symbolic Reasoners]
* [https://arxiv.org/abs/2305.03053 ZipIt! Merging Models from Different Tasks without Training]
* [https://arxiv.org/abs/2305.16635 Impossible Distillation: from Low-Quality Model to High-Quality Dataset & Model for Summarization and Paraphrasing]
* [https://arxiv.org/abs/2305.09967 Variable Length Embeddings]
* [https://arxiv.org/abs/2305.07759 TinyStories: How Small Can Language Models Be and Still Speak Coherent English?]
 
==== April 2023 ====
 
* [https://arxiv.org/abs/2304.14108 DataComp: In search of the next generation of multimodal datasets]
* [https://arxiv.org/abs/2304.14402 LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructions]
* [https://arxiv.org/abs/2304.13653v1 Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement  Learning]
* [https://arxiv.org/abs/2304.13705 Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware]
* [https://arxiv.org/abs/2304.13013 Stable and low-precision training for large-scale vision-language models]
* [https://arxiv.org/abs/2304.12244 WizardLM: Empowering Large Language Models to Follow Complex Instructions]
* [https://arxiv.org/abs/2304.11490 Boosting Theory-of-Mind Performance in Large Language Models via Prompting]
* [https://arxiv.org/abs/2304.11267 Speed Is All You Need: On-Device Acceleration of Large Diffusion Models via GPU-Aware Optimizations]
* [https://arxiv.org/abs/2304.11062 Scaling Transformer to 1M tokens and beyond with RMT]
* [https://arxiv.org/abs/2304.10970 Can GPT-4 Perform Neural Architecture Search?]
* [https://arxiv.org/abs/2304.10592 MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models]
* [https://arxiv.org/abs/2304.08466v1 Synthetic Data from Diffusion Models Improves ImageNet Classification]
* [https://arxiv.org/abs/2304.08460 LongForm: Optimizing Instruction Tuning for Long Text Generation with Corpus Extraction]
* [https://arxiv.org/abs/2304.07327 OpenAssistant Conversations -- Democratizing Large Language Model Alignment]
* [https://arxiv.org/abs/2304.06939 Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved With Text]
* [https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tuned-llm Free Dolly: Introducing the World's First Truly Open Instruction-Tuned LLM]
 
==== March 2023 ====
 
* [https://arxiv.org/abs/2303.16199v1 LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention]
* [https://arxiv.org/abs/2303.12712v5 Sparks of Artificial General Intelligence: Early experiments with GPT-4]
* [https://arxiv.org/abs/2303.11366 Reflexion: Language Agents with Verbal Reinforcement Learning]
* [https://arxiv.org/abs/2303.09540 SemDeDup: Data-efficient learning at web-scale through semantic deduplication]
 
==== Februrary 2023 ====
 
* [https://arxiv.org/abs/2302.10866 Hyena Hierarchy: Towards Larger Convolutional Language Models]
* [https://openreview.net/forum?id=__czv_gqDQt EfficientTTS 2: Variational End-to-End Text-to-Speech Synthesis and Voice Conversion]
* [https://arxiv.org/abs/2302.13971 LLaMA: Open and Efficient Foundation Language Models]
* [https://arxiv.org/abs/2302.13939 SpikeGPT: Generative Pre-trained Language Model with Spiking Neural Networks]
* [https://arxiv.org/abs/2302.12353 Autonomous Restructuring of Asteroids into Rotating Space Stations]
* [https://arxiv.org/abs/2302.06868 SwitchPrompt: Learning Domain-Specific Gated Soft Prompts for Classification in Low-Resource Domains]
* [https://arxiv.org/abs/2302.06675v3 Symbolic Discovery of Optimization Algorithms]
* [https://arxiv.org/abs/2302.01925v1 Learning a Fourier Transform for Linear Relative Positional Encodings in Transformers]
* [https://arxiv.org/abs/2302.00923v4 Multimodal Chain-of-Thought Reasoning in Language Models]
 
==== January 2023 ====
 
* [https://arxiv.org/abs/2301.13196 Looped Transformers as Programmable Computers]
* [https://arxiv.org/abs/2301.12314 Progressive Prompts: Continual Learning for Language Models]
* [https://arxiv.org/abs/2301.04589 Memory Augmented Large Language Models are Computationally Universal]
* [https://arxiv.org/abs/2301.04246 Generative Language Models and Automated Influence Operations: Emerging Threats and Potential Mitigations]
* [https://arxiv.org/abs/2301.02111v1 Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers]
* [https://arxiv.org/abs/2301.00774v3 SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot]
* [https://arxiv.org/abs/2301.00303v1 Rethinking with Retrieval: Faithful Large Language Model Inference]
 
==== December 2022 ====
 
* [https://arxiv.org/abs/2212.10509 Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions]
* [https://arxiv.org/abs/2212.10560v1 Self-Instruct: Aligning Language Model with Self Generated Instructions]
* [https://arxiv.org/abs/2212.09689 Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor]
* [https://arxiv.org/abs/2212.08751v1 Point-E: A System for Generating 3D Point Clouds from Complex Prompts]
* [https://arxiv.org/abs/2212.08073 Constitutional AI: Harmlessness from AI Feedback]
* [https://arxiv.org/abs/2212.08051v1 Objaverse: A Universe of Annotated 3D Objects]
* [https://arxiv.org/abs/2212.03848v2 NeRFEditor: Differentiable Style Decomposition for Full 3D Scene Editing]
 
==== November 2022 ====
 
* [https://arxiv.org/abs/2211.11602 Improving Multimodal Interactive Agents with Reinforcement Learning from Human Feedback]
* [https://arxiv.org/abs/2211.10438 SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models]
* [https://arxiv.org/abs/2211.09119v2 Token Turing Machines]
* [https://arxiv.org/abs/2211.06127v1 English Contrastive Learning Can Learn Universal Cross-lingual Sentence Embeddings]
 
==== October 2022 ====
 
* [https://arxiv.org/abs/2210.12217 Entailer: Answering Questions with Faithful and Truthful Chains of Reasoning]
* [https://arxiv.org/abs/2210.11416 Scaling Instruction-Finetuned Language Models]
* [https://arxiv.org/abs/2210.11948v2 lo-fi: distributed fine-tuning without communication]
* [https://arxiv.org/abs/2210.11610v2 Large Language Models Can Self-Improve]
* [https://arxiv.org/abs/2210.07229 Mass-Editing Memory in a Transformer]
* [https://arxiv.org/abs/2210.06407v1 Interactive Language: Talking to Robots in Real Time]
* [https://arxiv.org/abs/2210.05836v1 CLIP also Understands Text: Prompting CLIP for Phrase Understanding]
 
==== September 2022 ====
 
* [https://arxiv.org/abs/2209.15189 Learning by Distilling Context]
* [https://arxiv.org/abs/2209.07662 Dynamic Generation of Interpretable Inference Rules in a Neuro-Symbolic Expert System]
* [https://arxiv.org/abs/2209.10655v3 Mega: Moving Average Equipped Gated Attention]
* [https://arxiv.org/abs/2209.00626v4 The alignment problem from a deep learning perspective]
 
==== August 2022 ====
 
* [https://arxiv.org/abs/2208.07339 LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale]
* [https://arxiv.org/abs/2208.01448v2 AlexaTM 20B: Few-Shot Learning Using a Large-Scale Multilingual Seq2Seq Model]
* [https://arxiv.org/abs/2208.00748v3 Efficient Long-Text Understanding with Short-Text Models]
 
==== July 2022 ====
 
* [https://arxiv.org/abs/2207.07061v2 Confident Adaptive Language Modeling]
* [https://arxiv.org/abs/2207.05608v1 Inner Monologue: Embodied Reasoning through Planning with Language Models]
* [https://www.researchgate.net/publication/343082781_Robust_and_efficient_forward_differential_and_inverse_kinematics_using_dual_quaternions Robust and efficient forward, differential, and inverse kinematics using dual quaternions]
 
==== June 2022 ====
 
* [https://arxiv.org/abs/2206.15472v3 On-Device Training Under 256KB Memory]
* [https://arxiv.org/abs/2206.11795v1 Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos]
* [https://arxiv.org/abs/2206.07568v2 Contrastive Learning as Goal-Conditioned Reinforcement Learning]
* [https://arxiv.org/abs/2206.04615v2 Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models]
* [https://arxiv.org/abs/2206.02690v3 A Survey on Sentence Embedding Models Performance for Patent Analysis]
* [https://arxiv.org/abs/2206.00843v2 DepthShrinker: A New Compression Paradigm Towards Boosting Real-Hardware Efficiency of Compact Neural Networks]
 
==== May 2022 ====
 
* [https://arxiv.org/abs/2205.14135v2 FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness]
* [https://arxiv.org/abs/2205.14100v5 GIT: A Generative Image-to-text Transformer for Vision and Language]
* [https://arxiv.org/abs/2205.12910 NaturalProver: Grounded Mathematical Proof Generation with Language Models]
* [https://arxiv.org/abs/2205.11916v4 Large Language Models are Zero-Shot Reasoners]
* [https://arxiv.org/abs/2205.06175 A Generalist Agent]
* [https://arxiv.org/abs/2205.05270v1 Relational Triple Extraction: One Step is Enough]
* [https://arxiv.org/abs/2205.05131v3 UL2: Unifying Language Learning Paradigms]
* [https://arxiv.org/abs/2205.05055v6 Data Distributional Properties Drive Emergent In-Context Learning in Transformers]
* [https://arxiv.org/abs/2205.01917 CoCa: Contrastive Captioners are Image-Text Foundation Models]
 
==== March 2022 ====
 
* [https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html In-context Learning and Induction Heads]
* [https://arxiv.org/abs/2203.13474 CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis]
* [https://arxiv.org/abs/2203.02155 Training language models to follow instructions with human feedback]
* [https://arxiv.org/abs/2203.16329v2 Parameter-efficient Model Adaptation for Vision Transformers]
* [https://arxiv.org/abs/2203.15556v1 Training Compute-Optimal Large Language Models]
* [https://arxiv.org/abs/2203.07852v3 Block-Recurrent Transformers]
* [https://arxiv.org/abs/2203.06904v2 Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Models]
* [https://arxiv.org/abs/2203.03466v2 Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer]
* [https://arxiv.org/abs/2203.02155v1 Training language models to follow instructions with human feedback]
 
==== February 2022 ====
 
* [https://arxiv.org/abs/2202.12205v2 Is Neuro-Symbolic AI Meeting its Promise in Natural Language Processing? A Structured Review]
* [https://arxiv.org/abs/2202.10447v2 Transformer Quality in Linear Time]
* [https://arxiv.org/abs/2202.06991v3 Transformer Memory as a Differentiable Search Index]
 
==== January 2022 ====
 
* [https://arxiv.org/abs/2201.12122v3 Can Wikipedia Help Offline Reinforcement Learning?]
* [https://arxiv.org/abs/2201.11903v6 Chain-of-Thought Prompting Elicits Reasoning in Large Language Models]
* [https://arxiv.org/abs/2201.06910v2 ZeroPrompt: Scaling Prompt-Based Pretraining to 1,000 Tasks Improves Zero-Shot Generalization]
 
==== Februrary 2022 ====
 
* [https://arxiv.org/abs/2202.05262 Locating and Editing Factual Associations in GPT]
 
==== December 2021 ====
 
* [https://transformer-circuits.pub/2021/framework/index.html A Mathematical Framework for Transformer Circuits]
* [https://arxiv.org/abs/2112.05682 Self-attention Does Not Need <math>O(n^2)</math> Memory]
* [https://arxiv.org/abs/2112.05253v2 MAGMA -- Multimodal Augmentation of Generative Models through Adapter-based Finetuning]
* [https://arxiv.org/abs/2112.04426v3 Improving language models by retrieving from trillions of tokens]
* [https://arxiv.org/abs/2112.03254v3 Human Parity on CommonsenseQA: Augmenting Self-Attention with External Attention]
* [https://arxiv.org/abs/2112.08654 Learning to Prompt for Continual Learning]
 
==== November 2021 ====
 
* [https://arxiv.org/abs/2111.05204v1 Reason first, then respond: Modular Generation for Knowledge-infused Dialogue]
* [https://arxiv.org/abs/2111.02114v1 LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs]
 
==== October 2021 ====
 
* [https://arxiv.org/abs/2110.08207v3 Multitask Prompted Training Enables Zero-Shot Task Generalization]
* [https://arxiv.org/abs/2110.07732 The Neural Data Router: Adaptive Control Flow in Transformers Improves Systematic Generalization]
* [https://arxiv.org/abs/2110.05651v2 Learning with Algorithmic Supervision via Continuous Relaxations]
* [https://arxiv.org/abs/2110.00296 Powerpropagation: A sparsity inducing weight reparameterisation]
* [https://arxiv.org/abs/2111.00210 Mastering Atari Games with Limited Data]
* [https://arxiv.org/abs/2110.04176 PHNNs: Lightweight Neural Networks via Parameterized Hypercomplex Convolutions]
 
==== September 2021 ====
 
* [https://arxiv.org/abs/2109.14076v3 RAFT: A Real-World Few-Shot Text Classification Benchmark]
* [https://arxiv.org/abs/2109.10686v2 Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers]
* [https://arxiv.org/abs/2109.09193v1 Towards Zero-Label Language Learning]
* [https://arxiv.org/abs/2109.08603 Is Curiosity All You Need? On the Utility of Emergent Behaviours from Curious Exploration]
* [https://arxiv.org/abs/2109.06668v6 Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain]
* [https://arxiv.org/abs/2109.01652v5 Finetuned Language Models Are Zero-Shot Learners]
* [https://arxiv.org/abs/2109.00157v2 A Survey of Exploration Methods in Reinforcement Learning]
 
==== August 2021 ====
 
* [https://arxiv.org/abs/2108.01077v3 Generating Master Faces for Dictionary Attacks with a Network-Assisted Latent Space Evolution]
 
==== June 2021 ====
 
* [https://arxiv.org/abs/2106.15515v4 What Is Consciousness? Artificial Intelligence, Real Intelligence, Quantum Mind, And Qualia]
* [https://arxiv.org/abs/2106.13884v2 Multimodal Few-Shot Learning with Frozen Language Models]
* [https://arxiv.org/abs/2106.09685v2 LoRA: Low-Rank Adaptation of Large Language Models]
* [https://arxiv.org/abs/2106.06295v2 Going Beyond Linear Transformers with Recurrent Fast Weight Programmers]
* [https://arxiv.org/abs/2106.04647v2 Compacter: Efficient Low-Rank Hypercomplex Adapter Layers]
* [https://arxiv.org/abs/2106.04489v1 Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks]
* [https://arxiv.org/abs/2106.00882v1 Efficient Passage Retrieval with Hashing for Open-domain Question Answering]
* [https://arxiv.org/abs/2106.00874v2 Conversational Question Answering: A Survey]
 
==== May 2021 ====
 
* [https://arxiv.org/abs/2105.13290v3 CogView: Mastering Text-to-Image Generation via Transformers]
* [https://arxiv.org/abs/2105.10311v2 Pretrained Language Models for Text Generation: A Survey]
* [https://arxiv.org/abs/2105.01883v3 RepMLP: Re-parameterizing Convolutions into Fully-connected Layers for Image Recognition]
 
==== April 2021 ====
 
* [https://arxiv.org/abs/2104.14294v2 Emerging Properties in Self-Supervised Vision Transformers]
* [https://arxiv.org/abs/2104.12763v2 MDETR -- Modulated Detection for End-to-End Multi-Modal Understanding]
* [https://arxiv.org/abs/2104.06303v1 Learning and Planning in Complex Action Spaces]
* [https://arxiv.org/abs/2104.06069v2 1-bit LAMB: Communication Efficient Large-Scale Large-Batch Training with LAMB's Convergence Speed]
* [https://arxiv.org/abs/2104.05565v3 Survey on reinforcement learning for language processing]
* [https://arxiv.org/abs/2104.04644v3 Fast and Efficient Locomotion via Learned Gait Transitions]
* [https://arxiv.org/abs/2104.00298v3 EfficientNetV2: Smaller Models and Faster Training]
 
==== March 2021 ====
 
* [https://arxiv.org/abs/2103.12656v2 Replacing Rewards with Examples: Example-Based Policy Search via Recursive Classification]
* [https://arxiv.org/abs/2103.12407v4 Detecting Hate Speech with GPT-3]
* [https://arxiv.org/abs/2103.11955v3 Improving and Simplifying Pattern Exploiting Training]
* [https://arxiv.org/abs/2103.08541v1 Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence]
* [https://arxiv.org/abs/2103.00020v1 Learning Transferable Visual Models From Natural Language Supervision]
 
==== February 2021 ====
 
* [https://arxiv.org/abs/2102.11174v3 Linear Transformers Are Secretly Fast Weight Programmers]
* [https://arxiv.org/abs/2102.08597 Beyond Fully-Connected Layers with Quaternions: Parameterization of Hypercomplex Multiplications with  Parameters]
* [https://arxiv.org/abs/2102.04906 Dynamic Neural Networks: A Survey]
 
==== January 2021 ====
 
* [https://arxiv.org/abs/2101.03961v3 Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity]
 
==== December 2020 ====
 
* [https://arxiv.org/abs/2012.08508v3 Attention over learned object embeddings enables complex visual reasoning]
* [https://arxiv.org/abs/2012.06884 AIR-FI: Generating Covert Wi-Fi Signals from Air-Gapped Computers]
* [https://arxiv.org/abs/2012.05876v2 Neurosymbolic AI: The 3rd Wave]
 
==== November 2020 ====
 
* [https://arxiv.org/abs/2011.09284v2 3D imaging from multipath temporal echoes]
* [https://arxiv.org/abs/2011.06392v2 Using IPA-Based Tacotron for Data Efficient Cross-Lingual Speaker Adaptation and Pronunciation Enhancement]
* [https://arxiv.org/abs/2011.05315v2 Is Private Learning Possible with Instance Encoding?]
 
==== October 2020 ====
 
* [https://arxiv.org/abs/2010.11967v1 Language Models are Open Knowledge Graphs]
* [https://arxiv.org/abs/2010.06775v1 Vokenization: Improving Language Understanding with Contextualized, Visual-Grounded Supervision]
 
==== September 2020 ====
 
* [https://arxiv.org/abs/2009.07118v2 It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners]
* [https://arxiv.org/abs/2009.06489v2 The Hardware Lottery]
 
==== August 2020 ====
 
* [https://arxiv.org/abs/2008.02268v3 NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections]
 
==== July 2020 ====
 
* [https://arxiv.org/abs/2007.08794v3 Discovering Reinforcement Learning Algorithms]
 
==== September 2020 ====
 
* [https://arxiv.org/abs/2009.01325 Learning to summarize from human feedback]
 
==== June 2020 ====
 
* [https://arxiv.org/abs/2006.15191 Is SGD a Bayesian sampler? Well, almost]
* [https://arxiv.org/abs/2006.08517v1 The Limit of the Batch Size]
* [https://arxiv.org/abs/2006.07549v2 Hindsight Expectation Maximization for Goal-conditioned Reinforcement Learning]
* [https://arxiv.org/abs/2006.04768 Linformer: Self-Attention with Linear Complexity]
* [https://arxiv.org/abs/2006.04182v5 Predictive Coding Approximates Backprop along Arbitrary Computation Graphs]
 
==== May 2020 ====
 
* [https://arxiv.org/abs/2005.14165v4 Language Models are Few-Shot Learners]
* [https://arxiv.org/abs/2005.13213v1 Give Me Convenience and Give Her Death: Who Should Decide What Uses of NLP are Appropriate, and on What Basis?]
* [https://arxiv.org/abs/2005.07327v2 ViTAA: Visual-Textual Attributes Alignment in Person Search by Natural Language]
* [https://arxiv.org/abs/2005.05957v3 Flowtron: an Autoregressive Flow-based Generative Network for Text-to-Speech Synthesis]
* [https://arxiv.org/abs/2005.05106v2 Multi-band MelGAN: Faster Waveform Generation for High-Quality Text-to-Speech]
 
==== April 2020 ====
 
* [https://arxiv.org/abs/2004.15011v3 TLDR: Extreme Summarization of Scientific Documents]
* [https://arxiv.org/abs/2004.13637v2 Recipes for building an open-domain chatbot]
 
==== March 2020 ====
 
* [https://arxiv.org/abs/2003.06965v1 OmniTact: A Multi-Directional High Resolution Touch Sensor]
* [https://arxiv.org/abs/2003.04887v2 ReZero is All You Need: Fast Convergence at Large Depth]
 
==== February 2020 ====
 
* [https://arxiv.org/abs/2002.11794v2 Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers]
* [https://arxiv.org/abs/2002.10585v1 Backpropamine: training self-modifying neural networks with differentiable neuromodulated plasticity]
* [https://arxiv.org/abs/2002.09571v2 Learning to Continually Learn]
* [https://arxiv.org/abs/2002.00388v4 A Survey on Knowledge Graphs: Representation, Acquisition and Applications]
* [https://arxiv.org/abs/2003.02645 SentenceMIM: A Latent Variable Language Model]
 
==== January 2020 ====
 
* [https://arxiv.org/abs/2001.04063 ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training]
* [https://arxiv.org/abs/2001.09977v3 Towards a Human-like Open-Domain Chatbot]
* [https://arxiv.org/abs/2001.07676v3 Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference]
* [https://arxiv.org/abs/2001.06782v4 Gradient Surgery for Multi-Task Learning]
* [https://arxiv.org/abs/2001.04463v3 Unsupervised Audiovisual Synthesis via Exemplar Autoencoders]
 
==== December 2019 ====
 
* [https://arxiv.org/abs/1912.07768v1 Generative Teaching Networks: Accelerating Neural Architecture Search by Learning to Generate Synthetic Training Data]
* [https://arxiv.org/abs/1912.02315v2 12-in-1: Multi-Task Vision and Language Representation Learning]
* [https://arxiv.org/abs/1912.01412v1 Deep Learning for Symbolic Mathematics]
* [https://arxiv.org/abs/1912.01320v1 ATIS + SpiNNaker: a Fully Event-based Visual Tracking Demonstration]
 
==== November 2019 ====
 
* [https://arxiv.org/abs/1911.08265v2 Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model]
 
==== October 2019 ====
 
* [https://arxiv.org/abs/1910.14238v1 Learning Disentangled Representations for Recommendation]
* [https://arxiv.org/abs/1910.06764v1 Stabilizing Transformers for Reinforcement Learning]
* [https://arxiv.org/abs/1910.06711v3 MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis]
* [https://arxiv.org/abs/1910.03175v5 MIM: Mutual Information Machine]
* [https://arxiv.org/abs/1901.07677 Modeling Human Motion with Quaternion-based Neural Networks]
 
==== September 2019 ====
 
* [https://arxiv.org/abs/1909.08593v2 Fine-Tuning Language Models from Human Preferences]
* [https://arxiv.org/abs/1909.03341v2 Neural Machine Translation with Byte-Level Subwords]
 
==== July 2019 ====
 
* [https://arxiv.org/abs/1907.05242v2 Large Memory Layers with Product Keys]
* [https://arxiv.org/abs/1907.03976v3 Better-than-Demonstrator Imitation Learning via Automatically-Ranked Demonstrations]
 
==== June 2019 ====
 
* [https://arxiv.org/abs/1906.05833v2 There is no Artificial General Intelligence]
 
==== May 2019 ====
 
* [https://arxiv.org/abs/1905.11946v5 EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks]
* [https://arxiv.org/abs/1905.07579v1 Combining Experience Replay with Exploration by Random Network Distillation]
 
==== April 2019 ====
 
* [https://arxiv.org/abs/1904.10278v2 Improving Differentiable Neural Computers Through Memory Masking, De-allocation, and Link Distribution Sharpness Control]
* [https://arxiv.org/abs/1904.00962v5 Large Batch Optimization for Deep Learning: Training BERT in 76 minutes]
 
==== January 2019 ====
 
* [https://arxiv.org/abs/1901.10995v4 Go-Explore: a New Approach for Hard-Exploration Problems]
* [https://arxiv.org/abs/1901.01753v3 Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions]
 
==== November 2018 ====
 
* [https://arxiv.org/abs/1811.12927v2 Hierarchical Policy Design for Sample-Efficient Learning of Robot Table Tennis Through Self-Play]
* [https://arxiv.org/abs/1811.11742v2 3D human pose estimation in video with temporal convolutions and semi-supervised training]
* [https://arxiv.org/abs/1811.00945v2 Image Chat: Engaging Grounded Conversations]
* [https://arxiv.org/abs/1811.00207v5 Towards Empathetic Open-domain Conversation Models: a New Benchmark and Dataset]
 
==== October 2018 ====
 
* [https://arxiv.org/abs/1810.12894v1 Exploration by Random Network Distillation]
* [https://arxiv.org/abs/1810.12217v1 Dreaming neural networks: forgetting spurious memories and reinforcing pure ones]
* [https://arxiv.org/abs/1810.04805v2 BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding]
 
==== September 2018 ====
 
* [https://arxiv.org/abs/1809.09600v1 HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering]
* [https://arxiv.org/abs/1809.08267v3 Neural Approaches to Conversational AI]
 
==== July 2018 ====
 
* [https://arxiv.org/abs/1807.08107v1 Person Search via A Mask-Guided Two-Stream CNN Model]
 
==== June 2018 ====
 
* [https://arxiv.org/abs/1806.03822v1 Know What You Don't Know: Unanswerable Questions for SQuAD]
 
==== April 2018 ====
 
* [https://arxiv.org/abs/1804.02464v3 Differentiable plasticity: training plastic neural networks with backpropagation]
* [https://arxiv.org/abs/1804.01756v3 The Kanerva Machine: A Generative Distributed Memory]
 
==== March 2018 ====
 
* [https://arxiv.org/abs/1904.06736v1 A Short Survey On Memory Based Reinforcement Learning]
* [https://arxiv.org/abs/1803.11175v2 Universal Sentence Encoder]
* [https://arxiv.org/abs/1803.10760 Unsupervised Predictive Memory in a Goal-Directed Agent]
* [https://arxiv.org/abs/1803.10122v4 World Models]
 
==== February 2018 ====
 
* [https://arxiv.org/abs/1802.01561v3 IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures]
 
==== January 2018 ====
 
* [https://arxiv.org/abs/1810.10222v1 Universal Language Model Fine-Tuning with Subword Tokenization for Polish]
* [https://arxiv.org/abs/1801.07243v5 Personalizing Dialogue Agents: I have a dog, do you have pets too?]
* [https://arxiv.org/abs/1801.05667v1 Innateness, AlphaZero, and Artificial Intelligence]
 
==== December 2017 ====
 
* [https://arxiv.org/abs/1712.05884v2 Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions]
* [https://arxiv.org/abs/1712.02950v2 CycleGAN, a Master of Steganography]
* [https://arxiv.org/abs/1712.01815v1 Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm]
 
==== November 2017 ====
 
* [https://arxiv.org/abs/1711.00489v2 Don't Decay the Learning Rate, Increase the Batch Size]
 
==== October 2017 ====
 
* [https://arxiv.org/abs/1710.05941v2 Searching for Activation Functions]
* [https://arxiv.org/abs/1710.03957v1 DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset]
 
==== September 2017 ====
 
* [https://arxiv.org/abs/1709.07871v2 FiLM: Visual Reasoning with a General Conditioning Layer]
 
==== August 2017 ====
 
* [https://arxiv.org/abs/1808.04293v1 Fast, Better Training Trick -- Random Gradient]
* [https://arxiv.org/abs/1708.02190v3 Intrinsically Motivated Goal Exploration Processes with Automatic Curriculum Learning]
 
==== July 2017 ====
 
* [https://arxiv.org/abs/1707.01495v3 Hindsight Experience Replay]
 
==== June 2017 ====
 
* [https://arxiv.org/abs/2002.05202v1 GLU Variants Improve Transformer]
 
==== May 2017 ====
 
* [https://arxiv.org/abs/1705.03551v2 TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension]
* [https://arxiv.org/abs/1705.05363 Curiosity-driven Exploration by Self-supervised Prediction]
 
==== March 2017 ====
 
* [https://arxiv.org/abs/1703.10135v2 Tacotron: Towards End-to-End Speech Synthesis]
 
==== November 2016 ====
 
* [https://arxiv.org/abs/1611.08669v5 Visual Dialog]
* [https://arxiv.org/abs/1611.04558v2 Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation]
 
==== October 2016 ====
 
* [https://arxiv.org/abs/1610.00956v1 Embracing data abundance: BookTest Dataset for Reading Comprehension]
 
==== September 2016 ====
 
* [https://arxiv.org/abs/1609.04836v2 On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima]
* [https://arxiv.org/abs/1609.02228v2 Learning to learn with backpropagation of Hebbian plasticity]
 
==== July 2016 ====
 
* [https://arxiv.org/abs/1607.06450v1 Layer Normalization]
 
==== June 2016 ====
 
* [https://arxiv.org/abs/1606.08415v4 Gaussian Error Linear Units (GELUs)]
* [https://arxiv.org/abs/1606.06565v2 Concrete Problems in AI Safety]
 
==== May 2016 ====
 
* [https://arxiv.org/abs/1605.07683v4 Learning End-to-End Goal-Oriented Dialog]
 
==== November 2015 ====
 
* [https://arxiv.org/abs/1511.02301v4 The Goldilocks Principle: Reading Children's Books with Explicit Memory Representations]
 
==== June 2015 ====
 
* [https://arxiv.org/abs/1506.08909v3 The Ubuntu Dialogue Corpus: A Large Dataset for Research in Unstructured Multi-Turn Dialogue Systems]
* [https://arxiv.org/abs/1506.03340v3 Teaching Machines to Read and Comprehend]
 
==== May 2015 ====
 
* [https://arxiv.org/abs/1505.06366v2 Open Ended Intelligence: The individuation of Intelligent Agents]
 
==== June 2015 ====
* [https://arxiv.org/abs/1506.08909v3 The Ubuntu Dialogue Corpus: A Large Dataset for Research in Unstructured Multi-Turn Dialogue Systems]
* [https://arxiv.org/abs/1506.03340v3 Teaching Machines to Read and Comprehend]
 
==== July 2014 ====
 
* [https://arxiv.org/abs/1407.3501v4 Robots that can adapt like animals]
 
==== December 2012 ====
 
* [https://ieeexplore.ieee.org/document/6386109 MuJoCo: A physics engine for model-based control]
 
==== September 2003 ====
 
* [https://arxiv.org/abs/cs/0309048 Goedel Machines: Self-Referential Universal Problem Solvers Making Provably Optimal Self-Improvements]
 
== Previously ==
 
{{Tidyup|This page needs to be completely reformatted. Will be changing the tl;drs to be title text so you can hover over links to get the gist of them without clicking.}}
 
=== [[Instruction tuning]] ===
{{paper|title=Evol-Instruct: Mass-Producing Open-Domain Instruction Data with Varying Levels of Complexity using Large Language Models|url=https://arxiv.org/abs/2304.12244|tldr=The paper proposes a method called '''[[Evol-Instruct]]''' for creating large amounts of instruction data with different levels of complexity using a large language model (LLM) instead of humans. The generated data is used to fine-tune another LLM called WizardLM. Human evaluations show that Evol-Instruct instructions are better than human-created ones, and WizardLM is preferred over OpenAI ChatGPT for complex tasks. The study suggests that fine-tuning LLMs with AI-evolved instructions is a promising approach for improving their performance.|authors=Xu et al|publication=arXiv:2304.12244|year=2023}}
 
== 2022 ==
 
=== November 2022 ===
{{paper|title=Large Language Models Are Human-Level Prompt Engineers|url=https://arxiv.org/abs/2211.01910|tldr=[https://openreview.net/pdf?id=92gvk82DE- OpenReview version]. Automatic Prompt Engineer (APE) is a method that can generate instructions automatically. It uses a pool of generated instruction candidates and evaluates the quality of them by the zero-shot performance of another LLM following a selected instruction.|authors=Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, Jimmy Ba|publication=arXiv|year=2022}}
 
== 2021 ==
{{Protip|You can use [https://huggingface.co/sshleifer/distilbart-cnn-12-6 sshleifer/distilbart-cnn-12-6] to help with summarizing papers. Check the [[Template:Paper|paper template]] for usage instructions.<br>'''2023 update:''' Leaving this note here as a relic of how much things have progressed. '''''PROTIP''''': Use GPT-4.}}


== Recent papers ==
{{Protip|You can use [https://huggingface.co/sshleifer/distilbart-cnn-12-6 sshleifer/distilbart-cnn-12-6] and [https://scitldr.apps.allenai.org/ SciTLDR] to help with summarizing papers. Check the [[Template:Paper|paper template]] for usage instructions.}}
=== August 2021 ===
=== August 2021 ===


Line 10: Line 582:


==== [[Simulation]] ====
==== [[Simulation]] ====
{{paper|title=iGibson 2.0: Object-Centric Simulation for Robot Learning of Everyday Household Tasks|url=http://svl.stanford.edu/igibson/|tldr=iGibson 2.0 is a novel simulation environment using [[Bullet]] that supports the simulation of a more diverse set of household tasks through three key innovations. Firstly, it supports object states, including temperature, wetness level, cleanliness level, and toggled and sliced states, necessary to cover a wider range of tasks. Second, it implements a set of predicate logic functions that map the simulator states to logic states like Cooked or Soaked. Third, the simulator can sample valid physical states that satisfy a logic state. This functionality can generate potentially infinite instances of tasks with minimal effort from the users.|publication=arXiv:2108.03272|authors=Chengshu Li, Fei Xia, Roberto Martín-Martín, Michael Lingelbach, Sanjana Srivastava, Bokui Shen, Kent Vainio, Cem Gokmen, Gokul Dharan, Tanish Jain, Andrey Kurenkov, Karen Liu, Hyowon Gweon, Jiajun Wu, Li Fei-Fei, Silvio Savarese|year=2021}}
{{paper|title=iGibson 2.0: Object-Centric Simulation for Robot Learning of Everyday Household Tasks|url=http://svl.stanford.edu/igibson/|tldr=iGibson 2.0 is a novel simulation environment using [[Bullet]] that supports the simulation of a more diverse set of household tasks through three key innovations. Firstly, it supports object states, including temperature, wetness level, cleanliness level, and toggled and sliced states, necessary to cover a wider range of tasks. Second, it implements a set of predicate logic functions that map the simulator states to logic states like Cooked or Soaked. Third, the simulator can sample valid physical states that satisfy a logic state. This functionality can generate potentially infinite instances of tasks with minimal effort from the users.|publication=arXiv:2108.03272|authors=Chengshu Li, Fei Xia, Roberto Martín-Martín, Michael Lingelbach, Sanjana Srivastava, Bokui Shen, Kent Vainio, Cem Gokmen, Gokul Dharan, Tanish Jain, Andrey Kurenkov, Karen Liu, Hyowon Gweon, Jiajun Wu, Li Fei-Fei, Silvio Savarese|year=2021}}


Line 21: Line 592:


==== [[Multimodal learning]] ====
==== [[Multimodal learning]] ====
{{paper|title=Multimodal Few-Shot Learning with Frozen Language Models|url=https://arxiv.org/abs/2108.03880|tldr=When trained at sufficient scale, auto-regressive language models exhibit the notable ability to learn a new language task after being prompted with just a few examples. Here, the authors present a simple, yet effective, approach for transferring this few-shot learning ability to a multimodal setting (vision and language).|authors=Maria Tsimpoukelli, Jacob Menick, Serkan Cabi, S. M. Ali Eslami, Oriol Vinyals, Felix Hill|publication=arXiv:2106.13884|year=2021}}
{{paper|title=Multimodal Few-Shot Learning with Frozen Language Models|url=https://arxiv.org/abs/2108.03880|tldr=When trained at sufficient scale, auto-regressive language models exhibit the notable ability to learn a new language task after being prompted with just a few examples. Here, the authors present a simple, yet effective, approach for transferring this few-shot learning ability to a multimodal setting (vision and language).|authors=Maria Tsimpoukelli, Jacob Menick, Serkan Cabi, S. M. Ali Eslami, Oriol Vinyals, Felix Hill|publication=arXiv:2106.13884|year=2021}}


==== [[Optimizers]] ====
==== [[Optimizers]] ====
{{paper|title=A Generalizable Approach to Learning Optimizers|url=https://arxiv.org/abs/2106.00958|tldr=Learning to update optimizer hyperparameters instead of model parameters directly using novel features, actions, and a reward function.|authors=Diogo Almeida, Clemens Winter, Jie Tang, Wojciech Zaremba|publication=arXiv:2106.00958|year=2021}}
{{paper|title=A Generalizable Approach to Learning Optimizers|url=https://arxiv.org/abs/2106.00958|tldr=Learning to update optimizer hyperparameters instead of model parameters directly using novel features, actions, and a reward function.|authors=Diogo Almeida, Clemens Winter, Jie Tang, Wojciech Zaremba|publication=arXiv:2106.00958|year=2021}}


Line 36: Line 605:
==== [[Fine-tuning]] ====
==== [[Fine-tuning]] ====
{{paper|title=The Power of Scale for Parameter-Efficient Prompt Tuning|url=https://arxiv.org/abs/2104.08691|tldr=In this work, the author's explore "prompt tuning" a simple but effective mechanism for learning "soft prompts" to condition frozen language models to perform specific downstream tasks.|publication=arXiv:2104.08691|year=2021|authors=Brian Lester, Rami Al-Rfou, Noah Constant}}
{{paper|title=The Power of Scale for Parameter-Efficient Prompt Tuning|url=https://arxiv.org/abs/2104.08691|tldr=In this work, the author's explore "prompt tuning" a simple but effective mechanism for learning "soft prompts" to condition frozen language models to perform specific downstream tasks.|publication=arXiv:2104.08691|year=2021|authors=Brian Lester, Rami Al-Rfou, Noah Constant}}
=== March 2021 ===
==== [[Computer vision]] ====
{{paper|title=NeX: Real-time View Synthesis with Neural Basis Expansion|url=https://nex-mpi.github.io/|publication=arXiv:2103.05606|authors=Suttisak Wizadwongsa, Pakkapon Phongthawee, Jiraphon Yenphraphai, Supasorn Suwajanakorn|tldr=The authors present NeX, a new approach to novel view synthesis based on enhancements of multiplane image (MPI) that can reproduce next-level view-dependent effects -- in real time. The method achieves the best overall scores across all major metrics on these datasets with more than 1000× faster rendering time than the state of the art.|year=2021}}


=== October 2020 ===
=== October 2020 ===


==== Computer vision ====
==== [[Computer vision]] ====
{{paper|title=GRF: Learning a General Radiance Field for 3D Scene Representation and Rendering|url=https://arxiv.org/abs/2010.04595|tldr=General Radiance Fields construct an internal representation for each 3D point of a scene from 2D inputs and renders the corresponding appearance and geometry of any 3D scene viewing from an arbitrary angle.|authors=Alex Trevithick, Bo Yang|publication=arXiv:2010.04595|year=2020}}
{{paper|title=GRF: Learning a General Radiance Field for 3D Scene Representation and Rendering|url=https://arxiv.org/abs/2010.04595|tldr=General Radiance Fields construct an internal representation for each 3D point of a scene from 2D inputs and renders the corresponding appearance and geometry of any 3D scene viewing from an arbitrary angle.|authors=Alex Trevithick, Bo Yang|publication=arXiv:2010.04595|year=2020}}


Line 45: Line 618:
==== [[Summarization]] ====
==== [[Summarization]] ====
{{paper|title=Learning to Summarize with Human Feedback|url=https://openai.com/blog/learning-to-summarize-with-human-feedback/|tldr=Human feedback models outperform much larger supervised models and reference summaries on TL;DR.|authors=Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, Paul Christiano|publication=arXiv:2009.01325|year=2020}}
{{paper|title=Learning to Summarize with Human Feedback|url=https://openai.com/blog/learning-to-summarize-with-human-feedback/|tldr=Human feedback models outperform much larger supervised models and reference summaries on TL;DR.|authors=Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, Paul Christiano|publication=arXiv:2009.01325|year=2020}}
=== December 2019 ===
==== [[Meta-learning]] ====
{{paper|title=Generative Teaching Networks: Accelerating Neural Architecture Search by Learning to Generate Synthetic Training Data|url=https://arxiv.org/abs/1912.07768|publication=arXiv:1912.07768|year=2019|authors=Felipe Petroski Such, Aditya Rawal, Joel Lehman, Kenneth O. Stanley, Jeff Clune|tldr=This paper investigates the intriguing question of whether learning algorithms can automatically generate training data, learning environments, and curricula in order to help AI agents rapidly learn. GTNs are deep neural networks that generate data and/or training environments that a learner trains on for a few SGD steps before being tested on a target task. It then differentiates through the entire learning process via meta-gradients to update the GTN parameters to improve performance on the target task.}}


== Older papers ==
== Older papers ==


* [[List of 2018 papers]]
* [[List of 2018 papers]]
== See also ==
* [[Requests for research]]
* [[Research and development]]


== References ==
== References ==
<references />
<references />
__NOTOC__
[[Category:Research]]

Latest revision as of 23:02, 11 July 2023

This page requires expansion!
This page needs papers! Probably should set up an automated system so I can just drop Twitter and Arxiv links.
In Robowaifu.tech papers hold onto you.

This page serves to collect notable research papers related to robotics and artificial intelligence, particularly ones that can be used by hobbyists with minimal resources towards creating robowaifus. Feel free to add new papers to the list and discuss any papers on the talk page. Papers posted on /robowaifu/ will also eventually appear here.

Search sites

Social media sources

List of papers

Unsorted

Need to summarize these papers into tl;dr. An automated system for this would be great.

June 2023

May 2023

April 2023

March 2023

Februrary 2023

January 2023

December 2022

November 2022

October 2022

September 2022

August 2022

July 2022

June 2022

May 2022

March 2022

February 2022

January 2022

Februrary 2022

December 2021

November 2021

October 2021

September 2021

August 2021

June 2021

May 2021

April 2021

March 2021

February 2021

January 2021

December 2020

November 2020

October 2020

September 2020

August 2020

July 2020

September 2020

June 2020

May 2020

April 2020

March 2020

February 2020

January 2020

December 2019

November 2019

October 2019

September 2019

July 2019

June 2019

May 2019

April 2019

January 2019

November 2018

October 2018

September 2018

July 2018

June 2018

April 2018

March 2018

February 2018

January 2018

December 2017

November 2017

October 2017

September 2017

August 2017

July 2017

June 2017

May 2017

March 2017

November 2016

October 2016

September 2016

July 2016

June 2016

May 2016

November 2015

June 2015

May 2015

June 2015

July 2014

December 2012

September 2003

Previously

This page requires tidying up!
This page needs to be completely reformatted. Will be changing the tl;drs to be title text so you can hover over links to get the gist of them without clicking.

Instruction tuning

Evol-Instruct: Mass-Producing Open-Domain Instruction Data with Varying Levels of Complexity using Large Language Models (arXiv:2304.12244)

tl;dr The paper proposes a method called Evol-Instruct for creating large amounts of instruction data with different levels of complexity using a large language model (LLM) instead of humans. The generated data is used to fine-tune another LLM called WizardLM. Human evaluations show that Evol-Instruct instructions are better than human-created ones, and WizardLM is preferred over OpenAI ChatGPT for complex tasks. The study suggests that fine-tuning LLMs with AI-evolved instructions is a promising approach for improving their performance.[1]


2022

November 2022

Large Language Models Are Human-Level Prompt Engineers (arXiv)

tl;dr OpenReview version. Automatic Prompt Engineer (APE) is a method that can generate instructions automatically. It uses a pool of generated instruction candidates and evaluates the quality of them by the zero-shot performance of another LLM following a selected instruction.[2]


2021

PROTIP: You can use sshleifer/distilbart-cnn-12-6 to help with summarizing papers. Check the paper template for usage instructions.
2023 update: Leaving this note here as a relic of how much things have progressed. PROTIP: Use GPT-4.

August 2021

Computer vision

NeuralMVS: Bridging Multi-View Stereo and Novel View Synthesis (arXiv:2108.03880)

tl;dr Multi-view stereo is a core task in 3D computer vision. NeRF methods do not generalize to novel scenes and are slow to train and test. We propose to bridge the gap between these two methodologies with a novel network that can recover 3D scene geometry as a distance function.[3]


Simulation

iGibson 2.0: Object-Centric Simulation for Robot Learning of Everyday Household Tasks (arXiv:2108.03272)

tl;dr iGibson 2.0 is a novel simulation environment using Bullet that supports the simulation of a more diverse set of household tasks through three key innovations. Firstly, it supports object states, including temperature, wetness level, cleanliness level, and toggled and sliced states, necessary to cover a wider range of tasks. Second, it implements a set of predicate logic functions that map the simulator states to logic states like Cooked or Soaked. Third, the simulator can sample valid physical states that satisfy a logic state. This functionality can generate potentially infinite instances of tasks with minimal effort from the users.[4]


July 2021

Audio processing

SoundStream: An End-to-End Neural Audio Codec (arXiv:2107.03312)

tl;dr A novel neural audio codec that can efficiently compress speech, music and general audio at bitrates normally targeted by speech-tailored codecs. SoundStream at 3kbps outperforms Opus at 12kbps and approaches EVS at 9.6kbps.[5]


June 2021

Multimodal learning

Multimodal Few-Shot Learning with Frozen Language Models (arXiv:2106.13884)

tl;dr When trained at sufficient scale, auto-regressive language models exhibit the notable ability to learn a new language task after being prompted with just a few examples. Here, the authors present a simple, yet effective, approach for transferring this few-shot learning ability to a multimodal setting (vision and language).[6]


Optimizers

A Generalizable Approach to Learning Optimizers (arXiv:2106.00958)

tl;dr Learning to update optimizer hyperparameters instead of model parameters directly using novel features, actions, and a reward function.[7]


May 2021

Memory

Not All Memories are Created Equal: Learning to Forget by Expiring (arXiv:2105.06548)

tl;dr The authors propose Expire-Span, a method that learns to retain the most important information and expire the irrelevant information, which enables Transformers to scale to attend over tens of thousands of previous timesteps efficiently.[8]


April 2021

Fine-tuning

The Power of Scale for Parameter-Efficient Prompt Tuning (arXiv:2104.08691)

tl;dr In this work, the author's explore "prompt tuning" a simple but effective mechanism for learning "soft prompts" to condition frozen language models to perform specific downstream tasks.[9]


March 2021

Computer vision

NeX: Real-time View Synthesis with Neural Basis Expansion (arXiv:2103.05606)

tl;dr The authors present NeX, a new approach to novel view synthesis based on enhancements of multiplane image (MPI) that can reproduce next-level view-dependent effects -- in real time. The method achieves the best overall scores across all major metrics on these datasets with more than 1000× faster rendering time than the state of the art.[10]


October 2020

Computer vision

GRF: Learning a General Radiance Field for 3D Scene Representation and Rendering (arXiv:2010.04595)

tl;dr General Radiance Fields construct an internal representation for each 3D point of a scene from 2D inputs and renders the corresponding appearance and geometry of any 3D scene viewing from an arbitrary angle.[11]


September 2020

Summarization

Learning to Summarize with Human Feedback (arXiv:2009.01325)

tl;dr Human feedback models outperform much larger supervised models and reference summaries on TL;DR.[12]


December 2019

Meta-learning

Generative Teaching Networks: Accelerating Neural Architecture Search by Learning to Generate Synthetic Training Data (arXiv:1912.07768)

tl;dr This paper investigates the intriguing question of whether learning algorithms can automatically generate training data, learning environments, and curricula in order to help AI agents rapidly learn. GTNs are deep neural networks that generate data and/or training environments that a learner trains on for a few SGD steps before being tested on a target task. It then differentiates through the entire learning process via meta-gradients to update the GTN parameters to improve performance on the target task.[13]


Older papers

See also

References

  1. ↑ Xu et al. Evol-Instruct: Mass-Producing Open-Domain Instruction Data with Varying Levels of Complexity using Large Language Models. arXiv:2304.12244, 2023.
  2. ↑ Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, Jimmy Ba. Large Language Models Are Human-Level Prompt Engineers. arXiv, 2022.
  3. ↑ Radu Alexandru Rosu, Sven Behnke. NeuralMVS: Bridging Multi-View Stereo and Novel View Synthesis. arXiv:2108.03880, 2021.
  4. ↑ Chengshu Li, Fei Xia, Roberto Martín-Martín, Michael Lingelbach, Sanjana Srivastava, Bokui Shen, Kent Vainio, Cem Gokmen, Gokul Dharan, Tanish Jain, Andrey Kurenkov, Karen Liu, Hyowon Gweon, Jiajun Wu, Li Fei-Fei, Silvio Savarese. iGibson 2.0: Object-Centric Simulation for Robot Learning of Everyday Household Tasks. arXiv:2108.03272, 2021.
  5. ↑ Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, Marco Tagliasacchi. SoundStream: An End-to-End Neural Audio Codec. arXiv:2107.03312, 2021.
  6. ↑ Maria Tsimpoukelli, Jacob Menick, Serkan Cabi, S. M. Ali Eslami, Oriol Vinyals, Felix Hill. Multimodal Few-Shot Learning with Frozen Language Models. arXiv:2106.13884, 2021.
  7. ↑ Diogo Almeida, Clemens Winter, Jie Tang, Wojciech Zaremba. A Generalizable Approach to Learning Optimizers. arXiv:2106.00958, 2021.
  8. ↑ Sainbayar Sukhbaatar, Da Ju, Spencer Poff, Stephen Roller, Arthur Szlam, Jason Weston, Angela Fan. Not All Memories are Created Equal: Learning to Forget by Expiring. arXiv:2105.06548, 2021.
  9. ↑ Brian Lester, Rami Al-Rfou, Noah Constant. The Power of Scale for Parameter-Efficient Prompt Tuning. arXiv:2104.08691, 2021.
  10. ↑ Suttisak Wizadwongsa, Pakkapon Phongthawee, Jiraphon Yenphraphai, Supasorn Suwajanakorn. NeX: Real-time View Synthesis with Neural Basis Expansion. arXiv:2103.05606, 2021.
  11. ↑ Alex Trevithick, Bo Yang. GRF: Learning a General Radiance Field for 3D Scene Representation and Rendering. arXiv:2010.04595, 2020.
  12. ↑ Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, Paul Christiano. Learning to Summarize with Human Feedback. arXiv:2009.01325, 2020.
  13. ↑ Felipe Petroski Such, Aditya Rawal, Joel Lehman, Kenneth O. Stanley, Jeff Clune. Generative Teaching Networks: Accelerating Neural Architecture Search by Learning to Generate Synthetic Training Data. arXiv:1912.07768, 2019.
Retrieved from "https://robowaifu.tech/w/index.php?title=Latest_papers&oldid=358"