Multimodal AI
Vision-language-audio unification, cross-modal understanding, and unified sequence modeling. Making AI see, hear, and understand the world.
Key Benchmarks
Recent Papers
From SRA to Self-Flow: Data Augmentation or Self-Supervision?
Dengyang Jiang, Mengmeng Wang, Harry Yang +1 more
AnyGroundBench: A Specialized-Domain Benchmark for Video Grounding in Vision-Language Models
Rintaro Otsubo, Ryo Fujii, Reina Ishikawa +6 more
Multi-Resolution Flow Matching: Training-Free Diffusion Acceleration via Staged Sampling
Xingyu Zheng, Xianglong Liu, Yifu Ding +4 more
Optimizing Visual Generative Models via Distribution-wise Rewards
Ruihang Li, Mengde Xu, Shuyang Gu +4 more
MER-R1: Multimodal Emotion Reasoning via Slow-Fast Thinking Synergy
Zhiyuan Han, Beier Zhu, Wenwen Tong +8 more
Not All Relations Rotate Alike: Transformation-Aware Decoupling for Viewpoint-Robust 3D Scene Graph Generation
Jingjun Sun, Chaowei Wang, Zhirui Liu +4 more
Hallucination in World Models is Predictable and Preventable
Nicklas Hansen, Xiaolong Wang
Running the Gauntlet: Re-evaluating the Capabilities of Agents Beyond Familiar Environments
Mykola Vysotskyi, Runqi Lin, Grzegorz Biziel +22 more
DragMesh-2: Physically Plausible Dexterous Hand-Object Interaction with Articulated Objects
Tianshan Zhang, Yijia Duan, Yanjun Li +2 more
S-Agent: Spatial Tool-Use Elicits Reasoning for Spatial Intelligence
Yalun Dai, Hao Li, Shulin Tian +10 more
Recent Milestones
OpenAI rolls out GPT‑Live real‑time voice
OpenAI launched GPT-Live on July 8, 2026, introducing new GPT‑Live‑1 and GPT‑Live‑1 mini voice models that power a more natural real-time ‘ChatGPT Live’ experience across its apps. The update lets ChatGPT listen and speak simultaneously, handle interruptions, and shows visual responses while adding safeguards to prevent voice impersonation.
Meta debuts Muse Image for social apps
Meta introduced Muse Image on July 7, 2026, its first in‑house image‑generation model built by Meta Superintelligence Labs. The system now powers image creation in Meta AI, Instagram Stories and WhatsApp chats, with more than 30 new AI effects and a preview of a coming Muse Video model. ([tech.yahoo.com](https://tech.yahoo.com/ai/meta-ai/articles/meta-launches-muse-image-first-185347805.html?utm_source=openai))
Tripo AI Secures $150M for 3D World Models
On July 6, 2026, Tripo AI announced it had raised $150 million in a Series A3 round to deepen work on 3D foundation models and world-model technologies. The financing drew strategic investment from Geely Capital and several Chinese gaming companies, including 4399 Network, Tanwan and Giant Network, alongside existing backers INCE Capital and Genesis Capital.
xAI completes Grok Imagine image+video suite
Elon Musk said “Done with Grok Imagine” on July 5, signaling that xAI has completed development of its Grok Imagine image and video generation feature. The multimodal tool, already in beta, is now positioned as a fully built‑out creative suite integrated across Grok and the X app.
Kling AI raises $3B to challenge Sora
Pandaily reports that Chinese short-video giant Kuaishou is spinning off its Kling AI video generation unit into an independent company and backing it with a funding round of up to $3 billion. Tencent, Alibaba and Baidu are named as key investors, and the new entity reportedly faces a mandatory IPO deadline in 2031.
China’s Kling AI Scores $2B for Video Models
Kuaishou’s video-generation unit Kling AI has closed an initial $2 billion funding round at around a $15 billion pre-money valuation. The round includes Alibaba, Tencent, Baidu and several major Chinese and Gulf investment firms, and could expand to $3 billion, cutting Kuaishou’s stake to about 68%.
Google Drives AI Image Costs Toward Pennies
Google has rolled out its Nano Banana 2 Lite (Gemini 3.1 Flash-Lite Image) model, a fast low-cost image generator now available via Gemini API, Google AI Studio and the Gemini Enterprise Agent Platform. The model generates 1K-resolution images in about four seconds at roughly $0.034 per image, with rollouts and coverage reported across India and Japan on July 1, 2026.([gadgets360.com](https://www.gadgets360.com/ai/news/google-personalised-gemini-intelligence-ai-image-creation-nano-banana-photos-free-11711303?utm_source=openai))
Gemini Makes Personal Image AI Free in US
Google made Gemini's personalized Nano Banana-powered image generation free for eligible U.S. users via the Gemini app on June 29, 2026.([techcrunch.com](https://techcrunch.com/2026/06/29/geminis-personalized-ai-image-generation-is-now-free-for-u-s-users/)) The feature, previously limited to paid tiers, uses data from services like Gmail, Photos and Search to create images tailored to each user’s interests and photos.([techcrunch.com](https://techcrunch.com/2026/06/29/geminis-personalized-ai-image-generation-is-now-free-for-u-s-users/))
L’Oréal turns ChatGPT into AI beauty counter
L’Oréal announced a strategic partnership with OpenAI at VivaTech 2026, enabling Maybelline New York’s ModiFace-powered virtual makeup try-on directly inside ChatGPT. The deal also includes promoting L’Oréal brands via product ads in ChatGPT in the US, with further integrations for Lancôme and Kérastase to follow.
Gemma 4 12B puts open multimodal AI on laptops
Regional coverage in Chinese and Korean tech media highlights Google’s new Gemma 4 12B model, a 12B‑parameter unified multimodal model released under Apache 2.0 that can run on laptops with 16GB of memory.([blog.google](https://blog.google/innovation-and-ai/technology/developers-tools/introducing-gemma-4-12b/?utm_source=openai)) The model removes heavy vision and audio encoders, enabling local image, audio and video understanding on consumer hardware.
Reactor raises $59M for real‑time AI world models
On May 28, 2026 AWS announced that startup Reactor emerged from stealth with $59 million in funding led by Lightspeed Venture Partners. The company is building a platform for real‑time “world model”–based generative video and interactive experiences, with early adopters like Overworld already building on it.
Microsoft’s MAI-Image-2.5 hits No.3 on Arena
On May 26–27, 2026, Microsoft announced MAI‑Image‑2.5, a new in‑house image generation model that ranks third on Arena’s text‑to‑image leaderboard. Japanese outlet GIGAZINE reported on May 27 that the model will roll out to the MAI Playground within about two weeks.([gigazine.net](https://gigazine.net/gsc_news/en/20260527-mai-image-25-generation-ai/))
Microsoft debuts in-house MAI speech & image AIs
On April 4, 2026, Tech Insider detailed how Microsoft has launched three in‑house MAI models — MAI‑Transcribe‑1, MAI‑Voice‑1 and MAI‑Image‑2 — following their April 2 release. The models target speech recognition, voice generation and image creation, and are being rolled out via Microsoft’s Foundry and MAI Playground platforms.
Microsoft MAI Models Cut OpenAI Dependence
On April 2–3, 2026, Microsoft’s MAI division rolled out three in‑house foundational models—MAI‑Transcribe‑1 for speech‑to‑text, MAI‑Voice‑1 for speech generation and MAI‑Image‑2 for image generation—through its Foundry and MAI Playground platforms. Coverage on April 3 details aggressive pricing that undercuts rival cloud providers and confirms the models are already being integrated into Copilot, Teams, Bing, PowerPoint and Azure Speech.
Krafton Raon: Open Voice + Vision Model Suite
Krafton on April 2, 2026 launched its new AI model brand ‘Raon’ and released four open-source models on Hugging Face. The suite includes a 9B-parameter speech LLM, a real-time full‑duplex speech chat model, a text‑to‑speech model, and a vision encoder that in some tasks outperforms Google’s SigLIP2.
Microsoft MAI launches cheaper AI model stack
On April 2, 2026, Microsoft AI unveiled three new foundation models — MAI-Transcribe-1, MAI-Voice-1 and MAI-Image-2 — for text, speech and image generation. The models are available immediately through Microsoft Foundry and MAI Playground, with pricing pitched as cheaper than rival offerings from OpenAI and Google.
ByteDance Seedance 2.0 tops AI video league
On February 10, 2026, Chinese media reported that ByteDance’s Seedance 2.0 AI video generation model, currently in limited testing, is being widely praised for creating movie‑level multi‑scene videos with synchronized audio from text or images. High‑profile creators and game developers called it the “strongest video generation model,” triggering a rally in China’s media and content stocks and sparking debate over copyright and deepfake risks.
ByteDance Seedance 2.0 Ups China’s Video Game
On February 9, 2026, Chinese media highlighted ByteDance’s Seedance 2.0, a next‑generation AI video model that can generate multi‑shot 2K videos with synchronized audio in under a minute. Official materials say the dual‑branch diffusion transformer model delivers cinematic, multi‑scene narratives about 30% faster than rivals like Kuaishou’s Kling, triggering a rally in China’s AI application stocks. ([finance.sina.com.cn](https://finance.sina.com.cn/roll/2026-02-09/doc-inhmesxr9395387.shtml))
Single‑prompt AI spins 16‑minute anime episodes
Hong Kong–based LAiPIC announced on February 8, 2026 that its Doratoon platform can automatically generate up to 16 minutes of continuous, story‑driven anime from a single text prompt. The company says Doratoon uses a proprietary visual intelligence engine and a library of 18 million assets to handle scripting, storyboarding, character design, scene rendering, voice acting and music with minimal human input.
Flux 2 small puts open image models on one GPU
German startup Black Forest Labs released Flux 2 small on January 17, 2026, a pair of image models (9B and 4B parameters) that combine text‑to‑image, editing and multi‑reference generation in a compact architecture. The 4B model runs on consumer GPUs like the RTX 3090 with around 13 GB of VRAM and is licensed under Apache 2.0 for commercial use, while the larger 9B model is non‑commercial.