Audio Face Animator documentation

Audio Face Animator turns speech into facial animation for characters with ARKit-compatible blendshapes. It uses the NVIDIA Audio2Face-3D model and runs locally, with CPU generation available by default.

A performance can carry an emotion, the eyes can be driven from the rig’s own eye bones, and the character blinks on its own. Individual ARKit shapes can drive several blendshapes or a bone, so rigs whose jaw is a bone are supported too.

Use preflight to check the selected backend before generating. CPU checks do not require an NVIDIA GPU or build a TensorRT engine.

Requirements

Built-in, URP and HDRP are all fine — the plugin drives blendshape weights and transforms, and never touches materials or shaders.

Versions Comparison: Lite vs Full

The Lite version has three generation limitations:

• CPU is the only backend. Full also offers NVIDIA GPU.
• Generation uses only the first five seconds of audio.
• Generation works in the Unity Editor only. Baked AnimationClips still play in Player builds.

Everything else — mapping, emotions, eyes, blinking, AnimationClip export — is the same in both.

If you own the full version and still see a five-second cap or an orange ⚠️ LITE VERSION banner on the Face Animator, the model files did not arrive intact; see Troubleshooting. The preflight check reports which edition is installed, and says when it cannot tell yet because the sync has not run.

Configuration

Sync the model files, choose Checks > CPU in preflight, then generate in the sample scene. Full users who explicitly select NVIDIA GPU must also prepare a TensorRT engine for their GPU.

Both editions add Preflight Check, Sync StreamingAssets and Documentation under Tools → AudioFaceAnimator. Full also adds Setup TensorRT Engine.

Preflight check

Tools → AudioFaceAnimator → Preflight Check. It opens automatically at Editor startup when common platform or model checks fail, including a fresh import whose model files have not been synced.

Automatic startup checks common platform and model requirements. Select Checks > CPU for CPU libraries and model readiness; in Full, select Checks > NVIDIA GPU separately for driver, GPU memory and TensorRT engine. Opening CPU diagnostics does not load the ONNX session.

Before the first CPU generation, CPU operation can show UNKNOWN with “No CPU operation has completed in this domain.” This is operation history, not a failed readiness check; the headline above can still say Ready to generate animation.

Generation needs Windows x64. A GPU below compute capability 7.5 blocks only the NVIDIA GPU backend; Full users can explicitly select CPU.

Re-check runs the selected checks again. Copy report puts the selected backend, edition and displayed results on the clipboard — which is what to paste into a support message. Two ticks sit at the bottom: Explain the checks that passed as well adds the explanation paragraph under the lines that are fine too, not only under the ones that are not, and Do not open this window automatically stops it opening on its own — except when the model files need syncing, which it always reports.

1. Sync StreamingAssets

Tools → AudioFaceAnimator → Sync StreamingAssets, then Copy All Files. It copies the model data, so give it time to finish. Run it again after every plugin update: until you do, generation keeps using the old model data without saying anything, which is why the preflight treats out-of-date files as a failure and opens itself to tell you.

This copies the model files to Assets/StreamingAssets/DevEloop/AudioFaceAnimator, which is where Unity looks for them at runtime. The window compares file content, not timestamps, so an updated model file shows up as a mismatch and a damaged one is repaired by copying again. No Differences means there is nothing to do.

2. Setup TensorRT engine (Full, NVIDIA GPU only)

CPU users can skip this step. Full’s GPU backend requires a separate TensorRT 10.16.1.11 Windows x64 / CUDA 12.9 installation. No TensorRT DLLs are included in the asset; CUDA DLLs are bundled.

1. Open Tools → AudioFaceAnimator → Setup TensorRT Engine and press Download TensorRT, or use the NVIDIA download page. Sign in to NVIDIA and select the exact version above, not TensorRT for RTX or a newer release.
2. Extract the whole ZIP, for example to C:\Tools\TensorRT-10.16.1.11. Keep all builder resources, not only nvinfer_10.dll.
3. Open Windows Edit environment variables for your account, then Path → Edit → New and add C:\Tools\TensorRT-10.16.1.11\bin. Preserve the existing entries.
4. Close the Unity Editor and fully exit Unity Hub, including its tray icon. Reopen Hub and the project to pick up the new PATH.
5. Press Check Installation in the setup window. Resolve any reported missing, incompatible or conflicting files before building the engine.

The plugin does not download dependencies or edit PATH for you. Python, pip and a separate CUDA Toolkit are not required. Once installed, generation runs offline.

Expand Manual installation instructions for the setup steps. This example was captured on a machine where TensorRT was already installed.

Tools → AudioFaceAnimator → Setup TensorRT Engine, then Build Engine for This GPU — the preflight report links straight to it. Building can take a minute or longer, depending on the machine.

The engine is compiled from the network.onnx synced in step 1, because a compiled engine only loads on the GPU architecture it was built for. Leave Build the full batch range declared by the model off — it doubles the engine size and reserves about 2 GB more GPU memory for a capability the plugin never uses.

The result is cached at

%USERPROFILE%\AppData\LocalLow\<company>\<product>\DevEloop\AudioFaceAnimator\engine\<gpu-signature>\

and reused on every later run. The cache key is your GPU’s signature plus a fingerprint of network.onnx, so the engine survives plugin updates but is rebuilt — correctly — after a GPU or driver change, or when the model itself changes. Clear Built Engine Cache in the same window forces a rebuild.

In a standalone build the first GPU generation call builds the engine on each player’s machine, after the same external TensorRT installation is available.

Preparing your character

Two components: a Face Blends Mapper on each face renderer, and one Face Animator on the character that drives them.

1. Face Blends Mapper

Add it and assign your face SkinnedMeshRenderer to Target Renderer.

It immediately creates all 52 ARKit rows and matches them against the mesh’s blendshapes — ignoring case, spaces and underscores, and accepting prefixed names like CC_Base_Body.jawOpen. The console reports the result, naming every row it could not match.

Add one mapper per face renderer (head, teeth, tongue and eyes are often separate). The three shipped prefabs carry five each.

Matching is deliberately generous, so read down the list once. A Filter box narrows the 52 rows while you work, and each row expands to show everything it drives:

• Blendshapes — one or more mesh blendshapes, each with its own max weight (0–100, default 100). Max weight is both scale and ceiling: 50 halves the shape’s travel. This is the tuning control — lower jawOpen if the mouth opens too far. Use Add blendshape target for a rig that splits one ARKit shape across several shapes, or needs a corrective; the collapsed row then shows a +N badge.
• Bones — see below.
• Preview — a 0–1 slider that drives the row live in the Scene view, without generating anything. The quickest proof that a row is wired correctly. Collapsing the row, or selecting another object, returns the face to rest.

Auto-Map (fill empty) touches only unmapped rows; Auto-Map (overwrite all) redoes everything, discarding manual edits. Several rows may point at the same mesh blendshape — their contributions are summed and clamped, so nothing is lost.

jawOpen, mouthClose, mouthFunnel and mouthPucker are the rows speech cannot do without. Check those first.

Driving a bone instead of a blendshape

For a rig whose jaw, lips or brows are bones, expand the row and press Add bone target.

Assign the Bone and its current local pose is captured as the rest pose immediately — so do this with the rig neutral, or press Re-capture rest afterwards. Then enter the pose the expression should reach, either by typing into Rotation (deg) delta and Position offset, or by posing the bone in the Scene view and pressing Capture delta from current pose. Influence scales the whole target. Verify with the row’s Preview slider: 0 is rest, 1 is the captured pose.

Three rules that will otherwise bite:

• A bone may appear in only one mapper. Several rows within one mapper may share a bone and accumulate; two mappers claiming the same bone is an error.
• Bones must be descendants of the Face Animator’s transform or their curves are skipped when baking.
• Eye bones belong to the Eyes Rotation Mapper, not here. A bone claimed by both is an error.

Reusing a mapping

Press Save As… in the mapper’s Preset row to write a FaceBlendsPreset asset, then drop it into the Preset field on the next character and press Load. Save overwrites the selected preset.

A preset stores blendshape and bone names, not object references. Blendshape names must match the new mesh exactly — loading a preset does not re-run the fuzzy auto-map — and bones are resolved by searching the transforms under the mapper’s own GameObject, so bone targets only transfer when the mapper sits inside the rig. The console reports exactly what was applied and what it could not find.

2. Face Animator

Add it to the character root and drag every mapper into Blendshapes Configuration → Blends Mappers. Any renderer whose mapper is not listed will not move.

Generate animation in Editor

Model Type — which Audio2Face-3D model generates the animation: Claire, James or Mark. Each was trained on a different performer and produces a different speaking style. This is a performance style, not a character: it has nothing to do with what your character looks like, and Claire will drive a male character perfectly well. Try all three on the same audio.

Audio Clip, under Animation Playback — the speech to animate. Full has no edition length cap; Lite generates only the first five seconds. Mono or stereo clips at supported sample rates are resampled to 16 kHz mono. No other preparation is needed, though a clip whose import Load Type prevents AudioClip.GetData from returning samples cannot be read.

Keep CPU selected in Generation > Backend. Full can explicitly choose NVIDIA GPU; Lite offers CPU only. An older Lite scene saved with NVIDIA GPU selected must be switched to CPU. Then enter Play mode — outside it the inspector says so and Generate is disabled — press Generate, and press Play to preview the result. Stop ends it early and resets the face to neutral. Generation runs on a background thread, so the Editor stays usable.

The first CPU generation loads the ONNX model. The sample scene’s Prewarm button can do that ahead of time; Release CPU frees the shared context after queued generation finishes. Completed animations remain playable. NVIDIA GPU in Full may build a TensorRT engine on its first run. Changing the AudioClip or other generation inputs invalidates the generated result; generate again.

Saving an AnimationClip

Once animation exists, a Create AnimationClip button appears. It writes a standard Unity AnimationClip, which must be saved inside your project’s Assets folder. The button is only there while generated data exists, which means before you leave Play mode — leaving discards it.

What lands in the clip: one curve per mapped blendshape per renderer, rotation curves for every driven bone (plus position curves only for bones some row actually offsets), the procedural blink, and the eye rotations if an Eyes Rotation Mapper is assigned. Tangents are smoothed and the clip is a normal non-legacy clip.

It has no dependency on the plugin, the SDK or an NVIDIA GPU, so it plays from an Animator or Timeline on any platform Unity builds for. Three things to know:

• Curve paths are relative to the GameObject holding the Face Animator, so moving that component to a different level of the hierarchy invalidates the clip. Put it on the character root and keep it there.
• Weights are baked in when the curves are written. Changing a mapper weight afterwards means generating and saving again.
• Set a non-zero blink seed before baking a set of clips, or every bake blinks at different moments and re-baking one line makes it inconsistent with the rest.

The sample scene

Assets/DevEloop/AudioFaceAnimator/Scenes/AudioFaceAnimator.unity has all three characters set up, a model selector, generation controls and a file browser for loading a WAV from disk. It is the quickest way to confirm the plugin works on your machine, and a working reference for wiring the API to UI.

For the sample’s buttons, install Input System (com.unity.inputsystem 1.14.2) through Window → Package Manager. Without it, the scene’s EventSystem has a missing UI component and buttons may not respond.

Input System 1.14.2 in Package Manager. The capture also shows a package signature warning from this Editor; it does not demonstrate a clean signature check. Generation through the Face Animator Inspector or API works independently of this sample UI dependency.

Generation processes a complete audio clip. Streaming audio generation is not supported. CPU and GPU use different solvers, so their facial poses can differ.

Emotions

Without a preset the model is driven with a neutral emotion, which reads as a competent but flat performance. An Emotion Preset colours it.

Press Create… next to the Emotions field on the Face Animator: it saves a new preset asset and assigns it in one step. An existing preset can be dropped into the same field, and presets can also be made from Assets → Create → DevEloop → Audio Face Animator → Emotion Preset.

A preset is an ordered list of segments, read left to right across the clip:

Share is how much of the clip the segment occupies, not how strong it is. Shares are normalised, so they do not have to add up to anything and the same preset fits a clip of any length — the example above gives 10%, 30%, 40% and 20% of whatever clip you generate from, and writing it as 1, 3, 4, 2 means exactly the same thing. Normalize Shares rewrites them as fractions of one, which changes the numbers and not the result.

Intensity is the value handed to the model, 0 to 1. Start around 0.5 and raise it until the performance reads the way you want; 1 is the strongest the model accepts, and anything above is clamped.

Crossfade is how long one segment takes to blend into the next, in seconds (0–2, default 0.25). It is applied symmetrically around each boundary and clamped per boundary, so a segment is never swallowed by its own transitions; a segment too short to hold anything becomes a brief peak instead. Raise it if transitions pop.

The coloured bar under the list previews the result — each slice shows its emotion, its percentage and its duration against Preview Clip (s), which only scales the seconds shown. The preset itself is never tied to a particular clip.

The available emotions come from the model, and are listed in network_info.json: amazement, anger, cheekiness, disgust, fear, grief, joy, outofbreath, pain and sadness. A segment carries one emotion at a time; an emotional rise and fall is built from several segments, as fear is above. Four ready-made presets ship in Assets/DevEloop/AudioFaceAnimator/EmotionPresets/.

Emotions are a generation-time input, so assign the preset before generating:

faceAnimator.Emotions = myEmotionPreset;   // null goes back to neutral
await faceAnimator.GenerateAsync();

Eyes and blinking

The model drives the face but produces almost no eyelid motion, and its eye rotations need bones to apply them to. Both gaps are filled inside Unity.

Blinking

Generated in Unity and mixed over the model’s output, both during playback and at bake time. It is on by default in the Blinking section of the Face Animator, and needs eyeBlinkLeft and eyeBlinkRight mapped to real blendshapes — without them there is no blink and no warning either. eyeSquint* and eyeWide* are used as well when mapped, so the lower lid follows and a wide eye does not fight a closed lid.

Blink weights pass through the mapper like any other weight, so a blink shape with max weight 50 blinks only halfway. Tune the amplitude or the ceiling, not both at once.

Eye rotation and gaze

Add an Eyes Rotation Mapper component and assign it to the Face Animator’s Eyes field. With that field empty the eyes are simply skipped, and nothing is logged — it is the most common reason for “the eyes still don’t move”.

Point Right Eye → Bone and Left Eye → Bone at the two eye bones, with the rig at rest; each bone’s local rotation is captured as its rest pose on assignment. Then use the component’s Preview pitch and yaw sliders to calibrate: the mapper rotates each bone about a Pitch Axis and a Yaw Axis in its own local space, and rig conventions differ. If an eye rolls instead of turning, change the axis; if it moves the right way but reversed, set that eye’s Pitch Scale or Yaw Scale to −1. Both eyes have independent axes and scales, which is what a mirrored rig needs. Stop preview returns them to rest.

The eye bones must not appear in any Face Blends Mapper, and both blink and eye rotation are included when you press Create AnimationClip. One limitation there: a Look At target is sampled once, at bake time, at its current position — a target that moves at runtime cannot be baked, so either keep the plugin’s own playback or clear the target and drive eye contact yourself.

Tuning the solver

When the animation is close but too soft, too strong or too twitchy, tick Model Configuration → Override Model Config. The fields are filled from model_config_<Model Type>.json in StreamingAssets and start being sent to the engine, so generation continues unchanged until you actually move something. Change one parameter at a time and regenerate.

Reset to Model Defaults reloads the values for the current Model Type; unticking the checkbox hands control back to the engine entirely. Values are clamped to the ranges the engine accepts, with a warning if a hand-edited JSON file is out of range.

Note the difference between this and a mapper’s max weight: these parameters change the whole performance, a max weight fixes one expression on one rig.

Generate animation at runtime

Runtime generation in a standalone player is a full-version feature; Lite generates in the Editor only. Full generates on CPU by default in Windows x64 Mono and IL2CPP Players. Baked clips can play on other Unity platforms.

using DevEloop.AudioFaceAnimator;
using UnityEngine;

public class Example : MonoBehaviour
{
    public FaceAnimator faceAnimator;
    public AudioClip clip;

    public async void Speak()
    {
        if (faceAnimator == null || clip == null || faceAnimator.IsGenerating)
            return;

        faceAnimator.Backend = AnimationBackend.Cpu;
        faceAnimator.AudioClip = clip;
        await faceAnimator.GenerateAsync();

        if (!faceAnimator.HasAnimationData)
            return;          // generation failed; the console says why

        faceAnimator.Play();
    }
}

GenerateAsync() does not throw for ordinary failures — a missing library or model file, or an unavailable selected backend. It logs the reason and leaves HasAnimationData false, so check that rather than catching exceptions. Use IsGenerating to disable your UI while it runs, as the sample scene does, and do not start a second generation while one is running.

For audio that is not an asset in your project — a file the player picks, or one downloaded — the plugin includes a WAV loader, which returns null and logs on failure:

AudioClip clip = await AudioUtils.LoadWavClipAsync(path);   // local path or URL
if (clip != null)
    faceAnimator.AudioClip = clip;

FaceAnimator adds an AudioSource in Awake if the object has none, so a character instantiated at runtime needs no extra setup for playback.

Shipping a standalone build

Run Sync StreamingAssets before building — the plugin reads Application.streamingAssetsPath, and the copy inside the package folder is a source, not what ships. Include audio2face-cpu.dll, onnxruntime.dll and the model files for CPU. Full also includes its NVIDIA GPU DLLs. Check the Windows x64 plugin importers and the built Player’s Plugins/x86_64 directory.

To prepare CPU generation ahead of a line, use AnimationGenerator from an async method entered on Unity’s main thread:

var generator = new AnimationGenerator();
if (!await generator.PrewarmAsync(AnimationBackend.Cpu))
    return; // See the Unity console for the failure.

faceAnimator.Backend = AnimationBackend.Cpu;
await faceAnimator.GenerateAsync();
if (faceAnimator.HasAnimationData)
    faceAnimator.Play();

// Optional when generation is finished; affects the shared CPU context.
await generator.ReleaseCpuResourcesAsync();

Release waits for preceding queued operations. Completed animations remain playable; a later generation loads the context again. CPU runs on machines without NVIDIA GPU, CUDA or TensorRT.

The engine advice below applies only when Full explicitly selects NVIDIA GPU. Each target machine also needs the external TensorRT installation. For version 2.0.0, keep the GPU Player and model paths ASCII-only: a GPU native crash was reproduced with a path containing Unicode characters.

Keep network.onnx in StreamingAssets and let the first generation call build an engine on each player’s machine. Your players’ GPUs are not yours, and the engine built on your machine will not load on most of theirs. There is no way to build one engine covering many architectures, so this is the only approach that works everywhere. The build takes about a minute, happens once per machine, and is cached under Application.persistentDataPath, which stays writable even when the game is installed somewhere read-only.

A minute of silence looks like a hang, so pre-warm the engine behind a loading screen of your own rather than letting it land on the player’s first line:

EngineStatus status = EngineProvisioner.GetStatus(out string detail);

if (status == EngineStatus.Unavailable)
    return;                        // no qualifying GPU; log `detail` and fall back

if (status == EngineStatus.NeedsBuild)
    await Task.Run(() => EngineProvisioner.EnsureEngine(false, Report, out string error));

// Called on the worker thread: store the values, render them on the main thread.
// Return false to cancel a build in progress.
bool Report(string phase, float progress) { Phase = phase; Progress = progress; return true; }

EnsureEngine blocks for about a minute, so it belongs on a worker thread. GetStatus checks NVIDIA GPU only. If it is unavailable, explicitly choose CPU or play an already baked AnimationClip. Baked clips play on any platform Unity supports, without native generation libraries.

PluginVersion.GetPackageState(out error) identifies the edition without loading a backend. PluginVersion.GetState(AnimationBackend.Cpu, out error) checks CPU library/package availability; the no-argument GetState() also checks CPU. Neither prepares an ONNX session or proves generation will succeed. Pass AnimationBackend.Nvidia to check NVIDIA libraries/package state, and use EngineProvisioner.GetStatus for GPU/engine readiness:

PluginVersion.Version is the shipped version string. The older IsModelComplete() is kept for compatibility; new code should call GetPackageState or GetState(backend, out error) as appropriate.

You can additionally ship a network.trt in Assets/DevEloop/AudioFaceAnimator/StreamingAssets, copied out of your engine cache. Players whose GPU it fits skip the build; everyone else falls back to the ONNX. It only helps players on the same GPU architecture as the machine that built it, so treat it as a minor optimisation, never a substitute.

If your game only needs a fixed set of lines, you probably want none of this in the build. Generate in the Editor, save AnimationClips, and leave the plugin’s StreamingAssets out — the clips carry no dependency on it, and local inference is not small.

Skills for AI coding assistants

Seven skills ship inside the package, under Assets/DevEloop/AudioFaceAnimator/AI/Skills/. Each is a plain SKILL.md describing one area of the plugin in the terms an assistant needs in order to act on it — the real component and API names, what each field does, and the failures worth checking first.

Point your assistant at that folder, or copy the skills into wherever it reads them from. They are ordinary Markdown with YAML frontmatter and carry no dependency on any particular tool, and both editions ship the same set.

Version 2.0.0 documentation correction: some instructions bundled in the package still describe TensorRT as included and omit the sample’s Input System dependency. Follow this page for those requirements: TensorRT is installed separately for Full GPU generation, and Input System is needed for the sample UI.

Worth doing because an assistant working from the plugin’s name alone will invent an API that reads plausibly and does not exist. These describe what is actually there.

Troubleshooting

Open the preflight check, select the backend used by the component and read its failed gate. If every line there passes, open the Unity console. Every failure reports the actual reason, including what TensorRT itself said, and failures print a specific line followed by a generic one — the first is the useful one.

Setup and the selected backend

Mapping and playback

Emotions, eyes and blinking

Diagnostic log

For CPU, use Unity Console, Editor.log or Player.log, plus Checks > CPU → Copy report for the latest CPU operation, timings and error.

The file logger below is for NVIDIA GPU in Full. It captures native SDK output that is not forwarded to the Unity console and is off by default.

In the Editor, open Tools → AudioFaceAnimator → Setup TensorRT Engine and expand Diagnostic Log. Tick Write a log file and pick a Level — Debug logs every step, Info the main ones, Error only failures. Change… moves the file, Show opens the folder, and the setting is remembered between sessions and re-applied after every recompile, in append mode — so turn it off again when you are done.

From code, called on the main thread before generation starts:

using DevEloop.AudioFaceAnimator;

NativeLog.Enable(NativeLog.DefaultFilePath, NativeLogLevel.Debug);
// ... generate ...
NativeLog.Disable();

The default file is %USERPROFILE%\AppData\LocalLow\<company>\<product>\DevEloop\AudioFaceAnimator\audio-face-animator.log. Every line is flushed as it is written, so the log survives a crash. Unity’s Player.log also carries the plugin’s managed logs, including engine-build progress.

Support

Press Copy report in the preflight check and paste the result into your message after selecting the affected backend. It carries the plugin and Unity versions, installed edition and displayed checks. CPU includes library information and the latest operation; NVIDIA GPU includes GPU, driver and engine information. Add the full console output alongside it — the messages identify the failure precisely, which makes most issues diagnosable at a glance. If neither is conclusive, turn on the diagnostic log, reproduce the failure and attach the file.

dev.eloop@outlook.com

← Back to Audio Face Animator