From Text to Video: Automating a Short-Form Video Pipeline with Python

Short-form video content dominates social media, but producing it manually is repetitive. Write a script, record audio, find footage, add captions, export, repeat. I built a Flask-based API that automates most of this pipeline: given a text script, it produces a ready-to-publish vertical video with synchronized captions.

This is not a tool that replaces creative judgment. It is a tool that automates the mechanical steps so the creative effort can focus on the content itself.

The Pipeline

The generation process has several stages:

1. Text validation and normalization: Clean the input, estimate word count and duration
2. Text-to-speech generation: Convert the script to an audio file using a TTS engine
3. Background footage selection: Choose or retrieve a background video
4. Caption generation: Create timed captions synchronized with the audio
5. Video composition: Combine audio, footage and captions into a final MP4
6. Cleanup: Remove temporary files

Each stage is independent enough to be tested and debugged separately, which turned out to be important when things go wrong (and they do go wrong).

Text Processing

The input text goes through normalization: stripping extra whitespace, handling special characters, and segmenting into sentences. The sentence segmentation is critical because it determines where captions break.

Duration estimation uses word count and an average speaking rate. This estimate drives the length of the background footage and the timing of the captions. The estimate is not perfect -- pauses, emphasis, and speaking speed vary -- but it is close enough for the synchronization to work.

def estimate_duration(text: str, words_per_minute: int = 150) -> float:
    word_count = len(text.split())
    return (word_count / words_per_minute) * 60

Text-to-Speech

The TTS engine converts the processed text into an audio file. The system supports multiple TTS backends, with a fallback mechanism if the primary service is unavailable. The audio file is the timing reference for everything else: captions are synchronized to the audio, not to an arbitrary clock.

The generated audio is saved as a temporary file and passed to the next stages of the pipeline.

Background Footage

The background video is selected from a pool of pre-approved footage. The system maintains this pool to avoid downloading or generating footage on every request, which would be slow and unreliable.

The footage is trimmed (or looped) to match the estimated duration of the audio. For vertical short-form content, the footage is expected to be in portrait orientation (9:16 aspect ratio). The system does not crop or resize footage automatically; it expects appropriately formatted input.

Caption Generation

Captions are generated by splitting the text into segments and distributing them across the video timeline based on the audio duration. Each caption segment has a start time, end time, and text content.

The synchronization is approximate. Without phoneme-level alignment (which would require a more sophisticated pipeline), the timing is based on word count ratios. A sentence that represents 20% of the total word count gets roughly 20% of the total duration.

def generate_captions(text: str, total_duration: float) -> list[Caption]:
    sentences = split_sentences(text)
    total_words = sum(len(s.split()) for s in sentences)

    captions = []
    current_time = 0.0

    for sentence in sentences:
        word_ratio = len(sentence.split()) / total_words
        duration = total_duration * word_ratio
        captions.append(Caption(
            text=sentence,
            start=current_time,
            end=current_time + duration,
        ))
        current_time += duration

    return captions

The captions are rendered onto the video using Pillow for text layout and MoviePy for compositing.

Video Composition with MoviePy

MoviePy handles the final composition: combining the background footage, the audio track, and the caption overlays into a single video file. Under the hood, MoviePy uses FFmpeg for the actual encoding.

The composition process:

1. Load the background footage
2. Trim or loop it to match the audio duration
3. Set the audio track
4. Apply caption overlays at their designated timestamps
5. Export as MP4 with appropriate encoding settings

MoviePy simplifies FFmpeg's complexity, but it also inherits some of its quirks. Temporary files accumulate quickly, and memory usage can spike with long videos. The system handles this by cleaning up temporary files after each generation and limiting the maximum input length.

Temporary File Management

Video generation creates a lot of temporary files: the audio file, individual caption frames, intermediate video segments, and the final output. If these are not cleaned up, disk space fills up quickly.

The system uses a temporary directory per generation request. After the final video is produced (or if the generation fails), the entire directory is deleted. This prevents file leaks even when errors occur mid-pipeline.

The Reusable Video Pool

To reduce generation time, the system maintains a pool of pre-downloaded background videos. Instead of fetching new footage for each request, it selects from the pool. This trades storage space for speed and reliability.

A background worker periodically refreshes the pool, removing old videos and adding new ones. This keeps the content fresh without blocking the generation pipeline.

Error Handling

Every stage of the pipeline can fail: the TTS service can be unavailable, the background footage can be corrupted, FFmpeg can crash on malformed input. The system wraps each stage in error handling that produces meaningful messages and cleans up resources.

The most common failure is the TTS service timing out. The fallback mechanism switches to a local TTS engine (lower quality, but functional) when the primary service does not respond within a configurable timeout.

Performance Characteristics

Video generation is CPU-intensive. The FFmpeg encoding step dominates the processing time, especially for longer videos. On a typical server, a 30-second video takes 15-30 seconds to generate, depending on the complexity of the captions and the encoding settings.

The system is not designed for real-time generation. It is designed for batch processing: submit a script, get a video a minute later. For use cases where immediate output is required, a different architecture (pre-rendered templates, real-time compositing) would be more appropriate.

What I Learned

Synchronization is harder than it looks. Matching captions to audio without precise timing data requires approximations. The results are good enough for most content, but they are not frame-perfect.

FFmpeg is powerful but fragile. It handles almost any video format, but it fails in unpredictable ways when inputs are malformed. Robust FFmpeg integration means handling a long tail of edge cases.

Cleanup is as important as generation. Temporary files, failed processes, and interrupted requests can leave the system in a dirty state. Explicit cleanup at every stage prevents disk exhaustion.

This pipeline does one thing: convert text to captioned video. It does not upload to social media, it does not schedule posts, it does not analyze engagement. Those are separate concerns, and keeping them separate makes the system easier to maintain and extend.