AlgoMap.io

Video compression is the process of reducing the file size of a video so it can be streamed easily over the internet.

Raw video files are massive because they contain up to sixty full-color images for every single second of footage.

To shrink this data compression algorithms exploit two types of redundancy known as spatial and temporal.

Spatial compression looks for redundancy within a single individual frame of the video.

If a frame shows a clear blue sky the algorithm does not save the color data for every single pixel.

Instead it groups those pixels together and records a single instruction to fill that entire area with blue.

This process is highly efficient for static backgrounds and works similarly to how a JPEG image is compressed.

Temporal compression looks for redundancy across multiple sequential frames over a period of time.

In most video scenes very little actually changes from one frame to the next millisecond.

If a person is walking in front of a house the house stays completely still while only the person moves.

Instead of saving a whole new picture for the next frame the algorithm only records the differences or changes.

It saves the moving parts as motion vectors and instructs the player to reuse the static background from the previous frame.

The algorithm organizes these frames into a structure called a Group of Pictures which includes I-frames P-frames and B-frames.

I-frames are full images that contain all the spatial data needed to render a complete picture without any outside context.

P-frames and B-frames are smaller predictive frames that only contain the temporal data of what changed since the last frame.

Combining spatial and temporal compression allows platforms to shrink video file sizes by over ninety-five percent without a noticeable loss in quality.

Adaptive streaming allows video platforms like YouTube to deliver smooth playback regardless of network speed or device type.

When a creator uploads a video the system immediately kicks off a background process called transcoding.

Transcoding compresses the original massive raw video file into a highly optimized digital format.

The system does not just create one optimized file it duplicates the video into multiple different resolutions.

It generates separate versions for 144p 360p 720p 1080p and even 4K quality levels.

Each of these resolution versions is then sliced up into tiny video segments that are only a few seconds long.

A manifest file is created to act as an index map for all these different resolutions and segments.

When a user watches a video the video player reads this manifest file to understand what streams are available.

The player constantly monitors the user's live network bandwidth and device CPU usage.

If the internet speed is fast the player requests the high-resolution 1080p segments.

If the internet connection suddenly drops the player automatically switches to fetch the lower-resolution 480p segments instead.

This seamless switching prevents the video from buffering and ensures the playback never freezes for the user.

Common protocols used to manage this adaptive switching include HLS by Apple and MPEG-DASH.

Geohashing is a hierarchical spatial data structure that encodes a geographic location into a short string of letters and digits.

It works by dividing the entire surface of the Earth into a grid of interlocking bounding boxes.

The first character of a geohash determines which of the primary global regions the coordinate falls into.

Each additional character added to the string divides that specific box into a smaller more precise grid.

A longer geohash string means a smaller box and a much more precise geographic location.

For example a geohash with an exact length of twelve characters is accurate down to a few centimeters.

This system converts two-dimensional latitude and longitude coordinates into a single one-dimensional string.

Because it is a single string databases can index and search geohashed locations incredibly fast.

Nearby locations often share the exact same prefix string which makes searching for local points of interest very efficient.

However it can suffer from edge cases where two points are close physically but sit on different sides of a major grid boundary.

Despite this limitation geohashing is a foundational technique used by location-based apps to find nearby drivers or restaurants.

Auto-increment IDs cause major challenges when scaling a system across multiple distributed databases.

In a single database auto-incrementing a number by one for each new record is simple and consistent.

When you distribute the data across multiple servers they can no longer easily coordinate the next number.

One solution is to configure each server with a unique starting offset and a fixed increment step.

For example with three servers the first starts at one the second at two and the third at three.

They all increment by three so the first server generates one four and seven while the second generates two five and eight.

This approach works initially but becomes highly complex when you need to add or remove servers.

Adding a fourth server requires recalculating the offset and increment step for every single machine in the cluster.

Changing these configurations on live databases without causing ID collisions or downtime is incredibly difficult.

To avoid this complexity some architectures use a centralized ticket server to hand out IDs.

The ticket server acts as a single source of truth that increments a single counter for the entire system.

However this centralized ticket server reintroduces a single point of failure into the architecture.

It also creates a massive performance bottleneck which completely defeats the purpose of having a distributed system.

This fundamental trade-off is exactly why companies like Twitter designed custom solutions like Snowflake IDs.

There's an order to learning Data Structures and Algorithms!

Twitter Snowflake is a distributed unique ID generator system created by Twitter.

It is designed to generate highly scalable and unique identifiers without using a centralized coordinator.

The system outputs a 64-bit integer instead of a massive 128-bit random UUID string.

The first bit of the 64-bit integer is always set to zero to ensure the number remains positive.

The next 41 bits encode a timestamp in milliseconds based on a custom defined epoch.

This time component allows the generated IDs to be naturally sortable in chronological order.

The next 10 bits represent the machine identifier which allows up to 1024 unique server nodes.

The final 12 bits represent a local sequence counter that increments for requests on that specific machine.

This counter resets every millisecond to allow a single machine to generate 4096 unique IDs per millisecond.

By combining these components distributed servers can generate unique IDs concurrently without any network collisions.

Master DSA for FREE at Algomap-io!

Chaos engineering is the practice of intentionally introducing failures into a system to test its resilience.

It involves conducting controlled experiments to see how software handles unexpected disruptions in production.

Instead of waiting for a crash to happen engineers cause things to fail on purpose to find hidden weaknesses.

Common experiments include shutting down server nodes injecting network latency or killing database connections.

The goal is to prove that the system can automatically recover from these issues without affecting the user.

It shifts the focus from hoping a system will not fail to ensuring it can survive failure gracefully.

This practice is essential for massive distributed systems where hardware or network errors are guaranteed to happen.

A famous tool for this is Chaos Monkey which was created by Netflix to randomly terminate production servers.

The insights gained from these experiments are used to fix vulnerabilities before they cause a real outage.

It builds deep confidence in the reliability and self-healing capabilities of a company's infrastructure.

Regression testing is the process of verifying that recent code changes have not broken existing features.

It ensures that fixing a bug or adding a new capability does not accidentally introduce new issues elsewhere.

This type of testing is critical because software components in large systems are heavily interconnected.

A regression test suite is typically a collection of previously written unit integration and system tests.

The suite is executed every time a new version of the software is built or deployed.

Automation is essential for regression testing because running these tests manually for every update is too slow.

Continuous Integration pipelines automatically trigger these tests to provide fast feedback to developers.

If a regression test fails it means a change has caused the system to take a step backward.

Catching these regressions early prevents broken code from ever reaching the end user in production.

It provides teams with the confidence to update and optimize code without fear of breaking the application.

Integration testing is the phase where individual software modules are combined and tested as a group.

It focuses on the interfaces and the flow of data between different parts of the system.

These tests ensure that a unit of code still works correctly when it interacts with a database or an external API.

Integration tests catch bugs that unit tests miss like incorrect database queries or broken network connections.

They are generally slower than unit tests because they involve actual input and output operations.

Developers often use a dedicated test environment that mimics the production setup to run these tests.

Common strategies include Top-Down Bottom-Up or the Big Bang approach to connecting modules.

Successful integration testing proves that the various components of an architecture can coexist and communicate.

It serves as the critical bridge between testing isolated logic and testing the entire user journey.

Unit testing is the process of verifying the smallest functional parts of an application.

A unit is typically a single function or a method within a specific class.

These tests are executed in isolation from the rest of the system and its dependencies.

Mocking and stubbing are used to simulate external factors like databases or web services.

This isolation ensures that a failure in the test is definitely caused by the specific code being tested.

Unit tests are written by developers as they write the actual application code.

Because they do not rely on a network or a disk they run in milliseconds.

A large suite of thousands of unit tests can be completed in just a few seconds.

They provide an immediate safety net that prevents old code from breaking when new changes are added.

High unit test coverage is a primary indicator of a maintainable and stable codebase.