OSINTThe Double Edge of Seeing the World in Real Time

Continuous Social Media Scrapers and Live Feeds: SOCMINT

Social media created something historically unusual: billions of people voluntarily publishing observations about their lives, surroundings, opinions, relationships, and activities.

SOCMINT, or social media intelligence, is the systematic collection and analysis of information derived from social-media platforms. David Omand, Jamie Bartlett, and Carl Miller gave the term prominence in their 2012 article, “Introducing Social Media Intelligence,” which examined both its potential value and the methodological and ethical conditions needed for its legitimate use.² This discussion concerns publicly or commercially available social-media information; access to private accounts and restricted conversations raises separate questions.

A continuous social-media scraper takes this one step further. Instead of a researcher manually reading individual posts, software can continually collect publicly accessible posts, hashtags, images, videos, timestamps, links, and other information. A live feed can then display emerging subjects, geographic concentrations, sudden changes in activity, or relationships among accounts.

The Good

This can provide an extraordinary early-warning system. During a hurricane, earthquake, wildfire, terrorist attack, or industrial accident, people near the event may begin posting photographs and observations before traditional news organizations arrive. Emergency agencies can potentially identify blocked roads, damaged buildings, missing people, or communities requiring assistance.

Journalists and human-rights investigators can use the same material to document events that governments or armed groups would prefer to conceal. Open-source investigators have demonstrated that social-media photographs and videos can help document conflicts, identify misinformation, reconstruct events, and investigate alleged human-rights abuses.³

SOCMINT can also reveal broader patterns without identifying particular individuals. Researchers may observe the spread of rumors, changing public attitudes, emerging scams, coordinated propaganda, or warnings about dangerous products.

In these applications, millions of individual observations become something resembling a distributed global sensor network.

The Bad

The same system can become mass surveillance. Something being publicly visible does not necessarily mean that the person who posted it expected it to be permanently collected, indexed, combined with other information, analyzed by machines, and retained indefinitely.

A person who makes ten harmless public posts may reveal very little in each one individually. Combine those posts with photographs, friends, workplaces, travel patterns, timestamps, and years of archived activity, however, and the resulting profile can become surprisingly intimate.

The distinction is between being observable in a particular moment and having one’s activities systematically assembled into a lasting record.

SOCMINT also has a problem with the quality of its evidence. Social media does not provide an unbiased sample of reality. Some people post constantly; others never post. Accounts may be automated. Videos may be old, edited, staged, or deliberately mislabeled. Popularity may be mistaken for prevalence. Repetition may be mistaken for corroboration. This helps explain why bad ideas survive.

Bellingcat, one of the best-known practitioners of open-source investigation, emphasizes tracing information to its original source and independently verifying it. Its criticism of poor OSINT warns that unattributed reposting and aggregation can create noise rather than knowledge.⁴ Faster collection alone cannot resolve these problems and may simply circulate misleading material more efficiently.

Live Signal and Telemetry Tracking

A second category of OSINT involves machines reporting their own activity. Aircraft, ships, satellites, environmental instruments, weather stations, and other systems constantly generate telemetry. Some of those signals are deliberately broadcast and can be received or redistributed publicly.

Commercial aircraft, for example, commonly use Automatic Dependent Surveillance-Broadcast, or ADS-B. According to the Federal Aviation Administration, equipped aircraft periodically broadcast information that includes position, altitude, velocity, and identification. The broadcast can be received by appropriately equipped receivers.⁵

Ships use a comparable technology called the Automatic Identification System, or AIS, which can transmit vessel identity, position, course, speed, and navigational status.⁶

When thousands of receivers contribute these broadcasts to Internet services, anyone with a browser may be able to observe portions of global aviation or maritime activity. Coverage and timeliness vary, however, and a display described as “live” may contain delayed, incomplete, or erroneous reports.⁶

The Good

The original purposes are overwhelmingly practical. Aircraft tracking improves air-traffic awareness and safety. Vessel tracking helps ships avoid collisions, allows ports to manage traffic, and assists rescue organizations in locating vessels. The U.S. Coast Guard describes AIS as an important navigation and safety capability, and large-scale monitoring systems use it to build maritime situational awareness.⁶

Once aggregated, telemetry also becomes useful for other legitimate purposes. Economists can observe shipping activity. Researchers can study transportation networks. Environmental organizations may detect unusual vessel behavior. Families can follow commercial flights. Journalists can reconstruct the movement of aircraft or ships associated with significant events.

A signal originally created to prevent one ship from colliding with another can therefore become part of our understanding of the global economy.

The Bad

The same precision creates a security problem. Real-time location information can reveal where an individual, aircraft, ship, or valuable asset is located and where it appears to be going.

The International Maritime Organization’s Maritime Safety Committee warned in 2004 that publishing AIS information on the Internet could harm the safety and security of ships and port facilities. That warning illustrates the tension between the navigational benefits of open broadcasts and the risks of making their contents widely searchable.⁷

There is also an important boundary to maintain. Not every signal traveling through the air is open-source information. Publicly broadcast telemetry differs from private communications and information obtained by defeating encryption, bypassing authentication, or intruding into restricted systems. Such activities involve different methods of intelligence collection and raise separate questions about authority, privacy, and the law.

The word open should therefore describe the accessibility of the source, not merely the ingenuity of the person collecting it.

Open-Web Satellite Dashboards and Multispectral Data

Perhaps the most dramatic transformation in OSINT has occurred above our heads.

Satellite imagery was once overwhelmingly associated with governments and intelligence agencies. Today, enormous quantities of Earth-observation data are openly available, while even higher-resolution imagery can be purchased commercially. Modern satellites also see more than the human eye.

A multispectral sensor records energy in several portions of the electromagnetic spectrum. Different surfaces reflect or absorb different wavelengths differently. Vegetation, water, bare soil, snow, smoke, burned land, and other materials can therefore sometimes be distinguished more clearly than they can in an ordinary photograph. Multispectral imaging does not necessarily include thermal sensing; what a satellite can detect depends on the particular instrument and wavelengths it records.

The European Space Agency's Sentinel-2 satellites, for example, use a multispectral instrument covering 13 spectral bands. ESA lists agriculture, forests, land-use change, inland and coastal waters, and disaster mapping among its principal applications.⁸

A web dashboard can place this information on an interactive map and allow a user to compare observations made on different dates. Such a display should not be mistaken for continuous live observation. Satellite revisit intervals, clouds, and the time needed to process and distribute imagery can leave significant gaps between an event and an available image.⁸

The Good

The humanitarian and scientific possibilities are enormous. Satellites can document wildfires, floods, drought, deforestation, crop health, pollution, disappearing lakes, damaged infrastructure, and changes in cities or coastlines.

Repeated satellite observations allow researchers to examine what has changed rather than simply how a place looks at one moment.

ESA notes, for example, that frequent satellite observations can help monitor agricultural production and contribute to forecasting crop yields and food shortages.⁹

Satellite imagery has also changed journalism and accountability. Destruction that might once have occurred beyond the reach of outside observers can sometimes be independently examined from orbit. Physical distance is no longer the barrier to observation that it once was.

The Bad

Earth observation creates a dual-use problem because the same ability to monitor conditions can reveal sensitive locations and activity. Information useful for disaster relief can disclose infrastructure, while imagery used to monitor deforestation may also show changes at sensitive installations. Historical imagery can reveal patterns, routines, and construction that no single photograph would disclose.

There is also a danger of false precision. Although a machine collects the image, interpreting it still requires judgment. A dark patch, unusual heat signature from a thermal sensor, group of vehicles, new excavation, or damaged structure may have several explanations. Resolution, clouds, viewing angle, sensor characteristics, image processing, and the date of collection all affect what can reasonably be concluded.

A satellite may accurately record the radiation reaching its sensor while an analyst still misunderstands what the resulting image shows.

Real-Time Geolocation and Ground-Truth Verification

Geolocation attempts to establish where an image, video, event, object, or person was located. Some determinations can be made quickly, while others require substantial research; “real-time” geolocation is a capability, not a guarantee.

Sometimes the evidence is obvious, such as a street sign. Often it is indirect: architecture, road markings, terrain, vegetation, mountains, shadows, languages, weather, utility poles, storefronts, or the position of buildings.

Open-source investigators frequently compare these clues with satellite imagery, maps, photographs, and other independent sources. Bellingcat describes geolocation as comparing visible landmarks and features in photographs or videos with other imagery to confirm where the material was recorded.¹⁰

Ground-truth verification checks remote observations against evidence obtained at the location, such as field measurements, site visits, or reliable firsthand reports. Comparing digital sources can strengthen a finding, but does not by itself establish ground truth. The underlying principle is to test an interpretation against independent evidence of what actually occurred.¹¹

The Good

Verification is one of OSINT's strongest defenses against deception. A video posted with the claim that it depicts one city can sometimes be shown to depict another. A photograph supposedly taken yesterday may have existed online for ten years. A purported battlefield video may have been filmed far from the claimed location. This is especially valuable in an age of inexpensive editing, synthetic media, and AI-generated content.

Properly performed, geolocation can help journalists authenticate evidence, investigators document crimes, researchers expose misinformation, and emergency organizations understand conditions on the ground.

The Bad

But locating an event can also mean locating a person. A photograph intended to show nothing more than a pleasant view may contain enough environmental information to reveal where it was taken.

That capability becomes troubling when applied to private individuals, political dissidents, witnesses, abuse victims, soldiers, refugees, or anyone attempting to remain anonymous.

The ethical distinction therefore lies partly in what is being located and why. Verifying that a widely circulated video really depicts a particular public event may serve the public interest. Publishing the precise location of a vulnerable individual may create danger without producing any comparable public benefit.

An accurate finding can therefore still cause harm when its disclosure exposes someone to danger.

Multi-Layered Intelligence Dashboards

Modern OSINT becomes especially useful when it combines information from several kinds of sources in a multi-layered intelligence dashboard.

Imagine a single geographic display containing satellite imagery, aircraft, ships, weather, fires, news reports, social-media activity, transportation networks, public databases, historical events, and alerts produced by automated analysis. Each layer may be ordinary by itself. Combined, the layers can reveal relationships that none of them shows independently.

Its importance lies in what intelligence analysts call fusion: assembling separate pieces of information into a more coherent picture.

An investigation might begin with a social-media post reporting an explosion. Satellite imagery acquired around the relevant time may show smoke, aircraft telemetry may indicate unusual flight activity, and weather records may establish wind direction. A photograph geolocated nearby and a report of arriving emergency vehicles can add further evidence. Each source contributes something different, provided its timing, provenance, and independence are checked.

No single observation necessarily establishes what happened. Together, genuinely independent observations can support a conclusion more strongly than any one source alone.

The Good

This kind of synthesis can dramatically improve situational awareness. Emergency managers can see several aspects of a disaster simultaneously. Companies can monitor supply-chain disruptions. Environmental researchers can combine weather, satellite, and transportation data. Journalists can investigate events across national borders without physically entering dangerous areas.

Properly designed dashboards can also preserve uncertainty by distinguishing confirmed information from probable or unverified reports.

The value of this approach lies in allowing independent sources to test one another, rather than merely accumulating reports that may all originate from the same claim.

The Bad

At its worst, the same dashboard becomes an extraordinarily efficient surveillance instrument. Information that seems harmless in isolation may become sensitive when combined.

A ship's position plus corporate ownership records plus satellite imagery plus employee social-media posts plus historical movement patterns can reveal something none of those sources reveals independently. This is sometimes called the mosaic effect: many innocuous fragments forming a surprisingly revealing whole.

Automation adds another danger. Algorithms encourage categorization, alerts, rankings, associations, and predictions. Once those outputs appear on a sophisticated dashboard, they can acquire an aura of authority: a red icon may look like an established fact even when it represents an uncertain inference.

If faulty information from one layer contaminates several others, an intelligence dashboard can produce not merely an error but a beautifully visualized error.

The OSINT Paradox

OSINT illustrates a recurring problem in the history of technology: an increase in our ability to acquire knowledge does not necessarily bring a comparable improvement in our judgment about how that knowledge should be used.

There is little about open-source intelligence that is inherently good or bad. Satellite imagery may document a mass grave, monitor a wildfire, measure the loss of a forest, or reveal activity at a sensitive location. Social-media archives can preserve evidence of atrocities and political repression, but they can also be assembled into remarkably detailed records of the lives and movements of ordinary people. Aircraft and maritime tracking systems improve transportation safety and allow researchers to observe global commerce, while at the same time making the movements of particular aircraft and vessels easier to follow. Geolocation can expose a falsely described photograph or establish where an important event took place, yet the identical methods can reveal the location of someone who has good reason to remain anonymous.

None of these contradictions is unusual. Maps, photography, radio, computers, and the Internet all expanded human knowledge while creating new opportunities for control, deception, or intrusion. What distinguishes modern OSINT is the scale and speed with which previously separate sources can now be combined. Information that once required days or weeks of research may appear on a single screen, updated continuously and correlated automatically with information from dozens of other sources.

That capability deserves neither automatic admiration nor automatic suspicion. Its value depends heavily on purpose, context, accuracy, and restraint.

The more difficult problem is that improved collection does not eliminate the weaknesses of the person interpreting what has been collected. Satellite sensors can record what is present within their technical limits, but they cannot explain the significance of everything they record. A social-media scraper can detect thousands of references to an event without determining whether those references are independent observations, repetitions of the same rumor, coordinated propaganda, or deliberate deception. A sophisticated dashboard can display an impressive number of colored symbols and alerts, while some of the apparent certainty on the screen may rest upon assumptions made several analytical steps earlier.

For that reason, one of the most important disciplines in OSINT is preserving the distinctions between observation, verification, inference, and judgment.

Observation establishes that something has been seen or recorded. Verification asks whether the information is authentic and whether independent evidence supports it. Inference, as part of analysis, attempts to determine what the information means. Judgment concerns what conclusions should reasonably be drawn and, ultimately, what should be done with them.

Modern technology assists enormously with the first three tasks, particularly collection and comparison. It can search more sources, examine larger datasets, recognize patterns, and perform correlations far beyond the practical ability of any individual analyst. But the final questions remain much less susceptible to automation. Whether an intrusion into privacy is justified, whether uncertain information should be published, whether a person should be identified, whether an inference is strong enough to support action, or whether the public interest outweighs the danger created by disclosure are not merely technical questions.

The central problem of OSINT extends beyond the familiar fact that technology can be used for beneficial or harmful purposes. Our capacity to observe the world is advancing faster than the social, ethical, and intellectual habits needed to interpret those observations responsibly.

As finding things out becomes easier, the more demanding task is deciding what the evidence actually establishes and what we should do with that knowledge.