Six Container Symptoms and the Causes Behind Them

Six Container Symptoms and the Causes Behind Them
Security Tech

Good diagnosis starts by separating the symptom from the cause. A fever is not the illness, and a failing container is not the fault. In security-minded operations, where an unexplained behaviour may be a sign of weakness, the habit is especially valuable. This guide takes six symptoms that teams meet when they run containers on a host, and pairs each with the cause the Ops Atlas gives under ContainerOps, plus a remedy. The Atlas uses the name for the care of single containers: images, registries, runtimes, limits and logs. It admits the label is its own and not a formal standard, which we respect by sticking to what the Atlas’s ContainerOps entry actually records.

Each section has the same shape: what you observe, what is probably going on, how to confirm, and what to change. Treat the causes as hypotheses. Where the Atlas offers a worked number, it is used as an illustration, not as a universal rule.

Symptom 1: the container disappears with exit code 137

Probable cause. The code equals 128 plus 9, so the process received the unconditional kill signal. Two common reasons exist. Either the memory ceiling was reached, or the stop request was ignored and the engine ran out of patience after its grace period, which in Compose defaults to ten seconds. The Atlas adds a subtlety: on current control-group generations the out-of-memory status can also reflect exhaustion across the whole host, so a container may be the victim, not the culprit.

Confirm. Check the container’s recorded state for an out-of-memory flag, and compare its memory use against the limit. Time a manual stop; ten seconds exactly points to signal handling.

Fix. If it is memory, find the leak or raise the ceiling deliberately. If it is signals, write the entrypoint in exec form, handle the stop request in the program, and add an init helper. The Atlas treats memory and process limits as mandatory, so do not remove them to hide the symptom.

Symptom 2: a service is reachable from the internet although the firewall says otherwise

Probable cause. Published container ports are processed in the network address translation stage, ahead of the rules most host firewall tools manage. The Atlas also notes that a mapping written without a host address is bound across all IPv4 and IPv6 interfaces, a default the Docker documentation itself calls insecure by default.

Confirm. Scan the host from outside your network, not from itself. Compare the result with what your firewall configuration claims.

Fix. Bind internal services to the loopback address, give containers that only talk to one another a network with no outside route, and expose a single reverse proxy deliberately. Do not rely on the firewall front-end’s status page as proof.

Symptom 3: a credential shows up in an image that “never contained” it

Probable cause. A build step fetched a secret and a later step deleted it. Because layers record changes, the earlier layer retains the file, and anybody holding the image can extract it.

Confirm. Inspect the layer history and look for steps that handle credentials. Pull a copy and unpack its layers.

Fix. Rotate the credential first, since removing a tag does not recall copies that have been pulled or mirrored. Then rebuild using secret mounts that exist only for the step that needs them. This is a good moment to check mirrors and caches for old copies.

Symptom 4: latency spikes while processor use looks healthy

Probable cause. Processor throttling. The Atlas’s worked example is instructive: a container limited to two processors used 90 seconds of processor time in a 60 second window, an average of 1.5, yet the kernel held it back in 240 of 600 enforcement periods, a 40 percent rate that cost six seconds of idle time. Short bursts hit the cap even though the average does not.

Confirm. Read the throttle counters exposed by the control group and divide the throttled count by the period count. For latency-sensitive services the entry suggests staying under five percent.

Fix. Loosen or remove the processor cap while keeping the memory cap, and check how the language runtime chooses its worker counts. The Atlas warns that a call asking for the processor count may still report the host’s total, which produces pools that are too large for the limit.

Symptom 5: builds start failing with “too many requests”

Probable cause. Registry rate limits. The entry gives anonymous pulls from one popular public registry as 100 per six hours when checked, and 200 for a signed-in free account. Its illustrative scenario has thirty build runners producing 180 anonymous pulls an hour against a budget of 100 per six hours, so refusals follow.

Confirm. Look at the response codes in the build logs and count pulls per hour.

Fix. Authenticate the runners and place a pull-through cache in front of the registry. Keep in mind the Atlas’s reminder that a cache is not a backup. Release images still need a retained home of their own.

Symptom 6: a patched package keeps coming back

Probable cause. Someone applied the fix by opening a shell in the running container. The change lived only in that instance, so the next restart restored the old image. The Atlas lists this habit among its anti-patterns.

Confirm. Compare the running container’s file differences with its image. A clean setup should show almost none.

Fix. Change the build recipe or the base image, rebuild, and redeploy by digest. The entry further proposes tracking how many days pass between a fix existing and that fix being live, giving seven days for critical items as a sample policy. If that number is poor, the pipeline is the thing to repair.

Reading across the six

Look at how often the root cause is a belief rather than a defect: that the firewall covers everything, that deleting is removing, that the average describes the burst, that a shell fix persists. Security work rewards the habit of testing assumptions directly. Our visitors planning a trip to the state do something similar when they read ways to explore Gujarat beyond the headline attractions and compare it with first-time visitor experiences across the state: they check what is really on offer against what the brochure implies.

Why the order of questions matters

It is tempting to jump straight to the exotic explanation, such as a kernel bug or an attack, because it makes the problem feel important. In practice the Atlas’s list of anti-patterns reads like a catalogue of ordinary shortcuts: floating tags, shell patches, privileged flags, unrotated logs, unowned services. Rule out the ordinary causes first. They are cheaper to test, far more frequent, and usually just as instructive.

A short triage routine

  1. Record the exit code, the time and the last log lines.
  2. Check limits and the out-of-memory flag.
  3. Check published ports against what you intended.
  4. Check how the container was launched: user, capabilities, entrypoint form.
  5. Check how the image was built and how it is named in production.

Five checks, in that order, resolve a large share of container mysteries before any exotic tool is needed. Write down what you found next to the symptom, and next time the cause will be a lookup instead of an investigation.