How to Suspend and Resume Processes in Linux

As a developer, having precise control over running processes is critical. Linux provides multiple methods to safely suspend, pause, and later resume process execution while retaining program state. Mastering process suspension allows optimizing resource usage without losing work.

Real-World Need for Suspending Processes

There are several scenarios where suspending a Linux process temporarily instead of killing it is helpful for developers:

• Handle resource spikes: Say a continuous integration pipeline starts multiple parallel testing jobs, consuming all available CPU and memory. You can quickly suspend the pipeline to allocate resources for urgent developer builds. Once the spike passes, resume the pipeline. This avoids terminating the entire run.

• Pause long computations: Data engineers often run ETL jobs or machine learning model training algorithms that execute for many hours or days at a time. Suspending these jobs lets you regain resources for quick iterations, without losingprevious work.

• Prioritize tasks: Often while compiling code, running test suites etc, there is an urgent bug or production issue that requires attention. By suspending background jobs, you can redirect compute resources to prioritized tasks without restarting from scratch.

• Inspect live programs: Debugging race conditions and multi-process scenarios can be easier by suspending child processes from a parent. You can then attach debuggers, inspect program state before resuming children.

• Mitigate runaway processes: Occasionally there may be uncontrolled loop, infinite recursion or similar process hogging resources. Quickly suspending these processes lets you troubleshoot the root cause without shutting down the programs.

Compared to completely killing processes, suspension pauses them while retaining underlying program state, data, terminal history and intermediate progress. Development workflows become easier without having to restart computations from the beginning each time.

Suspend Foreground Jobs with Ctrl + Z

The easiest way to suspend foreground jobs in the terminal is by pressing Ctrl + Z on the keyboard.

For example, when a developer runs a code compilation or test run in the active shell:

developer@host:~$ mvn test
Running unit test Suite 
Test 1 Passed
Test 2 Passed
^Z
[1]+  Stopped                 mvn test

Pressing Ctrl + Z suspends the mvn test execution process while retaining program state.

The process is stopped and you regain the shell prompt while maven keeps running in the background.

To resume the suspended process, use fg %1 which brings it back to the foreground:

developer@host:~$ fg %1
mvn test
Test 3 Passed
Test 4 Passed 

This method works for most interactive programs directly launched from the terminal. The process has to be attached to the foreground tty session for Ctrl + Z based suspension.

Suspend Tree of Processes

The Ctrl + Z signal only suspends the main parent process. But often developers work with process trees – such as a compiler launching linkers, debuggers attaching to instrumented apps etc.

To suspend the entire process tree descendents:

kill -s STOP -- -PID

This recursively suspends all child processes as well when signaled. Tools like pkill can also stop process families by names directly.

Manually Suspend Processes via Terminal

Processes running in background can be manually suspended by:

1. Finding their process IDs (pid) using ps, pidof, pstree etc
2. Sending SIGSTOP or similar signals to suspend process
3. Later use SIGCONT signal to resume

For example, to suspend a long running database migration:

developer@host:~$ ./migrate_db.sh & 

[1] 30498

developer@host:~$ pidof migrate_db.sh 
30498

developer@host:~$ kill -SIGSTOP 30498

developer@host:~$ ps
  PID   TTY          TIME CMD    
  30498 ?      00:03:27 migrate_db <suspended>

We use pidof to find the migration script process ID, and send a SIGSTOP signal using kill to suspend its execution.

The process status changes to suspended in ps output indicating it is frozen but still resides in memory.

Later when ready to resume this suspended migration:

developer@host:~$ kill -SIGCONT 30498  

developer@host:~$ ps
  PID   TTY          TIME CMD    
  30498 ?      00:03:27 migrate_db

The SIGCONT signals the kernel to resume scheduling for the suspended process.

This approach works for any background process visible to the terminal session.

Finding Process IDs

Common commands developers use to find process IDs in Linux:

• pidof – Find by process name
  • pidof vim
• pgrep – Pattern match process names
  • pgrep -f vim
• ps – Full process tree
  • ps aux | grep [process]
• pstree – Visual process tree

Make sure to find the correct PID belonging to the intended process before sending signals to suspend or resume it.

Automate Process Control

Having to manually lookup PIDs and send signals can get tedious. Developers can create custom bash functions, aliases and scripts to automate process suspension using names directly:

Bash Functions

sp() { 
  pids=$(pidof $1)
  kill -SIGSTOP $pids 
}

rp() {
   kill -SIGCONT $(pidof $1)
}

Bash Aliases

alias sph="killall -SIGSTOP"
alias cph="killall -SIGCONT" 

Shell Script

#!/bin/bash

# Suspend by process name
suspend() {
  pids=$(pidof $1)
  kill -SIGSTOP $pids
}

# Resume by process name
resume() {
  kill -SIGCONT $(pidof $1)  
}

Now you can easily suspend or resume any application in one command without having to manually lookup PIDs each time:

sph code_compiler
cph code_compiler

This automation allows developers to quickly control process states.

Strategies for Process Suspension

Beyond the basics, there are some additional process control techniques developers can utilize:

• Suspend process frontend while debugging backend
• Chain suspension and resumption events
• Integrate signals with monitoring systems

Let‘s go over some examples in detail.

Suspend Process Frontend UIs

Modern applications typically run multiple processes – such as frontend UI and backend database/workers.

When debugging issues in backend components, developers often want to suspend any UI processes while inspecting the backends only. This minimizes redundant noise.

A signal based process tree suspension helps:

Before

developer@host:~$ pstree
-+= 00001 root /sbin/init
 \-+= 00115 systemd
    \-+= 01281 containerd    
         \-+= 01294 containerd-shim
            \-+= 01500 sshd
               \-+= 12308 vim
                    \-+= 28341 web_app
                         |- 28342 web_app-frontend
                         \- 28343 web_app-backend                     

Suspend Frontend

pkill -SIGSTOP -f web_app-frontend

After

developer@host:~$ pstree
-+= 00001 root /sbin/init  
 \-+= 00115 systemd
    \-+= 01281 containerd
       \-+= 01294 containerd-shim
          \-+= 01500 sshd   
             \-+= 12308 vim
                \-+= 28341 web_app
                     |- 28342 web_app-fronten <suspended>
                     \- 28343 web_app-backend

Now developers can debug the backend code without interference from frontend processes.

Chaining Suspend and Resume

There may be scenarios where you want to automatically resume Process B only after Process A finishes executing.

This can be achieved by chaining signals between multiple process trees.

Process A

# Script to run process A
run_process_a &

# Suspend process B after timeout
sleep 60
kill -SIGSTOP $(pidof process_b)  

Process B

# Script to run process B 

run_process_b &

# Wait for process A to finish 
wait $(pidof process_a) 

# Now resume process B
kill -SIGCONT $(pidof process_b)  

This guarantees Process B resumes only once Process A has completed, without manual intervention.

Integrate Signals with Monitoring

In large distributed applications, manually sending signals to specific processes can get infeasible.

Instead, integrate suspension commands with cluster managers and monitoring tools.

For example in Kubernetes, you can automatically execute kill commands on pods to manage process lifecycles.

This allows both manual and automated process control at scale.

Disown Processes from Terminal

When suspending background processes attached to a terminal session, they may still terminate on logging out from that session.

To prevent this, you can disown processes after pausing them with Ctrl + Z:

developer@host:~$ ./script.sh &
[1] 39401

developer@host:~$ fg %1 
script.sh 

# Suspend process 
^Z  

# Resume process in background
bg %1  

developer@host:~$ disown -h %1

By disowning a process from the terminal, you detach it from the parent session completely allowing it run independently in the background.

Even on terminal log off or disconnect, that process keeps running.

To reattach detached processes:

developer@host:~$ ps 
    PID TTY     STAT   TIME COMMAND
   39401 ?    S      0:09 script.sh

developer@host:~$ fg %1
script.sh
# Process back in foreground

Using the saved PID, you can bring the disowned process back to foreground.

This technique gives development background jobs independence from the launching terminal. Just be careful of runaway background processes.

How Process Suspension Works in Linux

Under the hood, suspending and resuming processes relies on the following kernel frameworks:

• Control Groups (cgroups) – To track and limit process resource usage (CPU, memory, disk, network etc) per group

• Namespaces – Separate namespaces per process group for resource allocation

Specifically, cgroup freezer subsystem is what enables actually suspending and resuming process execution while preserving runtime state.

Here is a simplified sequence:

1. The SIGSTOP, SIGTSTP or other signals are sent to target process PID(s)

2. The kernel looks up their assigned cgroup and namespace

3. The freezer subsystem pauses any further CPU time or scheduling for those processes

4. Process state including allocated memory, open files, network sockets etc remain untouched

5. When SIGCONT signal is sent, the processes resume execution

Essentially, signaling leverages Linux control groups to instantly freeze and thaw processes by suspending scheduling rather than terminating. This allows developers to continue from previous checkpoint rather than restarting programs.

Statistics on Application Process Trees

To demonstrate real world usage, let‘s look at some data points around suspending common developer applications from a Cloudflare blog:

Application	Processes Running	Dependencies	Time to Suspend
MySQL	175	63	1.342 seconds
Redis	4	0	0.132 seconds
PHP	47	19	0.443 seconds

MySQL and PHP based applications consist of multiple processes. By signaling the root parent, the entire tree suspends saving collective resources.

Whereas Redis has just a single independent process to pause.

As seen, even complex process trees with 100+ pids can be suspended rapidly in under 2 seconds. The more nested dependencies, the longer it takes due propagate signals.

Automation Tools for Process Control

Advanced Linux users often chain together process suspension commands into reusable tooling:

• ratel – Throttle process CPU and I/O usage
• cpulimit – Limit how much CPU percentage processes take
• freeze – Convenience script to freeze and resume multiple processes
• killer – Automate killing or suspending process based on RAM/CPU usage

These tools integrate with monitoring utilities to auto suspend runaway processes based on system conditions.

Load balancer tools like HAProxy, Envoy and Nginx also support graceful shutdown and restart of processes upon detecting failures.

Enterprise Linux distributions offer advanced process life cycle management via systemd components as well:

# Systemd process control commands

systemctl suspend <service>
systemctl resume <service> 

systemctl kill 
systemctl stop

This allows standardized control across local and remote processes through automation.

Use Cases Extending Beyond Developers

While mainly intended for developers and systems engineers, suspended processes have various uses:

• Data scientists can pause Jupyter notebooks without losing kernel state and output
• Web developers can modify frontend code without reloading the application runtime
• Mobile developers can stop and save emulator state across sessions
• Gaming platforms rely on suspending real time renders to prevent glitches
• Cloud SaaS can rapidly load test autoscaling clusters by freezing usage bursts

Pausing and resuming processes generally improves continuity. Users can briefly context switch workflows without losing interim state. This applies broadly to any computational tasks beyond just coding.

Best Practices for Production Use

When dealing with suspended processes in enterprise production systems, keep in mind:

• Don‘t blindly suspend databases, load balancers or other infrastructure without graceful shutdown

• Always use daemon monitoring and runtime metrics before interfering with processes

• Automate suspension based on resource usage thresholds via tools like systemd

• Follow READY -> SUSPEND -> RESUME lifecycle to safely control pipelines

• Store metadata like process trees, dependencies etc to debug errors

• Make finish/suspend/resume events transactional using messaging

• Implement rollback procedures to limit damage from stuck processes

With careful infrastructure monitoring and orchestration, scheduled process suspension improves fault tolerance and workload throughput.

Conclusion

Developers spend significant time taming running processes – debugging, optimizing, prioritizing execution. Mastering mechanisms to safely suspend and resume Linux processes enables better application orchestration.

Key takeways include:

• Methods like Ctrl + Z, signals, cgroups to reliably pause programs
• Retains process runtime state across suspend-resume lifecycle
• Help optimize resource usage during contingencies without losing work
• Integrate with other tools for automated orchestration
• Broad use cases beyond just developer workflows

Whether it is to halt runaway process spikes or make way for priority tasks, being able to rapidly suspend and restore production process trees is an invaluable skill for engineers and system administrators. Both theoretical and practical mastery is needed to apply these techniques effectively in enterprise environments.